search

frozenpandaman / splatnet2statink

Takes battle data from the SplatNet 2 app and uploads it to stat.ink
https://github.com/frozenpandaman/splatnet2statink/wiki
346 stars 61 forks source link

Nintendo Switch Online v1.4.1 update #56

Closed frozenpandaman closed 5 years ago

frozenpandaman commented 6 years ago

To look into & update user agent info in iksm.py. Let's hope it doesn't break anything…

frozenpandaman commented 6 years ago

https://github.com/frozenpandaman/splatnet2statink/blob/0c150ca524a0c9748e4d607281717d4d6f91da30/iksm.py#L61 Changed to "Select this person"

frozenpandaman commented 6 years ago

v1.4.1 adds certificate pinning, so mitm no longer works (getting error 2817-0599). v1.1.2 and below still work. Edit: No longer true. Also, not cert pinning, just targeting a higher API version.

edmiyaa commented 5 years ago

v1.4.1 adds certificate pinning, so mitm no longer works. v1.1.2 and below still work.

I'm having this problem right now, I was trying to extract the iksm_session key as I did many times before, but then I realized that it seemed to ignore the mitmproxy certificate. So, what would be the solution? Downgrade the app and try again?

frozenpandaman commented 5 years ago

@edmiyaa Yup. Manually install the 1.1.2 apk and use that. I updated the mitmproxy instructions to say this as well. (Hopefully this version will keep being supported, and Nintendo won't force you to upgrade…) Or, use splatnet2statink automatic cookie generation and you can generate an iksm_session cookie right from your computer instead of having to go through the app! cookie generation is broken now

edmiyaa commented 5 years ago

Alright thanks! I'll try splatnet2statink and see how it goes.

frozenpandaman commented 5 years ago

Bizarrely, @clovervidia says it's still working for him using v1.4.1 of the app: image

But it's still not for me – or @edmiyaa, evidently.

I tried with Charles Proxy (couldn't get Fiddler's Mac version to work) to verify it wasn't some mitmproxy issue to also no luck: image

It may have worked for @clovervidia since he's on a version of Android lower than 7.0…? image

🤔

Tiryoh commented 5 years ago

I'm not sure about this, but it seems like v1.1.2 client is not working for now.

$ ./splatnet2statink.py -r
splatnet2statink v1.1.9
The stored cookie has expired.
Attempting to generate new cookie...
Error from Nintendo:
{
  "status": 9427,
  "errorMessage": "Upgrade required.",
  "correlationId": "9456d333-********"
}

$ git log
commit 186ba89cb67d9bd03dc41b23bdf768db237004cc (HEAD -> master, upstream/master)

Writing config.txt manually by using v1.4.0 app (iOS) + mitmproxy worked fine.

_2018-09-18_15_40_08
frozenpandaman commented 5 years ago

Was just about to post, and talked about this with @hymm earlier too… Yup, old versions of the app are now no longer allowed. Changing the user agent to 1.4.1 (or 1.4.0 as that's the current version for iOS Edit: iOS is now on 1.4.1 as the latest version too) fixes the "Upgrade required" error, naturally, but then it changes to "Bad request": image

It appears the key used to generate the f HMAC has changed. 🙁

Great that you can still use mitmproxy on iOS. I think it's blocked on Android 7.0 and up, which certainly is an issue…

Tiryoh commented 5 years ago

It appears the key used to generate the f HMAC has changed. 🙁

Oh... Thanks for the information. I will use the iOS app + mitmproxy for the time being.

frozenpandaman commented 5 years ago

Network Security Configuration is what prevents mitmproxy from now working in Android 7.0 and higher. More info here and here. I installed the app on an emulator running an older version of Android and I'm now able to MITM the requests fine.

There have been some changes to /v1/Account/Login, namely introducing two new parameters:

"requestId": "123e4567-e89b-12d3-a456-426655440000",
"timestamp": 1537249847

That's a UUID version 4 and unix (epoch) timestamp. When these are included, the error changes from "Bad request" to "Invalid token": image

YDKK commented 5 years ago

It seems they generate new f parameter in the native library, which is difficult to analysis. 😞

hymm commented 5 years ago

I decompiled the so, but can't really read assembly :'(

frozenpandaman commented 5 years ago

Yeah, I played around with some tools too – apktool, dex2jar, dnSpy, IDA Free, RetDec – and got the disassembly of gen_audio_h within (x86) libvoip.so as well… but this is so far outside my area of knowledge I don't know how to proceed. :x

edmiyaa commented 5 years ago

So, what is the impact of not knowing how to generate the f parameter? It only affects the automatic generation of the iksm_session cookie right? For now I'm doing it manually (older version of the app together with mitm proxy), but I'm worried when Nintendo starts forcing to use the latest version of the app.

hymm commented 5 years ago

@edmiyaa You can use an older version of android like on the Nox emulator with the latest version of the app.

edmiyaa commented 5 years ago

@hymm Thanks! I'll keep that in mind.

chrhasse commented 5 years ago

I don't think it's that the app implemented certificate pinning, but rather that the app now targets a more recent API level (since google now requires that), and more recent API levels not trusting user certificates (only system certificates) by default.

If you use Magisk you can just install this module to trust user certs.

If you have root you can also install the mitm certificate as a root certificate by naming it with the output of openssl x509 -noout -subject_hash_old -in ~/.mitmproxy/mitmproxy-ca-cert.pem followed by .0 (for example c8450f0d.0), and copying it to /etc/security/cacerts/. you might need to remount system as read/write before you can copy it.

Without root you can also use this tool to modify the APK to use additional certificates from a specified path.

This info is mostly taken from this thread if anyone wants more discussion on this.

frozenpandaman commented 5 years ago

@edmiyaa Nintendo already forces you to use the latest version of the app. But yeah, just use an older version of Android (like the mitmproxy instructions page suggests), or one of the solutions just posted above.

@chrhasse Oooh yeah, you're totally right. @YDKK was saying before that he hadn't found any evidence of cert pinning either. Thanks for the correction. This is also supported by this email I got from Google a few days ago… :P For reference, the Google blog post about is here. I'll update the docs for accuracy – thanks!

I can't root my phone (thanks Verizon for locking the bootloader on my Pixel!) but those are good resources to have! I'll look into the tool you linked, and thanks for the link to the thread on the mitmproxy repo as well. None of these methods of are of course ideal though – there are for sure multiple workarounds, but at this point what I'm really hoping for is to figure out the new way f is generated… though, again, it's proving pretty tough. 😣

Edit: AddSecurityExceptionAndroid doesn't seem to be working for me – after installing the output apk, launching the app just leads it to hang on the splash screen, whether connected via mitmproxy or not.

valknight commented 5 years ago

@frozenpandaman Does the app work in an emulator? You could try using a rooted android emulator, and then using mitmproxy on that.

frozenpandaman commented 5 years ago

@Valerokai Doesn't even need to be rooted – mitmproxy works fine as long as you're running a version of Android below 7.0. (The mitmproxy instructions have said this & have suggested an emulator to use for a few days now as well. 😜)

Getting your cookie manually like that has always been possible (and always should be). It's just the generation that is broken now – i.e. the splatnet2statink API returns f values that no longer work – because we don't know how the new f is generated.

frozenpandaman commented 5 years ago

f can now be different lengths (here are 10 examples): image

Namely, it seems it can be any even number between 32 and 46 in length – maybe even slightly longer or shorter and I just haven't seen them yet in all my testing, since the length can vary.

Also fwiw, the DLLs in the iOS .ipa aren't obfuscated (via Dotfuscator :B) like those in the Android .apk are, at least. But not even sure that helps, really.

chrhasse commented 5 years ago

Is there a way to get the ipa file without an iOS device? if it's not obfuscated it would definitely be easier for me to look at

edit ok I found out how to download an app, but anything interesting seems to be hidden in the CoraliOS file which is apparently encrypted and can only be decrypted by a jailbroken iphone. If anyone has access to one and can use Clutch to decrypt it we might be able to get somewhere.

frozenpandaman commented 5 years ago

@chrhasse I downloaded it from my (very old) jailbroken iPod and didn't have any trouble opening that file. I don't see any contact info on your GitHub profile, so DM me on Twitter. :)

vlee489 commented 5 years ago

Any chance I could get my hand on those files @frozenpandaman , I haven't been able to use mitm cause university internet and I left my jailbroken iPod/rooted S4 back home >.<

This update broke my work on splatstats.ink

frozenpandaman commented 5 years ago

Oops, I tried to combine your multiple comments in a row into one and GitHub decided to let everybody know about it lol. Sorry about that.

Anyway, @vlee489, I'm not familiar with the project but it doesn't look like it's public yet. Isn't weapon usage & team stats something @hymm's http://squidtracks.ink/ already does? (EDIT: There's now also https://www.splatmeta.ink/.)

Re: the .ipa see my above comment.

vlee489 commented 5 years ago

yeah but I'm adding some data analysis into what squidtracks itself has. I'm still working on the code that does the comparing and displaying of data, just d3.js last update broke a ton of my code for the data visualisation part of my website.

I had planned to release it in beta last month but having to optimise the server is taking a bit longer than I wanted.

my main issue is that I can't access splatnet to download the weapon data anymore and I can't use mitmproxy on my Uni network to do the reverse engineering work.

edit: I shouldn't had my last comments as 1 instead of 2, my bad

frozenpandaman commented 5 years ago

Well, hopefully we can figure out stuff soon (it took multiple months previously, when f was introduced for the first time 😉) so you can continue work on it.

(A bit off-topic now, but what if you make a Wi-Fi hotspot from your phone, connect your computer to it, and then use an Android emulator?)

splatoon2.ink has a few API endpoints for making things like stage rotation, salmon run schedule, gear shop, etc. public, i.e. taking the data from the app's iksm_session-required API endpoints, cleaning them up a bit, and making it accessible through a new URL. You could ask @misenhower if he would want to make one for League data, too, which would prevent you from having to use your own cookie.

vlee489 commented 5 years ago

(I can't really efficiently run Hyper-V on top of KVM, saying that I already have a makeshift router in my room working on rerouting my switch to get my switch to play online)

I could ask @misenhower about making an API endpoint but, there are 72 league sets day to get (96 if you include ALL), and that's a lot to cache and make sure they are valid. plus I pull more in then just league data to make things work.

clovervidia commented 5 years ago

@vlee489 I might be able to help if you're looking for ways to get league leaderboard data. Do you have a preferred way to DM, like Twitter or Discord?

frozenpandaman commented 5 years ago

Moving main discussion of f generation to #62. Keeping comments unlocked here in case there is further discussion around related issues or continued conversations from earlier.