Protobuf structure for MapCells has changed

[JSchwerberg]

An update today caused the Protobuf structure for Mapcells to change -- looks like it is now in a tuple (?)

Proto files need update to reflect.

PLEASE KEEP THIS ISSUE THREAD ON TOPIC ie donations/thankyous/unrelated questions/flaming etc.

Thankyou

[broach]

Please stop posting nonsense.

The repo people are linking to without knowing how to program / understanding it is a change to a bot's code to have their own internal method return a tuple of everything in one call. The first item in the tuple is the MapCell proto.

THAT PROJECT USES THIS REPO FOR PROTOBUFS FFS

[JSchwerberg]

You're right, all three of those projects that linked here for the same issue changed their internal method to return a tuple of everything in one call. Then all of those devs (myself included) magically forgot, and decided to trace the problem all the way back to here, just for you to comment and tell us that. My hero, @broach. <3

[JSchwerberg]

Someone dumped the changed protobuf Responses here -- this may be useful if anyone understands Protobuf well enough (I don't) to make a PR to fix this.

[Trolldemorted]

how reliable are the sizes of types in that dump?

This looks like a pokemon spawn:

100 {
  1 {
    1: "030a3476-668a-47fb-95ed-2bcfc5c15637/1467338129695000"
    2: "pm0094"
    3: 1467338129695000
    4: 0x4111e825
    5: 401141
    6: "\r\014\222M\355\223\257\222\366\0222\200G\"\267\244"
  }
[...] 

so i guess the interior values are:

1: some sort of id (?) + timestamp 2: pm+pokedex id (94 = Gengar) 3: same timestamp 4: spawnpoint_id (?) 5: time till hidden in ms (?) 6: unknown

and since i assume lat/lng are transmitted, it would make sense to me if they were in 6.

edit: some person on reddit said that that is no get_map_objects response, and states that the responses did not change.

[justMaku]

That's correct, responses did not change, they just figured a way to distinguish between real app and 3rd party API clients.

We're probably not sending some fields (I'd wager on Unknown6 message) at that's how they know not to serve us any Pokemon.

[tcmaps]

"030a3476-668a-47fb-95ed-2bcfc5c15637" is a GUID

[Trolldemorted]

@justMaku it should be rather easy to test unknown6 if you got a working mitm proxy at hand - just change the value to something arbitrary and check whether you see pokemon or not, or am i missing something?

[justMaku]

@Trolldemorted that's correct, i'm running iOS though so it's much harder to get MITM proxy to run.

[justMaku]

@BoBeR182 please stop reposting random things that you find on the internet because that's not correct and you're just confusing everyone.

[trisk]

@BoBeR182 you are definitely posting in the wrong issue

[trisk]

I have a working MITM proxy. Are we talking about unknown6 in the RequestEnvelope?

[justMaku]

@trisk correct, from what I've seen none of the available API clients send that value. There's probably some reverse engineering work needed to be done on the client to know what that value actually stands for though.

[justMaku]

Also @trisk It'd help us all greatly if you (or anyone else with MITM setup) would publish a whole dump of the tcp stream from login up to map update received.

[0xAcid]

@justMaku, I'll work on it. So, you need connection up to the moment I see my trainer on the map right ?

[waryas]

https://gist.github.com/trisk/8e0ec7203e637f25274c19e4b309c6b5

[trisk]

Also binary dump of requestenvelopess and responseenvelopes from another session in single file: https://drive.google.com/file/d/0B2BOEbAy3h0hT0ZadDhYUW03QWM/view?usp=sharing (Sorry, no delimiters between requests and responses)

[trisk]

Gonna try modifying unknown6 now that I verify I can reserialise the envelope.

[justMaku]

That binary dump is what I've been looking for, unfortunately can't use it much without the delimiters :/

[ur0]

@Trolldemorted, that's an asset digest.

Here's actual GET_MAP_OBJECTS output

  1 {
    1: 4316619620929765376
    2: 1470268521140
    5 {
      1: 0x4345ce401e3458bd
      2: 1470268521140
      3: 0x40333734b2cb8f27
      4: 0x405233faa7b2c695
      5: "3be7b6acc39"
      7 {
        2: 127
      }
      11: 201392
    }
    5 {
      1: 0x362864505220174d
      2: 1470268521140
      3: 0x4033371d27242400
      4: 0x405233faa7b2c695
      5: "3be7b6ace9d"
      7 {
        2: 118
      }
      11: 570144
    }
    5 {
      1: 0x7d0aead003ebd3cd
      2: 1470268521140
      3: 0x4033372e3af9373c
      4: 0x405233f92473c567
      5: "3be7b6acc3d"
      7 {
        2: 13
      }
      11: 422112
    }
    10 {
      1: "3be7b6acc39"
      2: 0x4345ce401e3458bd
      3: 127
      4: 1470268722532
      5: 0x40333734b2cb8f27
      6: 0x405233faa7b2c695
    }
    10 {
      1: "3be7b6ace9d"
      2: 0x362864505220174d
      3: 118
      4: 1470269091284
      5: 0x4033371d27242400
      6: 0x405233faa7b2c695
    }
    10 {
      1: "3be7b6acc3d"
      2: 0x7d0aead003ebd3cd
      3: 13
      4: 1470268943252
      5: 0x4033372e3af9373c
      6: 0x405233f92473c567
    }
    11 {
      1: 10
      2: 0x43480000
      3: 0x10849a8fe721be8d
    }
    11 {
      1: 98
      2: 0x43480000
      3: 0xde6a3fdfaa2e7b9d
    }
    11 {
      1: 127
      2: 0x43480000
      3: 0x4345ce401e3458bd
    }
    11 {
      1: 118
      2: 0x43480000
      3: 0x362864505220174d
    }
    11 {
      1: 129
      2: 0x43480000
      3: 0x26b6ea801fbfd1fd
    }
    11 {
      1: 13
      2: 0x43480000
      3: 0x7d0aead003ebd3cd
    }
  }
  1 {
    1: 4316619614487314432
    2: 1470268521140
  }
  1 {
    1: 4316619616634798080
    2: 1470268521140
  }
  1 {
    1: 4316613745414504448
    2: 1470268521140
  }
  1 {
    1: 4316613738972053504
    2: 1470268521140
  }
  1 {
    1: 4316613741119537152
    2: 1470268521140
  }
  1 {
    1: 4316613736824569856
    2: 1470268521140
  }
  1 {
    1: 4316619644552085504
    2: 1470268521140
  }
  1 {
    1: 4316619646699569152
    2: 1470268521140
  }
  2: 1
}

1.5 now seems to contain the pokemon info, 1.5.7.2 is the Pokemon ID.

[trisk]

Login doesn't get far with empty unknown6: https://gist.github.com/trisk/ec3db5c41fa6d1d74c64a33664632c0a

[cyraxx]

@ur0 That's the same as before, 1.5.7.2 has always contained Pokemon ID.

1 = GetMapObjectsResponse.map_cells 5 = MapCell.wild_pokemons 7 = WildPokemon.pokemon_data 2 = PokemonData.pokemon_id

[trisk]

@justMaku Split all requests/response envelopes into separate files, ordered by time: https://github.com/trisk/pkre-dumps/tree/master/dump1

[0xAcid]

If this is of any use, I used apktool to get .smali of apk 0.29 and 0.31 and then made a diff of all the files. You can access the diff here : https://gist.github.com/Axi0m-S/2ec2c74a26c722440f371ab46c45eb5d It might help someone that know .smali better than me.

---

EDIT : Another Diff for dex2jar-red files : https://gist.github.com/Axi0m-S/a1298143654d64021f13cca126447d40 if that is of any interest. could that :

"> getTrustManager(java.lang.String java.security.KeyStore )"

be something useful ? concerning request signing etc.

[FabianTerhorst]

I don´t see any big changes between the last release. https://github.com/FabianTerhorst/PokemonGo/commit/22ee9af7f40c9fa2bfe2b063e430e8be488f1584

[Saicheg]

could it be something that lives on unknown6 field?

https://github.com/FabianTerhorst/PokemonGo/commit/22ee9af7f40c9fa2bfe2b063e430e8be488f1584#diff-c4b70be9b0036965856ebbd284eae545R41

[trisk]

I just verified that replacing unknown6.unknown2.unknown1 with a pattern of the same length produces the empty cell info. Leaving it alone I get the correct GetMapObjects response.

[iGio90]

@Saicheg there is nothing that use that method. Im on smali, trying out to find some references on smali

[trisk]

@iGio90 @Axi0m-S the Dalvik code is not relevant here because it's not responsible for the protocol, please discuss that somewhere else.

[DimaVIII]

Just found may it helps: https://www.reddit.com/r/pokemongodev/comments/4w0jum/all_ptcgoogle_logins_failing_from_api/d63553b

For what it’s worth, MITM proxies still work, data sent and received is still read correctly, but as soon as I try to change anything in what’s being sent to the server, it returns an empty response and the game says “Error”.

For example, if I add the field spin_modifier = 1.0 to the CatchPokemon requests the game sends to the server, it says “Error” whenever I try to catch a Pokémon with a non-spinning ball, however it works fine if the ball is spinning. Same goes for normalized_reticle_size, if I change it to anything that was not the value given by the game, the server sends an empty response.

Looks like there could be some kind of checksum to detect if the data was forged/tempered with.

[d-pollard]

I just verified that replacing unknown6.unknown2.unknown1 with a pattern of the same length produces the empty cell info.

Could you change the unknown6 to a pattern with a same length, then change it back to the original value to ensure that the request isn't also time (or 1 use) sensitive as well

[0xAcid]

@trisk From what I read on reddit, the field unknown6 may be related to request signing : https://www.reddit.com/r/pokemongodev/comments/4w124h/pokemon_go_antibots_reverse_engineering/d636euc Thats is why looking at the .dalvik code etc. may be useful don't you think ? I may be wrong.

[justMaku]

It is for sure some kind of request signing, but digging through Dalvik won't help much because for 99,(9)% the signing happens in the native library which is C++.

[Saicheg]

@justMaku welcome from neighborhood Belarus :) isn't encryptiong related to https://github.com/FabianTerhorst/PokemonGo/commit/22ee9af7f40c9fa2bfe2b063e430e8be488f1584#diff-c4b70be9b0036965856ebbd284eae545R41

or you are saying they are using C++ compiled library somehow?

[exverse]

Is it only me that sees the Hash coming through the DownloadSettings function, seems it is different every time...

[trisk]

@d-pollard you want me to send the request modified and then replay the original request? Not sure what that would accomplish, since they're supposed to be unique and the server has already sent the response to the modified request.

[trisk]

@Saicheg the protocol implementation lives in the native libraries (some of which are converted from C# by Unity).

[justMaku]

@Saicheg that's the encryption for Pokemon Go Plus accessory communications and it has nothing to do with this protocol.

And regarding to C++ code, Pokemon Go was made using Unity framework. Games in Unity are written in .NET runtime compatible language which is compiled to CIL which then is transpiled to C++ and finally that code is compiled by platform specific compiler into code that's going to be executed on user's device.

This makes reverse engineering a terrible experience, because you have to deal with code that was transformed so many times it doesn't look anything like it did in the first place.

[d-pollard]

@trisk if you are able to re-play a request multiple times, then they didn't put a use constraint on it (we might not have to generate a new "auth" check every time)

[justMaku]

@trisk if you could try sending a request but instead of changing unknown6, change any other field that we know about (preferably something in the GetMapObjects). If the request fails it means that it's trully signed (as in change in data should require change in signature). If it still works then it means that Unknown6 is not a signature, but a message we just have to understand.

[d-pollard]

@justMaku put in way better words than I did lol

[Saicheg]

@justMaku got it, thanks for explanation, sounds definitely like terrible experience. what is the strategy here and how i can help you?

[d-pollard]

@Martyn-Cleroux - I've seen that posted about 1000x, please stop posting random stuff you found on the internet

[justMaku]

@Saicheg well unless you know arm assembly there's not much that can be done :/

[Saicheg]

@justMaku can't we get that decompiled to plain C and share with people to explore different pieces?

[pedbsktbll]

hey I just started following this thread--- mainly specialize in c/c++ and RE.. I'd be happy to look through some code or help in any way that I can? I see someone posted diffs from the decompiled app itself-- is it worth digging through that? Sorry, dont want to interrupt anyone's train-of-thought here

[iGio90]

@pedbsktbll what we are digging for is this unknown6 field to pass to the request. There is almost nothing on smali, need to check somewhere else

[justMaku]

@pedbsktbll you might want to take a look at libNianticLabsPlugin.so then

[shark0der]

@pedbsktbll you might want to join the discussion on discord https://discord.gg/AZs6T

[DanippDm5vzDS8HWnc9fap]

@justMaku @pedbsktbll i am differing the libNianticLabsPlugin.so; but not much knowledge on arm code

[d-pollard]

It might be also worth mentioning that the update has only affected the Pokemon encounter/catch and fort aspect of the game - I can pull inventory without any issue