msgpack?

[numinit]

Hey...

The server-client binary comms look like msgpack.

[numinit]

Or something close. I was looking at the cc1e stanzas there earlier, and noticed that cc was the msgpack tag for uint8.

[nelix]

I don't think its message pack, it it shares some traits. The top level protocol is [4 size] [1 type] {[size payload]} Some commands (command and connection_accept) have a plain text json blob within. map_update seems to be an actual image... I haven't managed to get much out of it yet though. data_update is a bunch of types delimited in the same way as the top level protocol except certain types are fixed length or null terminated (no idea why they didnt use json for this!)

You can checkout parser.js for more info on what we have worked out about the format.

[hellyhansen]

I've been trying to decipher the game to app protocol (message type 3) for a few days now and I think I might have got it. Not sure if it is a standard format, I was wondering if it was AMF0 or AMF3 to start with as the app appears to have Adobe influences (maybe AIR, note the references to .swf files for the quest and perk animations), but it doesn't seem to be.

It's some kind of remote object protocol, the game sends an initial message that sets up the whole structure and initial values and then later smaller messages can mutate that object in the app.

The structure is tag based with each tag starting with a 5 byte header composed of a 1 byte type and 4 byte key, followed by some data depending on the tag type

00 [4 key] [1 value] Set a boolean 03 [4 key] [4 value] Unsure. 04 [4 key] [4 value] Set a int, value is little endian 05 [4 key] [4 value] Set a float (uncomfirmed?), value is 4 bytes 06 [4 key] [string value] Set a NUL terminated string value. 07 [4 key] [2 count] count x [4 key] Sets an array count is little endian followed by count keys 08 [4 key] [2 count] count x [4 key] [string name] Sets an object. count is little endian followed by count key NUL terminated string pairs

You'll see things like:

00 11 22 33 44 00 04 11 22 33 45 09 00 00 00 08 11 22 33 46 02 00 11 22 33 44 F l a g 00 11 22 33 45 V a l u e 00

The stream above would set 11223344 to false, 11223345 to 9 and 11223344 to an object {Flag: false, Value: 9}

But you can think of the keys as pointers so if the game later sends:

04 11 22 33 45 03 00 00 00

It will update the 'Value' key in the object.

For example log the values sent back from the game as you move around. You'll see a set of 0x05 type messages, search for those keys from those messages in the initial large message and you'll find them in a 0x08 block attached to the keys 'X' 'Y' and 'Rotation'

This is just from interpretation of Wireshark logs so far, I've also been proxying but not yet tried implementing this as a parser.

[NimVek]

You can look at my investigation https://github.com/NimVek/pipboy/blob/master/PROTOCOL.md

[TriPhoenix]

Oh the similar minds :-) Did my own investigation as well and came up with https://github.com/rgbkrk/pipboylib/pull/7 @hellyhansen has basically the gist of it. I built a small, crappy decoder in python at https://gist.github.com/TriPhoenix/01e74f0055b072b615a4 which dumps out the complete object dictionary (but does not process updates for now) which may help further playing around with the protocol and its data.

Edit: Added a networked variant (same link, call python pip-decode-net.py 127.0.0.1 27000) to grab automatic updates from the server as well. Works great until you pick up new items which introduces yet-unknown objectIDs which is not really handled well by the current code. Enough to tinker with I guess and now its time to do a proper parser in pipboylib.

[nelix]

Nice work guys, PROTOCOL.md is great. When I am done working I'll send a PR to fill some gaps!

[kitten]

@TriPhoenix @NimVek I'm currently working on a simple interfacing library that is using pipboylib for discovery and then connects to the game directly, without the app, and these docs were very helpful! Thx a lot :) I've got the binary parser done already and will hopefully be able to push a repository today. Maybe this could help the parser of pipboylib?

[rgbkrk]

That could definitely help the parser of pipboylib. PRs are awesome.

[0xE1]

What about disassembling PipBoy app itself? It's on C#. (legal side of it, practical is easy as few clicks)

[kitten]

@rgbkrk It's purpose is not that of a relay, but to be a standalone communication library with the game, but the parser could maybe be adapted to be used in pipboylib as well?

https://github.com/philplckthun/codsworth

[rgbkrk]

omg the name

[rgbkrk]

Have you looked at the parser work by @AlexanderDzhoganov in https://github.com/rgbkrk/pipboylib/blob/master/lib/pipdecode.js and https://github.com/rgbkrk/pipboylib/blob/master/lib/pipdb.js ? We've now got several parsers in node and I'd love to unify them somehow, because the next cool stuff is built on top of this.

[kitten]

@rgbkrk The only problem I can see is the dictionary parsing. It's not using the uint16 on line 11, which probably indicates keys to be deleted from the dictionary (As seen in https://github.com/NimVek/pipboy/blob/master/PROTOCOL.md). So if that's really the case, the parser could fail to parse data that contains such removal keys, which I haven't yet observed though.

It's useless to merge some of my code though, as it's built on another stack, namely RxJS and ES6 code. But the parser code by @AlexanderDzhoganov looks good and shouldn't cause any problems

[AlexanderDzhoganov]

@philplckthun

It's not using the uint16 on line 11, which probably indicates keys to be deleted from the dictionary

This is some new discovery I was not aware of, will PR an update soon, thank you!

[NimVek]

Found it, when i closed the game an go back to menue.

[kitten]

@NimVek Makes sense as then we'd still be connected, but need to erase the data tree.

@AlexanderDzhoganov @rgbkrk @TriPhoenix I think a Slack channel or similar might make sense unless we want to pollute pipboylib's issues with discussions and conversations :)

[rgbkrk]

On that same note, I created an org called vaultjs if we end up wanting to consolidate any repos there.

[rgbkrk]

@mattbaker created a slack channel, but I don't have a way to add people apparently.

By the way, would have made that org be pipboy or vault, but those were definitely taken.

[TriPhoenix]

Yeah a slack channel or something similar would probably be best, this poor issue has already taken way more non-msgpack discussion than it should ;-)

[rgbkrk]

Created a gitter room for this repo. Not too much worry about noise here just yet.

Closing the issue since the format has been deciphered.