Implement Game Sync

[mm201]

@polaris- already did some amazing work reverse engineering this; I should be able to do the rest myself.

Note that I have no plans of replicating the original dream world, since that would be a major violation of copyright.

[ghost]

what's news ? and for fixing the Matchmaking of gen5 random matchups ?

[mm201]

I need lots of free time to do game Sync. Random matchup is an altwfc issue. Send them your packets.

https://github.com/polaris-/dwc_network_server_emulator/issues/162

[ghost]

Random bump; @mm201 how is the stage of the framework as a whole?

[mm201]

It'll be done when it's ready.

[dmoeenm]

What are you planning to do with the whole dream world/ global link stuff then i know you won't replicate it but what are you going to use the gamesync for and what will happen to all the other functions of gamesync will you work with other's to create a way to grow berries and maybe get dream world event pokemon or something like that and do you have any plans for the other global link stuff

[the4anoni]

any news ?

[ghost]

Well ofc not. If there was any news, it'd have been posted :wink:

[the4anoni]

How can i help with this ? (im not dev ,dont have any captured packages)

[ketsuban]

Could you write up what you do know about the Game Sync protocol? The lay writeups focus on the Dream World (not unreasonable, but unhelpful for our purposes) and I'd be interested to know what uses it can be put to.

[mm201]

First, the game checks the account status. If it's able to put a pokemon to bed, it first downloads a banlist, then lets you pick a pokemon, then puts it to bed. The put-to-bed request actually uploads the entire savefile to the server. Another possible account status is that the pokemon is dreaming or that it's able to wake up. If you wake it up, the server sends a payload which contains any pokemon or items you brought back with you. Communications happen over https. It's not a gamestats server.

There should be other stuff in this payload I haven't had a chance to look at yet, such as trainer's pokemon who visited you and stuff for BW2 join avenue.

The neatest thing is that the entire savefile is sent, so this can be used for stuff like bulk IV checking. You should also be able to inject pokemon, which will then appear in the forest to be caught with dream balls, or items.

I can't give you detailed specifics, since I haven't figured them out yet and plan on figuring them out by actually implementing it. What I can provide is @polaris- 's toy Game Sync implementation which contains some valuable research he did.

[ketsuban]

Wow, that's a lot more exploitable than what I was expecting. Thanks!

[ghost]

How is the project going?

[zurgeg]

Yo, incase you need any help, I've been doing some research relating to Game Sync. I've built a simple Python server to run GS but that's besides the point. Currently I've found that when a Pokemon isn't currently sleeping in that save, returning anything relating to waking up a Pokemon will tell you that "Game Sync needs to recharge"

[zurgeg]

Alright, more research, the form data (I think?) when you respond with \x08 to playstatus, then with \x00\x00\x00\x00 to account.create.upload, is actually just your raw save data

[mm201]

@zurgeg I hear you're doing some good research! savedata.download is the current snag. Rating battle, i.e. worldbattle download/upload would also be great to have and we should be able to run rating battles without any copyright issue.

Can I contact you on Discord?

[zurgeg]

@zurgeg I hear you're doing some good research! savedata.download is the current snag. Rating battle, i.e. worldbattle download/upload would also be great to have and we should be able to run rating battles without any copyright issue.

Can I contact you on Discord?

Yeah, but I prefer GitHub issues for this kinda thing ;)

[zurgeg]

@mm201 I figured out some likely key info on savedata.download

During savedata.download, it runs the following math function 10 times, increasing x each time: f[x] = (x * 0x08) + 0x04, each time it runs that function, it checks the 2 bytes at that location in the response, if those are \x00\x00 then break the loop, otherwise if d <= 0x1ed where D is the data just pulled, then do something(!)

I believe that d is likely pointing to something in the response, from what I saw on polaris-'s gist. I can't seem to find much else at the address (21D7E78) polaris mentioned in the gist.

[mm201]

0x1ed is Arceus's Nat Dex number which is ... interesting. It looks pretty clearly like a null-terminated list of up to 10 whatsits involving Pokémon. Possibly the list of Pokémon to send to the Entree? 8 bytes would give us 4 shorts for species, 2 moves, and maybe some additional flags like gender? But I'm not sure why it would care whether the Pokémon is a Unova native or not. 0x289 seems like a more reasonable number to test.

[zurgeg]

0x1ed is Arceus's Nat Dex number which is ... interesting. It looks pretty clearly like a null-terminated list of up to 10 whatsits involving Pokémon. Possibly the list of Pokémon to send to the Entree? 8 bytes would give us 4 shorts for species, 2 moves, and maybe some additional flags like gender? But I'm not sure why it would care whether the Pokémon is a Unova native or not. 0x289 seems like a more reasonable number to test.

Odd, afaik Arceus isn't the last guy in the natdex (at least in gen 5). Additionally, PKHeX says that Entree pokemon can only have 1 move? Maybe the additional flags are actually replacing the PID? I know Entree pokemon don't have a PID. EDIT: I did just realize that Arceus is the last guy in the Gen4 dex however.

[426C7565]

Heyo, someone else who would be interested in helping out (but not sure how to) here! Took a look at everything mentioned in here so far, as well as the gist shared, and wanted to share my thoughts on it. Please note that I have never played PDW or used the Global Link extensively while it was around, so I might be completely wrong here.

Assuming that the gist was based off of the first set of gen V games (this would somewhat explain why the second loop uses the index of the first BW2 exclusive item), it would make sense for the game to check for any non-Unova Pokémon, as they aren't part of the regional dex. My guess is that the game performs an additional check on non-Unova Pokémon to see whether or not the player has obtained the national dex mode in-game, and if not, makes the Pokémon invisible in the Entree Forest (this is also an option in PKHeX) until they do.

As for the data itself, my guess is that it's laid out like this:

0x00 - 2 bytes - Internal Pokémon ID
0x02 - 2 bytes - First move ID
0x04 - 4 bytes - (Partial) personality value

The DW Pokémon have a set gender, as well as a guaranteed HA (if the Pokémon has one) and cannot be shiny from what I've read, so for these elements, a (partial) personality value would need to be created on the server's side before the transfer occurs (probably within the DW game when the Pokémon was encountered).

Hope any of this helps!

[zurgeg]

Heyo, someone else who would be interested in helping out (but not sure how to) here! Took a look at everything mentioned in here so far, as well as the gist shared, and wanted to share my thoughts on it. Please note that I have never played PDW or used the Global Link extensively while it was around, so I might be completely wrong here.

Assuming that the gist was based off of the first set of gen V games (this would somewhat explain why the second loop uses the index of the first BW2 exclusive item), it would make sense for the game to check for any non-Unova Pokémon, as they aren't part of the regional dex. My guess is that the game performs an additional check on non-Unova Pokémon to see whether or not the player has obtained the national dex mode in-game, and if not, makes the Pokémon invisible in the Entree Forest (this is also an option in PKHeX) until they do.

As for the data itself, my guess is that it's laid out like this:

0x00 - 2 bytes - Internal Pokémon ID
0x02 - 2 bytes - First move ID
0x04 - 4 bytes - (Partial) personality value

The DW Pokémon have a set gender, as well as a guaranteed HA (if the Pokémon has one) and cannot be shiny from what I've read, so for these elements, a (partial) personality value would need to be created on the server's side before the transfer occurs (probably within the DW game when the Pokémon was encountered).

Hope any of this helps!

Ah. Perfect! I used up my GS energy yesterday so I should be able to give this a test. I'll edit this once I do. EDIT: Nope, didn't work. I can't think of anything else that could go on...

[426C7565]

Does the problem originate from the Pokémon data parsing specifically, or can it be any of the other sections as well? And is what is described in the gist all that is sent, or is there more data (namely PGL stuff) that is sent in the same response? Supposedly there should be more data that the system receives once a Pokémon wakes up, but I can't find any mention of anything unrelated to the DW apart from the battle competition.

Also, would it be possible to share some snippets of the disassembled code that parses the response, or does that go against any rules/guidelines (not familiar with the legal side of these things)? I don't have any experience in the field reverse-engineering, but would like to take a look at it if possible.

[zurgeg]

Does the problem originate from the Pokémon data parsing specifically, or can it be any of the other sections as well? And is what is described in the gist all that is sent, or is there more data (namely PGL stuff) that is sent in the same response? Supposedly there should be more data that the system receives once a Pokémon wakes up, but I can't find any mention of anything unrelated to the DW apart from the battle competition.

Also, would it be possible to share some snippets of the disassembled code that parses the response, or does that go against any rules/guidelines (not familiar with the legal side of these things)? I don't have any experience in the field reverse-engineering, but would like to take a look at it if possible.

Good question! I haven't taken a look in a debugger since I can't get it to work under melonDS, but I will once I get back to you

[zurgeg]

Woah! @426C7565 Thank you soooo much! It actually worked! Only problem is... when entering 3rd right (the entralink area where it is), the game crashes! Must be an issue with my code

[426C7565]

That's amazing news, congrats! I'm assuming you mean the Entree forest? If so, it might be corrupted Pokémon data. Maybe try and see if manipulating the Pokémon in a save/hex editor (editing it's personality data, replacing/removing it) allows you to enter the zone

[zurgeg]

That's amazing news, congrats! I'm assuming you mean the Entree forest? If so, it might be corrupted Pokémon data. Maybe try and see if manipulating the Pokémon in a save/hex editor (editing it's personality data, replacing/removing it) allows you to enter the zone

Oddly enough, I went there in PKHeX, nothing in the 3rd right?

[426C7565]

Hm, what about opening up the save file in a hex editor? The offset for DW Pokémon should be 0x22C00 (as shown on ProjectPokémon). Maybe there's some corrupted data somewhere in that segment and PKHeX just skips over it entirely because of it

[zurgeg]

Hm, what about opening up the save file in a hex editor? The offset for DW Pokémon should be 0x22C00 (as shown on ProjectPokémon). Maybe there's some corrupted data somewhere in that segment and PKHeX just skips over it entirely because of it

Turns out that data was actually item related :P

[426C7565]

Turns out that data was actually item related :P

Which data are you referring to here? I'm not sure I completely follow anymore... If you were referring to the data block I mentioned in my previous comment, looking at the way PKHeX parses it confirms that it consists of 530 4-byte long Pokémon entries, followed by some flags and an encryption seed. If not, what exactly is it that you were referring to?

[zurgeg]

Turns out that data was actually item related :P

Which data are you referring to here? I'm not sure I completely follow anymore... If you were referring to the data block I mentioned in my previous comment, looking at the way PKHeX parses it confirms that it consists of 530 4-byte long Pokémon entries, followed by some flags and an encryption seed. If not, what exactly is it that you were referring to?

Exactly. The data block you mentioned actually had to do with items. It seems we can send up to 10 Pokemon and 10 items. The Pokemon block I haven't figured out yet.

[426C7565]

That's confusing to say the least. Looking at some old footage of the DW, the response should contain a maximum of 10 Pokémon and 20 different items. Judging from what has been written in the gist, the first section should contain the Pokémon, as it loops through said section 10 times. While I am not sure why, the way the Pokémon data in the response is structured likely does not match the way it is in the save file (the size alone is a clear indicator of this).

The items shouldn't be anything more than the internal ID and amount (so no more than 4 bytes for each entry), and the size of the supposed item data block in the response supports this (excluding the "header" and list terminator, you are left with 80 bytes, which the game loops through 20 times, 4 bytes for each item). On top of that, items should not appear in the Entree Forest, so it is unlikely anything put in this section would actually affect the Entree Forest at all.

That said, there might be things you found that I don't know about. Perhaps you could share what you found so far? It would also be helpful if you could share the save file you received the Pokémon on so others could take a look at it.

[zurgeg]

That's confusing to say the least. Looking at some old footage of the DW, the response should contain a maximum of 10 Pokémon and 20 different items. Judging from what has been written in the gist, the first section should contain the Pokémon, as it loops through said section 10 times. While I am not sure why, the way the Pokémon data in the response is structured likely does not match the way it is in the save file (the size alone is a clear indicator of this).

The items shouldn't be anything more than the internal ID and amount (so no more than 4 bytes for each entry), and the size of the supposed item data block in the response supports this (excluding the "header" and list terminator, you are left with 80 bytes, which the game loops through 20 times, 4 bytes for each item). On top of that, items should not appear in the Entree Forest, so it is unlikely anything put in this section would actually affect the Entree Forest at all.

That said, there might be things you found that I don't know about. Perhaps you could share what you found so far? It would also be helpful if you could share the save file you received the Pokémon on so others could take a look at it.

Sure, give me a minute to get it "minimized" (ie., put it into a state viable for reproduction) Additionally, 2 items can be held within each 8-byte item "block", since 2 * 10 is 20 (duh), 20 items can be sent in that response

[426C7565]

That is indeed true, although what would be the logic behind processing two items at once instead of a single item at a time, or using two loop iterations to process a single Pokémon? Besides that, the numbers that the first two bytes in each Pokémon/item entry gets compared to would also be illogical (why check for 0x272 in the Pokémon loop when Genesect would be 0x289?).

[zurgeg]

That is indeed true, although what would be the logic behind processing two items at once instead of a single item at a time, or using two loop iterations to process a single Pokémon? Besides that, the numbers that the first two bytes in each Pokémon/item entry gets compared to would also be illogical (why check for 0x272 in the Pokémon loop when Genesect would be 0x289?).

Well, if you look at this, 0x272 is the ID of the first B2W2 exclusive item, which is why it checks for that.

[zurgeg]

Turns out my Entree had been corrupted the whole time, made a new save, I'll try there

[zurgeg]

@mm201 So, me and my friend were joking around, and I suggested the answer lied in dn = resp[ym]

Turns out that happened to be the answer to pokemon downloading. He suggested that it was probably \x64\x65\x65\x7a\x6e\x75\x74\x73. That turned out to turn the key, and unleash a large amount of game crashing Blazikens into my entralink. Encountering any of them will crash your game, but this is progress. The related code is https://github.com/NDSLink/dream-server/blob/master/routes.py#L154-L157

[zurgeg]

Crashing issue with the Blazikens is fixed now.

[zurgeg]

0x1ed is Arceus's Nat Dex number which is ... interesting. It looks pretty clearly like a null-terminated list of up to 10 whatsits involving Pokémon. Possibly the list of Pokémon to send to the Entree? 8 bytes would give us 4 shorts for species, 2 moves, and maybe some additional flags like gender? But I'm not sure why it would care whether the Pokémon is a Unova native or not. 0x289 seems like a more reasonable number to test.

FYI, the answer to why it would care is that there are only a couple Unova pokemon with overworld sprites (Virizon, Reshiram, Zekrom, just to name a few). We also figured out that it comes out to 2-bytes species, 2-bytes move, 1-byte gender, and then some other things we haven't figured out.

[zurgeg]

Heyo, someone else who would be interested in helping out (but not sure how to) here! Took a look at everything mentioned in here so far, as well as the gist shared, and wanted to share my thoughts on it. Please note that I have never played PDW or used the Global Link extensively while it was around, so I might be completely wrong here.

Assuming that the gist was based off of the first set of gen V games (this would somewhat explain why the second loop uses the index of the first BW2 exclusive item), it would make sense for the game to check for any non-Unova Pokémon, as they aren't part of the regional dex. My guess is that the game performs an additional check on non-Unova Pokémon to see whether or not the player has obtained the national dex mode in-game, and if not, makes the Pokémon invisible in the Entree Forest (this is also an option in PKHeX) until they do.

As for the data itself, my guess is that it's laid out like this:

0x00 - 2 bytes - Internal Pokémon ID
0x02 - 2 bytes - First move ID
0x04 - 4 bytes - (Partial) personality value

The DW Pokémon have a set gender, as well as a guaranteed HA (if the Pokémon has one) and cannot be shiny from what I've read, so for these elements, a (partial) personality value would need to be created on the server's side before the transfer occurs (probably within the DW game when the Pokémon was encountered).

Hope any of this helps!

Well, that helped quite a bit given you were actually right about the first 2 values. What a coincidence that a random guess would've gotten us this far!

[zurgeg]

Alright, so I made a purely random guess about worldbattle.download (just bounced back my savedata) and yeah it worked? Problem is, it is connecting to syachi2ds.secure.sake.gs.nintendowifi.net. On top of that, not only is it connecting to a server out of my control (Wiimmfi), the error code it spits out (40755) is unknown...

[zurgeg]

I was mistaken. I coded a basic SAKE server (bounces back a simple XML upon a download request) and now we are connecting to dls1.ilostmymind.xyz. Unfortunately, that seems to be a NWFC server controlled by Wiimmfi, which is (actually this time) out of my control.

[zurgeg]

We did it! Also customizations can download now

[DrRubix1712]

Is there a way for me to get involved with this project as far as testing the sending and receiving of save data or anything to do with testing?

[zurgeg]

Is there a way for me to get involved with this project as far as testing the sending and receiving of save data or anything to do with testing?

We haven't quite gotten anything up and running for a production ready build, but maybe soon!

[Cowboyjunkie]

Is there a way for me to get involved with this project as far as testing the sending and receiving of save data or anything to do with testing?

We haven't quite gotten anything up and running for a production ready build, but maybe soon!

Hi! Anything one can do to help? I have my Pkmn-Classic Server on PiOS just up and running and I am already in the process of debugging and analyzing various parts of it. Regards, John

[zurgeg]

Is there a way for me to get involved with this project as far as testing the sending and receiving of save data or anything to do with testing?

We haven't quite gotten anything up and running for a production ready build, but maybe soon!

Hi! Anything one can do to help? I have my Pkmn-Classic Server on PiOS just up and running and I am already in the process of debugging and analyzing various parts of it. Regards, John

Not really. We don't actually have anything set up for front-end stuff, so basically all that you can do is download a bunch of Blaziken ;)

[mm201]

@Cowboyjunkie If you're feeling adventurous, you could drop some replay responses into pgl.ashx, especially for savedata.download. There's some discussion in this thread about them. (Just remove the 502 response to get rid of the maintenance error.) You should make sure you have a savefile backup, because wrong responses can mess up your Entralink.

Other than that, just have a bit of patience while I finish up GenIV and move onto this.

[zurgeg]

@Cowboyjunkie If you're feeling adventurous, you could drop some replay responses into pgl.ashx, especially for savedata.download. There's some discussion in this thread about them. (Just remove the 502 response to get rid of the maintenance error.) You should make sure you have a savefile backup, because wrong responses can mess up your Entralink.

Other than that, just have a bit of patience while I finish up GenIV and move onto this.

Busted Entralink was an issue with the save I was using fyi. But attempting to load in ----- (or really anything with no overworld sprite) would probably crash.

[qfoxb]

Any updates in the past few months?

[kuroppoi]

Has there been any cool & notable progress that I missed out on? I came across this a while back and it piqued my interest, so I decided to have a go at figuring this out for fun and experience. It seems that the hardest part has already been done, so admittedly all I have really done so far was write a local DNS server and some very simple server emulators based on the groundwork that has been laid down until now & messed with some of the values.

I'm currently at the point where I am able to tuck in and wake up a Pokémon, as well as catch downloaded Pokémon in the Entralink/Dream World. (Although the provided data is far from complete.)

I'm currently not very experienced when it comes to reverse engineering DS games (couldn't for the life of me figure out how to debug them in realtime..), but I'd love to share any of my findings with you, should you find them interesting!