SysNAND Virtual: ByteSwap EEPROM type agb saves

[cheatfreak47]

For some reason EEPROM8K (512b) and EEPROM64K (8kb) saves are byte swapped in hardware, so byteswap them back in input and output for gbavc.sav to avoid potential issues with restore / backup of saves.

To tell just check the size in the agbsave header, if it's 8kb or 512b it's EEPROM and needs swapped.

example:

Typical Emulator/Cartridge Save Dump:
0D 63 02 65 45 41 4D 41 
00 00 00 00 00 69 00 8E 
00 01 19 FE 00 01 9D 9C

AGBFIRM save:
41 4D 41 45 65 02 63 0D 
8E 00 69 00 00 00 00 00 
9C 9D 01 00 FE 19 01 00

[cheatfreak47]

Another thing, it'd be nice if you could have the option to inject clipboard data into gbavc.sav instead of having to make absolute sure the file name copied is "gbavc.sav" each time before pasting it into sysnand virtual.

This way people could just dump some emulator saves or raw dumps from real gba carts on the SD and inject them without having to sit there and manually rename them with the slow rename UI, or pull the SD repeatedly to rename them, or use ftpd to rename them, etc.

If it was possible to inject clipboard data, this would also negate the need for manually trimming improperly saved gba save files, which happens in emulators sometimes- with the save being at the beginning of the file, and the rest being padded out with FF.

[d0k3]

A quick, clumsy workaround (i.e. byteswapping the virtual file) on GM9 side is not the right course of action here and will do more bad than good in the long run. There is also a copy of the AGBSAVE on SD, which I plan to support soon (CMAC still has to be figured out), and which would require figuring out a workaround over again.

I suggest you instead support these two issues with additional info: https://github.com/visualboyadvance-m/visualboyadvance-m/issues/150 https://github.com/mgba-emu/mgba/issues/878

Also, the gbavc.sav does not have to have the right name. GM9 will suggest the correct location if the size matches. As for the trimming issue - yes, that's a pita. Take note that my feature requests would actually also fix that issue.

[cheatfreak47]

Are you positive that EEPROM type chips in real GBA games don't store the data byteswapped already? I highly doubt all emulators are deliberately byte swapping just one save type like this for no reason, I've dumped saves off of real carts using various tools and all of the EEPROM ones came out byte swapped, but all the rest didn't. (Unless all GBA dumping tools are also deliberately byte swapping just one save type too???)

In addition, I don't think that it would ever lead to more confusion or do any bad, since AGBFIRM is the only thing in the world dealing with manually un-byteswapped EEPROM type GBA save files, the byte swapping being done on your end would lead to nobody ever ending up with unswapped EEPROM saves to begin with (negating the need for tools to reswap the bytes on a PC, or changes to every emulator, etc.)

This is an idea created out of convenience. Granted, you would have to make it do the same thing later when implementing support for the SD GBA save, but that's kind of a smaller concern for now. Right now people may try to inject the wrong format EEPROM saves, and that'll cause the save to be wiped and ignored when booting the game in AGBFIRM, and with no feedback or info as to why. (Unless you do some digging and figure all this crap out like I did)

If you think you can do a better job with it, by all means, but a simple byte swap on your end would be a lot easier for everyone, not needing to convert existing saves to some new format before injection just adds another step for everyone, and for no tangible purpose either.

There was a discussion on the Nintendo Homebrew discord yesterday in #dev, which lead to the creation of this issue, feel free to go read through it if you'd like.

[d0k3]

From a chat with @AuroraWright yesterday (I hope it's okay to quote you, Aurora):

I just talked with the mGBA dev they said that it's that way because no one really knew how the 8 bytes are stored on the hardware and now they can't change it because it'd break existing saves so rip

Also, no offense meant, but you're underestimating the effort required for proper random byteswapped access - this is not a 10 minute job by far. For SD AGBSAVE access, we'd have to change the whole thing (I'd have to provide AGBSAVE mounting), in turn breaking basically all existing tutorials on it. That all doesn't even take into account the other inconvenience issues you brought up and the fact that at some point someone may have legit complaints about the byteswapping (f.e. when writing to carts). So... my philsophy here is, do it once, do it proper.

Best outcome here would be, the respective authors support the AGBSAVE save format natively, you'd just copy the whole partition and provide that to the emulator as save file. With the AGBSAVE format, the byte ordering is 100% clear, and an AGBSAVE file always has the correct size. Sure, that would also require a change in existing tuts, but only a small one.

[cheatfreak47]

Alrighty, as long as a convenient solution comes about in the end, I'm cool with it, I just figured this was the least effort solution, but of course least effort isn't always the best.

[d0k3]

Let's see what the respective authors have to say. We may not come up with a quick solution, but with one that works in the long run.

[WinterMute]

Some quick research shows that no$gba stores saves in the correct (unbyteswapped )order but EZF IV has byteswapped saves.

I don't really understand how something like this happens tbh.

Interesting historical info - http://zork.net/~st/jottings/GBA_saves.html

[endrift]

The EEPROM saves are read and written 8 bytes at a time. You can't address them more finely. What most emulators do is say "ok address x is actually address 8x, now let's write out the 8 bytes we get in the order we get them". Turns out that puts them in the file backwards*, but VBA people didn't notice. It's been a few years but I think I intentionally did this in mGBA to keep compat with VBA saves, although I did notice it was backwards.

[d0k3]

Interestingly, the VBA people haven't commented on this yet :). Anyways, I'm still unsure as to how to handle this from GM9 side. I know, there is no way to know about the byte ordering of a GBA savegame by just looking at it, so mGBA can't take GM9 extracted eeprom saves and vice versa.

Changing that behaviour in GM9, and doing it proper would require larger changes to how GM9 handles this, in turn breaking all tutorials on GBA VC save transfers. Oh, and did I already mention that, due to the way the console handles these transfers, GBA VC save transfers have always been sort of a pita?

I'd also strongly assume that this issue will also make a comeback once Switch GBA VC savegame transfers are a a thing.

Anyways, @endrift - you have my full support in getting imports / exports of the AGBSAVE format working.

[cheatfreak47]

@d0k3 this might be a surprise to you, but it turns out that there might not be any way to have it so all EEPROM save games are "correct" byte order no matter what.

Something I came across earlier is that some games seem to be purposely swapping the bytes on write to the save file, and others do not. The first few human readable bytes of Super Mario Advance are "EAMA" (meaning, are swapped) on saves dumped from a real cartridge, but some other games, such as The Legend of Zelda A Link to the Past & Four Swords, are not swapped when dumped from the cartridge, I noticed this myself looking through all the dumps of cartridges I made earlier. Both of these games are EEPROM type.

Because of this, AGBSAVE for ZELDA, the first few human readable byte is swapped to ADLEZ on 3DS and Super Mario Advance is not, with it being "AMAE".

There's no winning with this one. Because of that, it may actually be worth just implementing the byteswap in GM9 after all, but that's just my opinion of the situation of course.

What I'm not sure of is weather it's Zelda or Super Mario doing the swapping on save output, but it's gotta be one of them doing it. Which one doing it is what you could use to determine the "correct" save format as.

What I'll probably do next is look at all my EEPROM based GBA games, and dump any ones I skipped before to see what the general trend is for how games handle saves (provided the files have anything I can identify)

[d0k3]

The changes required for a byteswap inside GM9, if done proper, will break all existing tutorials (in addition to the implementation not being exactly trivial). Once we got the SD AGBSAVE working, we may break stuff again. Then, once GBA VC save transfers on Switch will become a thing (I'm pretty sure this will work similarily), the next problem will pop up. Whatever solution we come up with, it's better to do it once and do it right and not make this even more of a clusterfuck than it is now.

Also, yeah, fucking great. Not even Nintendo seems to know how to handle that clusterfuck. Also, thanks for this helpful info!

[cheatfreak47]

@d0k3 Well here's some more helpful info: Nintendo's own GBA VC on Wii U Seems to save in the reverse format that our typical emulators today use as well.

Proof is here in this screenshot This is the save data for Super Mario Advance on Wii U VC. data_008_0000.zip, the savedata seems to be located at offset 0x4080.

I don't know how this changes things but, even the officially licensed emulator for GBA is handling saves like this, so I guess that's probably worth something.

The way I'd still recommend implementing this is just to swap the bytes for the file if it's 512b and 8Kb respectively on the fly when putting the gbavc.sav file in SysNAND Virtual, and just as well, when people copy files to it, swapping the bytes when writing it into the agbsave.bin.

Bear in mind I have no idea how GodMode9 is currently doing this, and I'm not a programmer, so I have no idea what the ramifications of implementing something like this are.

From the sound of things the way you want to do this is to instead begin allowing people to mount the agbsave.bin instead of directly dumping the file in S: which is fine I guess, as long as it's comparably easy to do.

[d0k3]

Well... you're right, implementation wise, stuff is not as easy as it may sound. Also, we can break all tutorials in existance, but we should do so only once (not twice, when we got the SD CMAC thingy figured out). Anyways, can you, maybe confirm or reasearch a bit of stuff? There's some questions that I'm still unsure of.

1. Is 128kB the actual upper limit? From the AGBSAVE header, the upper limit would be roughly 4GB, but we all know this will not happen. There may be also a technical limit from the GBA game side. How does the limit relate to what you get out of emulators? Do emulators write smaller or bigger files? Maybe @AuroraWright can shed some light on this.
2. Are we 110% positive the size is enough to determine if the save chip is an EEPROM or not? If so, can you provide some kind of additional info that would prove this? (maybe also @endrift can shed some light here).
3. This one will be very difficult. We know GBA savegames don't have a universal identifying header or magic number (only AGBSAVE has this) and are thus not easily recognizable, but you apparently used some kind of magic number to check if byteswapped or not. So, what I wonder about right now is, do savegames at least have a unique identifying header? Meaning, do we have some means to easily decide a savegame belongs to one game, given we already have another savegame from that game?
4. And another difficult one - there are two mistery dwords in AGBSAVE, at offsets 0x60 and 0x64. How are they related to the save? Maybe they have something to do with flipping occuring or not?

[cheatfreak47]

@d0k3

1. Well, I'm not sure that it is a "limit" so much as none of the existing games on the market ever had a save with higher than 1mbit of FLASH type memory (or 128 kilobyte) storage. The only games I know of personally that use this are the Pokemon titles, and Super Mario Advance 4, which used it because it needed storage for the e-Reader level cards. The chips have a fixed amount of storage and all of said storage is allocated for the save data- how the game uses that storage is per-game.

2. Definitely, as far as storage goes, there's only 5 main types, each with their own distinctive sizes. The only ones here with the weird byte swapping business going on is EEPROM, though I will verify this myself with some additional testing- which I'll post when I get the chance. Some of these individually have variation- like chip manufacturer and such, but as far as I can tell, EEPROM actually has no such variance, except for what appears to be an unknown one, that no commercial games appear to use- I did actually check that too, games won't work with it or will not save with it.

   • EEPROM with 512 bytes
   • EEPROM with 8 kilobytes
   • SRAM/FRAM (either 32 kilobytes of “static RAM” with a long-life battery attached, or 32 kilobytes of ferro-electric RAM)
   • Flash RAM with 64 kilobytes
   • Flash RAM with 128 kilobytes

3. Actually the way I was checking this is probably rather silly, but essentially, by wiping out the save data on various EEPROM games on both an emulator (and in one case, real hardware) and on a 3DS, I was able to generate identical save files for the same game on both, and look at both files with a hex editor, and notice the difference this way, just by looking at the bytes. Games all handle this differently, and most of the time have no way to identify whether or not they are swapped or not, but, my findings were that agbsave is always like this with EEPROM saves, it's not selective. This is likely due to how the EEPROM saves work. As for the question about telling what save belongs to what game- on occasion, game saves have a identifying string in them, but it's never consistent where in the file from game to game, so it's not something that can be machine processed, the only real "user error prevention" possible here is making sure the save being injected is the same size at least but that's probably not necessary. Chances are, no user will try injecting a Mario Advance 2 Save into Zelda unless they aren't particularly smart to begin with.

4. No clue on that one, perhaps @AuroraWright might know or have some ideas?

---

One other thing that could be improved as I mentioned in my second post on this issue- is making it easier to inject GBA save files generally. As of now, the way you have to do it is to rename the save you want to inject as "gbavc.sav" and copy/paste it into S:/ -- which is functional, but the thing is, this makes it considerably more work/setup to do multiple jobs, or work with more than one save file for different games on the SD card- or even multiple saves from the same game.

It'd be wise to make sure the user can have the save filenames on the SD be whatever- and inject them from the clipboard regardless of the filename.

[WinterMute]

And another difficult one - there are two mistery dwords in AGBSAVE, at offsets 0x60 and 0x64. How are they related to the save? Maybe they have something to do with flipping occuring or not?

From that link I found earlier - http://zork.net/~st/jottings/GBA_saves.html

At start up, the code in the SDK would talk to the Flash chip and say “what’s your manufacturer code?” and use the protocol corresponding to that manufacturer.

So my theory would be that arm7_save_cfg_t probably contains a response to that request or something that tells the firmware how to respond.

[d0k3]

Thanks @cheatfreak47 and @WinterMute - this does sound reasonable.

Now, one unanswered question - how does actual save size relate to what emulators give you? Is the savegame from an emulator typically bigger? Fixed size?

[WinterMute]

@d0k3, heh some emulators detect save size better than others. Most emulator savefiles are typically bigger

[cheatfreak47]

@WinterMute Is that actually true? I guess it probably is with all the older crappy versions of VBA floating around. :V

All the games I've tried seem to have 1:1 exact sized stuff for the most part- at least on any modern emulator, like VBA-M and mGBA. (though up until recently, mGBA was giving the wrong size for EEPROM8K (512b), which I reported and endrift subsequently fixed.)

Regardless, this would would work well with my idea of injecting from clipboard as any emulator creating files larger than the actual file always just pad out the extra space with 00s or something at the end of the file, meaning that if you injected a incorrectly sized file into the correctly sized one, only the good part of the file would be injected anyway.

[WinterMute]

@cheatfreak47 yeah, auto-detect stuff got a lot better but I think people still use some old VBA builds (but m'uh accuracy) which err on the side of caution, It's not a big deal really, just a matter of ignoring stuff outside the size - emulated games won't use the extra space.

[dok3]

Sorry, some body include me in this thread by an error i'm @dok3 not @d0ke :)

But im seeing that you are working on game emulation?

I was talking a week a go with a Designer about to construct a product and a network like PSN but for game emulation.

What do you think about this idea?

Too crazy ? or existent idea ?

[d0k3]

Oh, some other guy chiming in! This is not about emulation, @dok3, it's about a clusterfuck someone (none of us) was responsible for years ago. Your idea sounds good, though, but pls no derailing my issues section.

Anyways (sigh)... @cheatfreak47 can you please thoroughly test this build and tell me if it does - not only for eeprom saves - exactly what you'd expect? https://transfer.sh/QcsSc/GodMode9-v1.4.0-20-g5beab1c1-20170920214115.zip

Also, all tutorials in existance broken, but w/e.

[cheatfreak47]

@d0k3 sure can, working on that now, so far the dumped saves match up well to what I would expect for each game- next I'll try injecting some EEPROM saves for various games and see how that works, and edit this post with some examples of the games I tested.

All Saves Injected were dumped with GBA Backup Tool w/ a real DS Lite and GBA Game

• Super Mario Advance [EEPROM 512b] - Dump: Works | Inject: Works
• The Legend of Zelda - A Link to the Past & Four Swords [EEPROM 8kb] - Dump: Works | Inject: Works
• Banjo Kazooie: Grunty's Revenge [EEPROM 8kb] - Dump: Works | Inject: Works
• WarioWare Inc. Mega Microgame$ [SRAM 32kb] - Dump: Works | Inject: Works
• Sonic Advance 2 [FLASH 64kb] - Dump: Works | Inject: Works
• Super Mario Advance 4: Super Mario Bros 3 [FLASH 128kb] - Dump: Works | Inject: Works

Conclusion: Everything is working that matters.

Injecting Larger Save than AGBSAVE has avaliable: Works

• situation for this happening: emulators producing oversized saves

Injecting Smaller Save than AGBSAVE has available: Fails

• situation for this happening: injecting a smaller eeprom emu save into an SRAM patched GBA title, people shouldn't be using SRAM patched titles anymore, but people are lazy- fix: pad out remaining space with FF bytes
• situation 2 for this happening: Pokemon maybe, no idea if this will work, but before, pokemon wouldn't work due to FLASH128K being unsupported, now that it is supported maaaaybe people will be able to migrate to the fixed FLASH128K GBA titles by exporting the save, and importing the old 64KB file into the 128KB new one, and allowing the game to repair the save, since half of the Pokemon Saves was used as a backup anyway. The solution would be to just again, fill missing space with FF bytes and hope the game can repair the save from there?