search

d0k3 / GodMode9

GodMode9 Explorer - A full access file browser for the Nintendo 3DS console :godmode:
GNU General Public License v3.0
2.12k stars 191 forks source link

Handling GBA VC .sav CMAC #494

Closed TurdPooCharger closed 4 years ago

TurdPooCharger commented 5 years ago

Continuing what was left off from,

this should solve transferring the entirety of the Nintendo 3DS folder, minus games that have saves with Secure Value (and saves made outside that folder).

Edit - Forgot to add the 'ARM7 registers' field. GBA VC diagram

GBA VC cmac calculation

I have more findings to share later when I get the chance to. Check back if you're interested in direct extraction and injection of GBA saves without the involvement of agbsave.bin.

TurdPooCharger commented 5 years ago

@BernardoGiordano / FlagBrew team / Checkpoint [Checkpoint] GBA game save support. #22

@J-D-K (JK) / JKSM

The .gba rom for a GBA Virtual Console title can be found at:

The .code consists of the gba rom with a .CAA footer. This footer is 864 (0x360) bytes.

CAA footer

The Game Boy Advance has fifteen (15) different save types.

Here are some conversion factors for their sizes.

You might have noticed the discrepancy that 8k was not included here. The native GBA lineup has their saves only come in those five (5) sizes. This can be checked or referenced at:

For save types 0x0 and 0x1, I don't have a clue where that 8k figure comes from. Perhaps it's the smallest container size that the 3DS can make when housing 4k of save data.

Referring back to the above post's diagram, Structure of the GBA Virtual Console save file.

The original GBA saves are found in the highlighted green block (adjust the size per save type).

Those save types can be grouped into two (2) distinct categories.

Group (B) saves are easy to handle. They can be extracted 1-to-1 with simple copy and paste.

Group (A) or EEPROM only saves require special attention. In order to extract these in their original/standard GBA save form, chunks of 0x8 bytes starting from the origin have to be flipped.

EEPROM reversal

Ending Note

  1. The size of the .gba rom is independent of save type.
  2. There are GBA games that don't have any saves.
  3. I was not able to verify the existences for types 0x1 and 0x3 in my NSUI test injections. My best guess is that they're the RTC versions of their respective 0x0 and 0x2 counterparts.
  4. Only a handful of test samples were done. While I haven't observed byte reversal for Flash and SCRAM/FRAM type saves, I can't say for certain this is true for all cases.
  5. My job here's done. Remember to account for the CMAC when restoring or injecting the save. Have at it adding GBA VC support to your save managers. Please. :|
  6. If none of you guys decide to make this happen, I'm really not looking forward to making a save manager in gm9 script.

Late Edit

  1. No clue if the ARM7 registers affect importing/exporting the saves in any way.
TurdPooCharger commented 5 years ago

Edit - This post is not specifically about GBA VC saves.

Oh nice.

To get around the hurdle of importing saves with Secure Value on a different system, this folder/file can be deleted:

TurdPooCharger commented 5 years ago

Small update. I was wrong about the section dubbed as "cache garbage collection". It actually serves an important role.

GBA VC makes the newest save alternating between top and bottom. Even numbered saves starting from 00 are placed in the lower half while odd numbered saves are put on top.

Whichever of the two save counters has the higher number is marked as the newer save.

@d0k3, sorry I didn't catch this sooner, but CMAC corrections have to be done both halves.

This binary saving mechanism will complicate making a dedicated GBA VC save manager.

ghost commented 5 years ago

Tell me what address is stored information about the save type of in the GBA rom? (footer of GBA generated from GBA rom)

TurdPooCharger commented 5 years ago

Here's a diagram showing the placement order of the newest made save.

I probably should have mentioned back in the Calculating the GBA Virtual Console CMAC that the 0x30 NormalKey is used to calculate the CMACs for the 00000001.sav. This is of course different than the 0x24 (normal?) KeySlot used on the agbsave.bin found in [S:] SYSNAND VIRTUAL.

Another observation I've found is that the top and bottom slots sometimes switch places with assigned odd/even numbers.

If the slot with the bigger number fails to load its save due to mismatched CMAC, the AGB_FIRM will default on using the other slot. This is a kinda neat redundancy in not losing all of your save progress.

Changes in GBA VC saves

d0k3 commented 5 years ago

@TurdPooCharger - as always, thanks a lot for your research. It's highly appreciated. You notify me when this is ready to implement, alright?

TurdPooCharger commented 5 years ago

@d0k3, much appreciated and will do.

With the latest GM9 builds, I will continue testing "SDTransfer" between the n3DSXL and o2DS with GBA VC games having all the different save types.

BTW, I was reading the Revision history of "3DS Virtual Console" and noticed you're one of the contributors that delved into the GBA VC NAND Savegame and NAND Savegame on SD.

Also, issue #205 started by @cheatfreak47, the weirdness of EEPROM saves was addressed and tackled before,

It seems like all the pieces were already in place.

Was the CMAC calculation for GBA VC saves on SD card not implemented due to difficulty distinguishing against the other 3DS 00000001.sav, accounting for the two save slots, and the mayhem of identifying the save types to adjust for the expected bottom slot's offsets?

cheatfreak47 commented 5 years ago

I definitely do seem to recall there being builds capable of injecting gba .sav into the SD card saves, but I seem to recall running into some issues with getting it to work consistently due to how the 3DS copies saves from SD to NAND before launching AGB-

iirc, if the last game launched was the same game you injected a save into SD for, the save on SD wouldn't be copied to NAND unless another game was launched in between, which made things even more confusing for the end user

I.E. You could launch a game, quit it, inject an SD savegame, and then launch the game again, and the old save would still be there because the SD card save wasn't copied to NAND, as it seems the copy from SD to NAND only happens when necessary.

The main goal I had for that issue you linked was just to make sure all GBA save types could be injected into GBA VC without needing to do any extra work aside from copying the .sav to the SD card and doing the injection process once.

@d0k3 remind me again, did you ever implement SD .sav injection as part of the GBA NAND Inject process? I seem to recall us discussing that as a potential option to remove any potential for failed injections, where you'd read the TID from the NAND savegame and after injecting the save into NAND, you'd inject the save into the SD card title with the same TID, and fix it's CMAC, but based on the discussion going on in this thread, I'm assuming you didn't.

Also @TurdPooCharger really good job on this issue thread, all the diagrams and explanations with sources are fantastic and quite useful. Very nice.

TurdPooCharger commented 5 years ago

@cheatfreak47, thanks for the backstory of previous attempts to directly dump and inject SD saves. This got me to looking closer at whether AGB_FIRM chooses to load the save from agbsave.bin as first pick if the game in question was played back-to-back.

From A/B testing, I did not observe example GBA VC game (The Minish Cap) load from the agbsave.bin.

However, this did lead to solving the mystery behind the odd/even save counters flipping between the top/bottom slots. @d0k3, it was your GodMode9 injection to agbsave.bin all along!

How GodMode9 (roughly) injects the save

  1. The selected raw GBA save has its size checked to match against the agbsave.bin.
  2. The 00000001.sav in Nintendo 3DS (A:/ drive) is looked for by TitleID. Its size and .SAV magic are also checked for.
  3. The RAW save is written into the [single] slot of the agbsave.bin. For EEPROM saves, GodMode9 does the 0x8 bytes chunks flipping.
  4. In the agbsave.bin, the save counter is raised by +1??
  5. CMAC correction done on the agbsave.bin. I can only guess with the 0x30 NormalKey.
  6. The agbsave.bin (up to the end of the raw save) is copied over to the 00000001.sav. To remove the guesswork, double injection is done on both top and bottom slots.
  7. A final CMAC correction is done on the agbsave.bin. This time it is with the 0x24 NormalKey.

Not that it really matters, save counter odd/even numbering gets thrown off due to the double injection.

Final Report I limited the SDTransfer testing to five (5) example games going from n3DSXL to o2DS, each chosen as archetype save size/related type. The manually CMAC corrected saves crossed over without issue.

GBA VC save types (condensed)

cheatfreak47 commented 5 years ago

Ah so it seems the injection to SD is happening then, right? Good, that's what I thought. It's probably possible to add injection into GBA saves on JUST the SD card, but I think that's probably a bit outside the scope of GM9's intended featureset. Something like that would be better suited to a Save Manager like Checkpoint, though that'd be quite a task to implement, I bet.

TurdPooCharger commented 5 years ago

@cheatfreak47, :-}

d0k3 commented 4 years ago

Hmmm... hold on. I had a few very busy weeks / months, thus I may have missed something here. Is there anything I need to change in the way GM9 handles GBA VC save injections? Asking @TurdPooCharger and @cheatfreak47 here.

d0k3 commented 4 years ago

Okay, trying to summarize the gist from here (for GM9 development, that is):

To get around the hurdle of importing saves with Secure Value on a different system, this folder/file > can be deleted:

  • 1:/data/< ID0 >/sysdata/00010011/00000000

Think we should delete that file on a system transfer?

Another observation I've found is that the top and bottom slots sometimes switch places with assigned odd/even numbers.

  • Bottom slot is (typically) numbered even.
  • Top slot is (typically) numbered odd.
  • I don't know the cause of the switched numbering.

I don't know the cause of switched numbering either. Seems completely random to me.

Was the CMAC calculation for GBA VC saves on SD card not implemented due to difficulty distinguishing against the other 3DS 00000001.sav, accounting for the two save slots, and the mayhem of identifying the save types to adjust for the expected bottom slot's offsets?

@d0k3 remind me again, did you ever implement SD .sav injection as part of the GBA NAND Inject process? I seem to recall us discussing that as a potential option to remove any potential for failed injections, where you'd read the TID from the NAND savegame and after injecting the save into NAND, you'd inject the save into the SD card title with the same TID, and fix it's CMAC, but based on the discussion going on in this thread, I'm assuming you didn't.

Yup, we do inject those saves to the SD. But not directly. The AGBSAVE is always used as base.

Not that it really matters, save counter odd/even numbering gets thrown off due to the double injection.

Oh! That sounds about right. Can you tell me how you would improve it?

TurdPooCharger commented 4 years ago

Regarding anti savegame restore,

Absolutely. As far as I can tell from helping others on GBAtemp when recovering their Nintendo 3DS library of games, having that in place adds headache. It's only purpose is stop you from rolling back your save progress when swapping in an older 00000001.sav.

Typical example real scenario Someone plays the game Animal Crossing: New Leaf, installs custom firmware on their 3DS system, and then backs up their sysnand *.bin image.

On a later date after extensive softmodding, this person bricks their 3DS firmware. Their first instinct is to restore their sysnand with that backed up image, albeit it with the lose of later installed DSiWare games & saves and their user nand setup reverted back to when that image was made.

Let's say that person launches AC:NL and doesn't know about "Secure Value". (S)he will be greeted with this unpleasant message, wrongly assume their save was corrupted from the softbrick, and then inadvertently (and permanently) delete their save data if they continue to that game's title menu.

Unless that file has some other use like with online play records that we're unaware of, I recommend deleting that 00010011/00000000 whenever doing NAND restore, CTRTransfer, or SDTransfer.

About the SD agbsave's top and bottom slots, whichever one has the higher order or newer save data,

I don't know the cause of switched numbering either. Seems completely random to me.

This was answered in one of the other posts. The flipping was discovered to be due to how the agbsave.bin double injects both top and bottom slots with the same counter. When a new save is made, one of the slots updates their counter and may change its odd/even order.

On why fixcmac, Fix CMACs for drive, and Calculate CMAC are missing or not implemented for GBA VC saves found on SD card,

Yup, we do inject those saves to the SD. But not directly. The AGBSAVE is always used as base.

In PM, you mentioned about differentiating the various gba save types like how the dbs series are handled.

I don't know a lot about C/C++ programming but did manage to find the nandcmac.c source code to having already included handling the saves for the top slot only.

about nandcmac

~ ~ ~ ~ ~

In the GBAVCSM script I created, the way I distinguish the gba vc saves from everything else is to look for the bottom slot first in increasingly larger sizes:

The saves can only be in three distinct states as shown in this previous diagram.

~ ~ ~ ~ ~

Here's a GM9 build that demonstrates fixing gba vc CMACs for both slots, albeit with the assistance of a gm9 script.

GM9 SDTransfer fixcmac SD agbsave.zip Included in this archive are:

GM9 compiled with the above changes to the nandcmac.c, and the Makefile with this edit: export VERSION := v1.8.0_SDTransfer

Perhaps the script itself may provide some ideas in how to implement their techniques into the actual cmac correction engine.

~ ~ ~ ~ ~

Personal attempts were made to try to define more CMAC_AGBSAVE_SD cmac types based on sizes at their expected bottom slot offsets. I got stuck with:

and also:

Improving the S:/agbsave.bin in how the Inject GBA VC save works?

Oh! That sounds about right. Can you tell me how you would improve it?

The double top and bottom slot injections works fine the way it does when restoring a user selected save. Sure, it's possible to implement single slot injection and have the one with lower counter serve as backup, but in my opinion, that adds unnecessary complex coding.

The GM9 gba vc save injection works. It's not broken. Let's not mess with it.

TurdPooCharger commented 4 years ago

@d0k3, I might have figured out a way to include gba vc sd cmac. Expect a possible pull request.

d0k3 commented 4 years ago

Okay, see my latest commit to staging. That fixes the slot handling with respect to number of times saved (as stored in the AGBSAVE header).

As for deleting that 1:/data/< ID0 >/sysdata/00010011/00000000 file - that's out of question for when copying the NAND file to the virtual drive as that's just what it is, a simple copy operation. It's also out of question for Safe NAND Restore, at least as long as we're not 110% sure deleting that file can never have unwanted effects. I'll be thinking about including that fix in the CTRNAND transfer, though.

I'd also be okay with including that hack in the GM9 MegaScript and maybe the NAND Manager script, as long as you're sure it's the right thing to do. You know how to do it and how to test it, so you can do a pull request (if you want to).

As for the remainder of what was discussed here, I think that's handled by your latest pull request.

Thank you again, your help is highly appreciated!

TurdPooCharger commented 4 years ago

@d0k3, totally understandable. I'll do pull requests later for those two scripts to include automatically deleting the 1:/data/< ID0 >/sysdata/00010011/00000000 after more people report back how their 3DS systems react having that file deleted. The 3DS homebrew scene is slow these days, but occasionally there is always someone who requires recovery from severe firmware softbrick or rebuilding their SD titles library.

That advice in deleting that specific file is in ongoing public field tests.

So far, so good. However, having more collected trial data doesn't hurt, so there's no rush in implementing GM9Megascript and NANDManager any time soon.

I'll check the staging commit with the agbsave cmac handling once I get the chance to swap back to linux SSD. Not enough free space to dual boot on a single hard drive. :/

This weekend I should have enough free time to do in-depth gbavc saves testing.

d0k3 commented 4 years ago

Okay, @TurdPooCharger - here's a test build for you. It has the improved GBA VC save injection, and it also removes that anti savegame restore file on CTRtransfers. Maybe you can test? https://f.secretalgorithm.com/sDcgQ/godmode9.firm

Looking into your pull request now.

d0k3 commented 4 years ago

Okay, it may be a good idea to continue the discussion from the closed pull request here. Here's what I last wrote:

@d0k3, can you clarify by what you mean about gba vc titles with extra large saves? Are you referring to titles that have FLASH 1Mbit saves? The pr nandcmac.c does handle that size.

I meant FLASH 1Mbit saves. There is no original GBA VC title (as in, available on the eShop) that has this.

In the latest staging for commit c38c99e on gameutil.c, am I checking for a difference in how the raw *.sav are injected to the slots found in 0000001.sav?

  • agbsave.bin → Inject GBA VC save

I'm not sure if I got that right... The GBA VC save injector needs to recalculate the CMAC cause the number (of times saved) changed between the slots, thus the CMAC changed as well.

Is this commit to be used in conjunction with the pr nandcmac.c, to see if cmacs are fixed for both slots for an individual 0000001.sav?

Also not sure if I got that right. Sure, it would be best to fix both slots. Fixing only the current one is an option, too.

Now, I got one more question... if the SD save contains two slots with identical number of times saved... which slot will it pull upon loading of the GBA VC rom? Also, what if the odd/even order is no more correct? Will it still pull the slot with the highest number?

TurdPooCharger commented 4 years ago

@d0k3, sorry late reply.

Further differential tests were done using this game. This was picked for its small sized eeprom save and due to my personal familiarity of this title.

Now, I got one more question... if the SD save contains two slots with identical number of times saved... which slot will it pull upon loading of the GBA VC rom? Also, what if the odd/even order is no more correct? Will it still pull the slot with the highest number?

The only time the slots had the same counter values was due to your older method of double injection to the agbsave.bin. In the grand scheme, it didn't matter which slot was loaded as both contained the exact savedata progress.

Let's suppose the slots had the same counter values with valid cmacs, BUT their save progresses were different. One slot does have higher priority over the other as you shall see below.

Switching the odd/even order between the slots does not affect the (greater/less than) comparison. Err... there are of course caveats to which slot is loaded depending on those counter values. Also see results below.

CASE SCENARIOS | COUNTER INFORMATION | SLOT SAVE | { CHOSEN SLOT }

0 - Fresh install; unitialized | Top = null | Bottom = null | { null }
1 - Top does not exist. Bottom is even (0x00000000). | Top = null | Bottom = hard | { BOTTOM }
2 - Top is even (0x00000000). Bottom does not exist. [This is not typically possible] | Top = easy | Bottom = null | { TOP }
- - - -
A - Top is odd, bottom is even. Top is bigger than bottom. | Top = easy | Bottom = hard | { TOP }
B - Top is even, bottom is odd. Top is bigger than bottom. | Top = easy | Bottom = hard | { TOP }
- - - -
C - Top is odd, bottom is even. Top is smaller than bottom. | Top = easy | Bottom = hard | { BOTTOM }
D - Top is even, bottom is odd. Top is smaller than bottom. | Top = easy | Bottom = hard | { BOTTOM }
- - - -
E - Both are odd. Both are equal. Top and bottom have different raw saves. | Top = easy | Bottom = hard | { TOP }
F - Both are even. Both are equal. Top and bottom have different raw saves. | Top = easy | Bottom = hard | { TOP }

Test Cases

gbavc_slot_selection

After using the GM9_v1.9.0-9-g499e301 build, this is what I observed:

Your reworked nandutil.c still does double slot injections with the agbsave.bin, except it now first injects at the bottom slot, increases the counter value by +1, and then injects the top slot with that +1 counter.

With the knowledge gained from the mapped out slot selection behavior from the above mentioned testings, I could not determine if this new injection method has any advantage or disadvantage versus the old method.

I will be submitting an additional small change to nandcmac.c in pull request # 548. Make slot selection for cmac correction follow closer to what gba vc chooses.

d0k3 commented 4 years ago

Okay, I beautified the GBA VC SD save CMAC fixer code a bit. We now have the removal of the Anti Savegame Restore file on CTRNAND transfers, improved GBA VC save injections, and proper fixing of GBA VC SD save CMACs.

You wanted to add something?

Also, what about the SD CID in those GBA VC saves? Is there any problem when it doesn't match the current SD? People actually switch to bigger / better SD cards all the time, so I think Nintendo doesn't invalidate those saves just because of a changed SD. I may be wrong, though.

TurdPooCharger commented 4 years ago

Okay, I beautified the GBA VC SD save CMAC fixer code a bit. We now have the removal of the Anti Savegame Restore file on CTRNAND transfers, improved GBA VC save injections, and proper fixing of GBA VC SD save CMACs.

For the GBA VC save injections with the agbsave.bin, did you implement the proposed method of single slot injection?

You wanted to add something?

A separate pull request that disables anti-savegame restore for the two gm9 scripts.

From (staging) commit cfae228, I take this to mean I can get the go ahead with the edits.

I might have failed to mention this before. When that 00010011/00000000 file gets deleted, the reset on the Secure Value is only good until the next time the game launches and sets in a new Secure Value.

Let's say for the Animal Crossing: New Leaf example title, I tripped the anti-gamesave restore by swapping in an older copy of its 00000001.sav. I manually navigate in the [1:] drive to delete that specific file. The next time I launch AC:NL, that older save will load. I make some progress in the game and then make a new save.

However, let's say I don't like how my house's interior redecoration turned out, so I try swapping in that older 00000001.sav again to start over. That third time I launch AC:NL, the anti-gamesave restore will the notice a mismatch in Secure Value and kick in again.

Also, what about the SD CID in those GBA VC saves? Is there any problem when it doesn't match the current SD? People actually switch to bigger / better SD cards all the time, so I think Nintendo doesn't invalidate those saves just because of a changed SD. I may be wrong, though.

I was wrong about needing to change or match the SD CID in the GBA VC's 00000001.sav when SDTransferring the Nintendo 3DS folder. As long as the CMACs are fixed, the target system will load the source system's gbavc saves perfectly fine.

What will happen is that newly made saves will automatically change the SD CID to that of the SD card that's being used.

d0k3 commented 4 years ago

For the GBA VC save injections with the agbsave.bin, did you implement the proposed method of single slot injection?

Good proposal! I somehow missed that. I implemented it now.

From (staging) commit cfae228, I take this to mean I can get the go ahead with the edits.

Yup, you can go ahead.

I might have failed to mention this before. When that 00010011/00000000 file gets deleted, the reset on the Secure Value is only good until the next time the game launches and sets in a new Secure Value.

Yup, that's what we expect, right? We'll fix that on the CTRNAND transfer, but everything the user does afterwards, he is on his own, right?

Anyways, as for this issue... you may want to test some stuff, but we can close it once you say we implemented everything, and everything is working that's been discussed.

TurdPooCharger commented 4 years ago

Sorry for the late reply. I had a immediate concern fixing the family's washing machine that kaputt. After 12 years of ownership, the flange that holds the steel drum broke apart... You can get a grime idea of the dilemma by google searching "samsung spider arm corrosion".

Anyway, I got an on-&-off chance testing the staging build over the week. There's a bug found in the agbsave.bin injection. Before the bug can be better described, I'm photo editing several, before-and-after diagrams of what's happening to the agbsave.bin and 00000001.sav.

Edit - I should have that bug report posted around Monday 11/18 4am EST.

d0k3 commented 4 years ago

Don't worry too much, it's not like this issue is holding us back. Thanks for testing this as much as you do! This will be done when you say it's done, not earlier.

Good luck with that washing machine, judging from the Google results it doesn't look good :/.

TurdPooCharger commented 4 years ago

Refer to this previous post in regards to the below diagrams.

This is the current method of injecting the GBA VC SD saves. Per the slot selection process chart, the top slot is always loaded even though both slots are carrying the same progress and equal counter values.

In case the user selected save fails to load, there is no safety net of falling back on the last, in-game made save (EASY).

01 - GM9 v1 9 0 (stable)

This is the first revision to the injection method made to nandutil.c from commit c38c99e.

Like the current method, it does double slot injections with the notable difference of staggering the counter values. The top slot is always +1 bigger than the bottom slot. The original idea behind this change was to maintain the slots' odd/even number order in case this mattered to the AGB_FIRM (which this doesn't).

Per the slot selection process chart, the top slot is always loaded. This method shares the same drawback as the current method.

02 - GM9 v1 9 0 (build 9-g499e301b)

This is the second revision to the injection method from commit 1a131b2.

Dealing with two save slots is maddening and tricky business.

03 - commit 1a131b2 (bug)

Lastly, here is the proposed single slot injection method as described from the pull request #548 (comment).

Although this fact doesn't really matter, the odd/even number order is maintained like what you were originally aiming for. The advantage of this method is the safety net. If the user's save injection fails to load, AGB_FIRM should load their last made save (EASY) as a redundancy.

04 - proposed injection method

d0k3 commented 4 years ago

Just a quick heads up - this is not forgotten. I'll get to it as quick as I can (I hope this week).

d0k3 commented 4 years ago

Okay, I think I nailed it this time. You may compile yourself from latest staging or just use this test build: https://f.secretalgorithm.com/sbKr4/godmode9.firm

Did you also test the removal of the Anti Savegame Restore file on CTRNAND transfers, and my rewrite of fixing of GBA VC SD save CMACs?

When done, we can close this issue (I think), as everything discussed should be implemented.

TurdPooCharger commented 4 years ago

Since gifting the o2DS to my nephew, I was down one system in not being able to do SDTransfer tests. Luckily, I was able to borrow my niece's n3DS for the time being.

Okay, I think I nailed it this time. You may compile yourself from latest staging or just use this test > build: https://f.secretalgorithm.com/sbKr4/godmode9.firm

That's quite a leap in size from GM9 v1.9.0 (492.0 kB) to GM9 v1.9.0-21-gaf95f038 (756.0 kB).

I tested both the provided sbKr4 build and one self complied with the latest staging.

Did you also test the removal of the Anti Savegame Restore file on CTRNAND transfers, and my rewrite of fixing of GBA VC SD save CMACs?

Using the sbKr4 build on the n3DSXL,

  1. Backed up the SysNAND *.bin image.
  2. Backed up a copy of ACNL 00000001.sav from the [A:] SYSNAND SD.
  3. Went to HOME Menu, launched the game, and made a quick save.
  4. Returned to GM9 and manually restored the backed up save file.
  5. Launched ACNL again. Checked that the anti-save protection kicked in. When asked to delete the save data, cancelled and went back to HOME Menu.
  6. Opened FBI and deleted every system titles (those listed in red). Basically, pretend to be a noob who wanted to de-clutter 'junk' titles.
  7. Restarted the 3DS to verify the firmware is indeed softbricked. Derpity, derp. Yup, it's bricked.
  8. GM9 → 11.5.0-38U_ctrtransfer_n3ds.bin → Transfer image to CTRNAND.
  9. Before exiting GM9, took a look in 1:/data/< ID0 >/sysdata/00010011 to see that the 00000000 was deleted.
  10. Launched faketik to restore missing tickets and relist HOME Menu SD titles.
  11. Launched ANCL again. The old save loaded up without hassle.
  12. Updated the firmware to 11.13U.
  13. GM9 dumped all the ctrnand system titles into legit CIAs. One last check to see CTRTransfer doing its job.

So yay, that anti-save game restore removal works just as intended. :+1:

When done, we can close this issue (I think), as everything discussed should be implemented. I got good news and bad news.

good news These five (5) chosen GBA VC titles were installed on the n3DSXL and then moved over to the n3DS; [A:] SYSNAND SD + Fix CMACs for drive.

Trying all sorts of combinations,

, the sbKr4 build fixed the CMACs for slots that should be deemed newest.

The SDTransfer tests passed flawlessly. :+1:

bad news Both the provided GM9 v1.9.0-21-gaf95f038 and independent compiled build still incorrectly does agbsave.bin SD injection like in the 3rd example before&after diagram.

d0k3 commented 4 years ago

Oooops.

I still missed something (you may take a look at the latest staging commit). It should be fixed and working as it should now. Either compile yourself or use this test build: https://f.secretalgorithm.com/lvebe/godmode9.firm

If this is is the last one down, you may close this.

TurdPooCharger commented 4 years ago

I like how you made the SD injection better by leaving alone the agbsave.bin counter value.

At a glance, everything seemed to be in order.

05 - lvebe (agb=top)

But then this was found. This one almost snuck through. Yikes.

06 - lvebe (agb=bot)

d0k3 commented 4 years ago

I actually did this deliberatly, to make sure the save actually gets injected and used afterwards. I understand that this case really only happens when tampering with stuff (as you did).

Dunno, you decide. Should I change this?

TurdPooCharger commented 4 years ago

I actually did this deliberatly, to make sure the save actually gets injected and used afterwards. I understand that this case really only happens when tampering with stuff (as you did).

Dunno, you decide. Should I change this?

While I did edit the agbsave.bin and 00000001.sav files in order to showcase the changes occurring from the injections, the two examples I posted with the starting scenarios can happen from typical usage of GBA VC.

While the SD injection method in lvebe does work regardless of which slot has the starting higher counter and whether the GM9 session has multiple injections (the last made injection should load per higher counter value), here's another side-by-side diagram to better reflect the pitfall that can occur from a second (bad) injection.

The progression is followed from left to right.

07 - lvebe (differences)

I believe you should tweak the injection so that the lower example's outputs symmetrically matches its counterpart.

d0k3 commented 4 years ago

Okay, I understand. Here's another try, I think that one should (finally!) nail it: https://f.secretalgorithm.com/YMEVi/godmode9.firm

TurdPooCharger commented 4 years ago

Okay, I understand. Here's another try, I think that one should (finally!) nail it: https://f.secretalgorithm.com/YMEVi/godmode9.firm

:(

08 - YMEVi

@d0k3, I discovered a foil in the slot selection process. Back in this post for the flow chart, I wrote that if the top slot counter is bigger than the bottom slot, the top slot would get loaded.

This is only true if the value of the top counter is exactly +1 bigger than the bottom's.

After injecting the normal.sav on the YMEVi build, I exited GM9 and launched the game to check if the injection was success.

In that AFTER state,

, bottom slot was the one ended up getting loaded.

To rule out if there was a flaw in the CMAC handling, I took a copy of the 00000001.sav in that AFTER state and did two further counter edits.

AFTER state with edit (add by +1)

AFTER state with edit (subtract by -1)

What this ultimately means is that the SD injection has to be very precise.

From the discord channel, I've read that you guys are aiming for a v1.9.1(?) release around Christmas time. I don't want you to pressure yourself with undue stress to force include this new injection method by that deadline if you have holiday and family obligations. We can postpone this feature's inclusion for v2.0.0. However, if you're super dead set on making this happen, my schedule is freed up to do testing up until Dec-24th EST.

d0k3 commented 4 years ago

Just a minor setback, it's actually very easy to mess up here. I do want this in v1.9.1, tho.

This one had an obvious mistake, in that I did not check the top counter for equality but for inequality instead, and then injected to the bottom slot.

Try this one: -- snip --

Also:

(4) Also, the slot that does get injected must always end up having a counter value exactly +1 compared to both agbsave.bin and the (other/not injected) slot.

You know that's impossible to ensure if we check only the top counter? On the other hand, with the current implementation, it should only mess up when there's manual tampering. It should not mess up from a typical state (ie. one the 3DS fw generated).

EDIT: Stop the testing, I found another mistake. Read below.

d0k3 commented 4 years ago

Okay, additional info, right now we check for three things:

  1. We check if the file is large enough and readable.
  2. We check if there's a proper agbsave at the top slot.
  3. We check if the SysNAND AGB counter is == the SD TOP AGB counter

Granted, the first one should never fail, if it does we got bigger problems. So we don't care for now. That leaves us with only two cases: TRUE (if ALL of the three conditions above are met) and FALSE (if any are false).

Internally, we increment the SysNAND AGB counter by 1 in any case, but we don't write back (to SysNAND) . TRUE: We inject to the bottom slot. Look at the three conditions above - that also means we only do this if there actually is a proper save in the top slot. Otherwise we need to assume the top slot has never been written to and is next. FALSE: We inject to the top slot. We also do this with a borked file (broken condition 1.), but that's just our easy way out there.

Try this test build (or compile from source): https://f.secretalgorithm.com/QBhvR/godmode9.firm Btw, no need to test borked case 1.

TurdPooCharger commented 4 years ago

The QBHvR build has hit the marks for every design criteria. All test scenarios were constrained to what can normally happen from regular usage of GM9 and GBA VC. For reasonable or meaningful testing, initial setups that are impossible to create without the aid of hex editing and CMAC corrections were excluded.

Before I close this issue which signals my complete approval (still pending), I will do several further (but less formal & less stringent) injection tests on my other games. Basically, I'll be checking if the injections are consistent for different save types. This will also give me so more time to ponder if anything else might have been missed.

Once this issue is closed, I'll shortly open a separate PR with those edits for the gm9 scripts regarding the anti savegame restore.

Scenario A - Typical setup for the agbsave.bin and 00000001.sav.

09 - QBHvR (top=04e, bot=03h)

Scenario B - Like scenario A. This build is not sensitive to whichever slot has higher precedence. The bug seen in the previous lvebe build has been fixed.

10 - QBHvR (top=03e, bot=04h)

Scenario C - Only the bottom slot exists due to the game being played one time.

11 - QBHvR (top=null, bot=00h)

Scenario D - Both slots and agbsave.bin all have the same savedata and counter values due to a previous injection from GM9 v1.9.0 or older.

12 - QBHvR (top=03n, bot=03n)

Secenario E - Exactly like Scenario D, except the counter values are even numbered.

13 - QBHvR (top=04n, bot=04n)

TurdPooCharger commented 4 years ago

Everything is in order. Other save types inject just like the Gradius example (EEPROM 512k).

I tested two other scenarios.

Scenario F - How does GM9 handles rolling over after 255?

BEFORE

AFTER

Result - PASSED :+1:

Scenario G - How does GM9 handles rolling over after 4,294,967,295? Kinda digging at the bottom of the barrel. By no means this was necessary to check for.

BEFORE

AFTER

Result - PASSED :+1: