Add TWL Cart support to NTR Cart Dumping...

[ApacheThunder]

Normmatt appears to have taken the liberty of adding TWL cart support to the NTR cart dumper you recently added:

https://gist.github.com/Normmatt/fe9c0308b6bec86443f9e94620b7aeff

It's in the form of a gist though. So I'll leave it to you to sort out the changes. :P

This should add support to allow dumping of DSi Enhanced games. I have not personally tested this however. My build environment is too out of date right now for me to compile this. :P

[thisokra]

Using the build from @Normmatt GM9 claims that my copies of Pokemon White 1 and 2, Pokemon Conquest, and interestingly, SMT Strange Journey have Invalid SecureCartIDs and doesn't let me view/dump Strange Journey is an NTR game I can see other NTR games like Mario Kart and Final Fantasy IV as normal

[YodaDaCoda]

Latest build with Normatt's fix gives the Invalid SecureCartId error once, before listing no files. Previous versions would give the error twice, before listing .nds and .trim.nds.

[YodaDaCoda]

The 20170117 build gives the exact same binary result for the 0x0000000 chunk, but the 0xC403000 differs, which I think is the opposite of expected?

[d0k3]

@YodaDaCoda well... I at least know what causes this, but this doesn't bring us any nearer to the solution. Here's a build with another fix @Normmatt suggested: https://transfer.sh/aaYqg/godmode9-20170118-171337.zip

Also, @thisokra, thanks! This happening with NTR stuff, too, is new.

[thisokra]

Sure thing. Both 20170117-104102 and 20170118-171337 builds treat Strange Journey normally and can dump it 20170118-171337 still gives the Invalid ID error for Pokemon White but not Pokemon White 2. I don't have a way to verify if the White 2 dump is valid unfortunately

[ApacheThunder]

I have a DSi nand dump working with No$GBA. If you wish, you can private message me that rom dump and I can see if it boots for you. I presume it won't work if the modcrypt section is bad. Because No$GBA expects modcrypt to be encrypted I think...Or at least valid anyways. :P

You'll have to pm me at GBATemp. Don't know if there is a PM feature in Github. Haven't bothered to check. :P

[d0k3]

We may be getting somewhere, @thisokra and @YodaDaCoda. A previously unable-to-dump cart worked.

Here's a new build that extends on the fix in the previous 171337 had: https://transfer.sh/PuH6y/godmode9-20170118-191146.zip

Let me know how this works for you. Note: the filename will change if the modcrypt area is bad (= when that invalid ID error happens). One thing that may also help is if you gave me the cart ids (both, from the error message).

[thisokra]

None of the TWL carts I have show an error now (and neither does Strange Journey) I can still get ID pairs from an older build if that would help

[d0k3]

@thisokra - this is fantastic news! We still need more carts tested, and of course these dumps also need to be checked (in no$gba, f.e., or checked against known good dumps). Other than that, this is a big step forward.

I also only need the cart ids in case something goes wrong, so no need to bother.

[YodaDaCoda]

Exciting!

$ md5sum *.nds 37bff1431eda9b3a525737c7f59a432d pokemon_b.gm9.nds 37bff1431eda9b3a525737c7f59a432d pokemon_b.twl.nds

I don't get SecureCartId error anymore and it seems to dump 100%. I'll reach out to @ApacheThunder on GBATemp and see if he has better luck getting past the AP stuff on no$GBA.

[YodaDaCoda]

Ok I've managed to get past the save file issue and can confirm that no$GBA is loading the GM9-dumped version of Pokemon Black in TWL mode. I'll test with Pokemon B2 and W2 (the only other TWL games I have) in about 8-10 hours.

To get past the save error, I had to find the correct memory addresses to patch - I don't know if it's because it's running in TWL mode or if it's because it's a proper dump, but the addresses are different to what's posted all over the internet. In order to make it a matter of public record, I'll post what I found here. Only the second address is different - the first still points to the same instruction. All codes are AR raw format.

Original code to get past save screen (this is the one posted all over the internet): 02006F2C 00000000 02180B8C 00000000

Pokemon Black (and possibly White - untested) TWL code to get past save screen: 02006F2C 00000000 02180BA4 00000000

NB: This doesn't bypass the AP checks in the rom - finding those offsets is rather beyond my abilities at the moment.

[YodaDaCoda]

I seem to have misplaced my B2/W2 carts, but I dumped Pokemon White, and confirmed it runs in TWL mode in no$GBA, though the code you need to bypass the save screen is the original one. I don't have any more TWL carts to test with.

[d0k3]

Alrigh, thank you!

By now I know of only one title that still has the SecureCart ID error - a somewhat obscure DSi exclusive title called "Face Training". Everything else dumped fine. Does anyone on here own this title? Also, testing with more titles is appreciated!

[d0k3]

If you'd rather test in D9, D9 has its cart dumping code updated as well. Grab your test build from here: https://transfer.sh/D2cQb/decrypt9wip-20170119-112252.zip

[ghost]

About DS cart dumping, not really related to this problem, but imo it seems a bit redundant to take the 01 from 0x10-0x11 when dumping (for the file name, with the game code.) Why not use the actual game version at 0x1E instead? Every DS rom I've checked has 01 there so that information is a bit pointless to have, while it might be more useful to have the dumps marked as (for example) ADAE05 for Pokémon Diamond v1.5, APAJ06 for v1.6, CPUE01 for v1.1, etc (It's such a small thing I didn't think it would be worth opening a different issue just for that so I figured I'd just put it there with the rest of the DS dumping discussion.)

[d0k3]

Well... you're mistaken. GM9 does not use the data located there. It just uses the title, which is located 0x00...0x0B. I assume you mean D9? The data located at 0x10...0x11 should be the maker code. Are you really sure this is the same in every case?

[ghost]

I'm not talking about GM9, though... I'm talking about what D9 does. This is an issue thread on the Decrypt9WIP github, is it not?

I looked at what Makercode actually does and it just tells who published the game (hence why everything I looked at was 01, since they're all from Nintendo.) I just feel like using the rom version rather than the makercode would be better... I can't really think of a reason why knowing the makercode would be more important than knowing the rom version :p (I guess unless someone -really- wants to sort their rom collection by publisher?)

[einstein95]

@Ammako You're very much mistaken. D9/GM9 doesn't read the maker code. While it would be good for D9/GM9 to read the revision and add that to the filename (like what wooddumper does), this is not the thread to request that feature. https://github.com/d0k3/Decrypt9WIP/blob/master/source/decryptor/game.c#L2129 If you can point out where in this file, or even in the program, where it reads the maker code, then please do so.

EDIT: It has been pointed out to me that D9/GM9 does indeed parse the maker code at 0x10-0x11 at https://github.com/d0k3/Decrypt9WIP/blob/master/source/decryptor/game.c#L2156

[ApacheThunder]

Why do half-measures when you can just parse the banner region for the full game's banner text? TWLoader now does this (when reading roms on SD though. Not directly off slot-1)...I imagine it would not be difficult to do it on arm9 baremetal as well. Though I suppose reading the version field would be useful for differentiating different revisions of the same game. Outside of that though, not many people will care about that. :P

[einstein95]

@ApacheThunder afaik, to do that you'd need to decrypt the header, parse the pointer, then decrypt the area where it is pointing to and then parse the UTF-8 at that point.

[ApacheThunder]

Beyond the usual NTR card protocol encryption, I don't think header and banner region have any additional layer of encryption. Only the NTR/TWL secure areas have an additional layer of crypto. The banner region for TWL carts is pretected by only a hmac hash in the DSi Extended Header which itself is protected at the end of the extended header with the RSA sig. Nintendo never felt the need to encrypt the banner region.

Besides, the basic NTR protocol code is already in D9/GM9. So this should work even with TWL carts that had issues dumping correctly. (those tended to have issues with the DSi Secure area not decrypting right but maybe that's fixed now)

[d0k3]

Alright, I agree that the rom_version should be part of the NDS dump filename. The maker_code is still useful, so I left it in, too. https://transfer.sh/EpVcc/decrypt9wip-20170120-133100.zip

@ApacheThunder is right - header and banner have no encryption, only the secure area / modcrypt area has. Yes, we could extract the "true" / full name from there. And we could do the same for 3DS carts if we decrypt the icon (smdh) inside the ExeFS. I still won't do it, for two reasons:

• These full names can contain all kinds of special / regional symbols, which would mean I'd either have to handle this or open up the door to all kinds of unforeseen consequences.
• After all, all we need is a unique, identifyable name. Beautifying these names and / or adding info is the job of a rom renamer, not D9 / GM9.

[d0k3]

Accidentially closed :). Anyways, I somewhat feel the original issue is solved with the most recent commits, thus I can close this. Anyone disagreeing on that?

[YodaDaCoda]

Did you deliberately put a smileyface in filenames of dumps generated by that test build? Because if so, that's awesome! If not, I might have a bug to report...

I can confirm that the build dumps files correctly (md5sum matches what I dumped before which ran on no$GBA in TWL mode) but I still get an Invalid secure area (3C1119DB C484A706). Also, the filename stops after the Product ID. E.g. Pokemon White dumps as IRA01 - no .nds or product version; I suspect this is because of the aforementioned smileyface ;).

[GerbilSoft]

The commit that added ROM versions to the filename didn't set a maximum width for the field, so the unitcode (0x02 or 0x03) ends up getting appended.

I sent PR #144 that fixes this, along with some other minor improvements. (Version number for CTR, and use the TWL used size for TWL trimmed dumps.)

I also tested the TWL dump updates with System FLAW. The MD5 before and after are the same, and both match no-intro: e45cc21a32b73a9b0e80e11b23e304ce

Decrypt9WIP does complain about an invalid secure area: 99289C58 5A3069D6

[d0k3]

Alright... invalid secure area is not an actual bug. and doesn't mean that the dump is invalid I guess I should just remove that line from the source code. @GerbilSoft already sent a p/r that solves this problem, there's just some small stuff that has to be adapted a little bit.

The issue in question (TWL cart dumping) is solved, so I'm finally closing this issue (after almost half a year, woohooo!). If anything else that is cart related pops up, feel free to open a new issue.

Also, thanks a ton to @YodaDaCoda, @thisokra, @ApacheThunder, @idgrepthat, @Normmatt, @osilloscorpion and everyone else involved in solving this issue and to