search

d0k3 / GodMode9

GodMode9 Explorer - A full access file browser for the Nintendo 3DS console :godmode:
GNU General Public License v3.0
2.09k stars 190 forks source link

Standardised dump format (ZArchive?) #814

Closed Pep-the-Dog closed 1 year ago

Pep-the-Dog commented 1 year ago

I'm in the process of creating a dump library of all my games, but 3DS games stick out like a sore thumb. Unlike other systems, the 3DS doesn't have a consistent format for roms.

For example, cartridge dumps vs eshop dumps. Ideally, we would be able to dump the same game from either source and end up with the same exact content (including matching checksums). However, this is not the case. Building a cia from a Zelda OOT cart does not produce the same result as one built from an eshop download. This is partly due to the lack of a digital manual in the cart, but the also because actual cxi files don't match either because of differing exheaders.

On top of that we have to juggle management of multiple cxi files per game (for manuals, and download play) plus updates and dlc! There's simply no standard way to manage all these...

I'd really like to be able to store 3DS games in a archive similar to Wii U's .wua format (aka ZArchive). All the game's data could be stored loosely in folders, right down to the .code file, instead of layers of containers. Each cxi could have their own top folder, as well as any updates and dlc. All while being uncompresseable for easy access and modding and being consistent across all dump sources!

This thinking, of course, lead me to discovering @d0k3 's suggestion of this very concept to the citra project: https://github.com/citra-emu/citra/issues/2808

Unfortunately, it doesn't look like they've taken much of an interested in it yet, but that doesn't mean gm9 can't implement it. Perhaps if implemented in gm9, other projects like citra would be more open to the idea?

To be clear, I'm not suggesting that the compression itself be done on a 3DS via gm9. However it would be nice if it could automate the file dumping/collection process, so that the user only needs to compress the resulting folder with ZArchive. Ideally gm9 would be able to read ZArchives as well, allowing them to be installed on real hardware, just like cia files.

I realize that this might be WAY more complicated than I understand, so I have no idea how feasible any of this really is. But I wanted to at least put the idea out there and I appreciate any time you might spend considering it.

ZeroSkill1 commented 1 year ago

All the game's data could be stored loosely in folders, right down to the .code file, instead of layers of containers. Each cxi could have their own top folder, as well as any updates and dlc. All while being uncompresseable for easy access and modding and being consistent across all dump sources!

Desktop tools like ninfs can mount 3DS game containers and provide exactly what you're looking for: a "broken-down" view of every major component of the game's files. You could also use pyctr, its relevant python interface to read game files, for writing a more sophisticated script that does exactly what you want.

Making an installable format in this way (compressed or not) would be challenging as it would require the generation of NCCH containers, TMDs and tickets on the fly, which is something only really feasible in GodMode9 itself. Even then, I'd argue that it's way more effort than it's worth.

However it would be nice if it could automate the file dumping/collection process, so that the user only needs to compress the resulting folder with ZArchive

GodMode9 can already mount game files in a similar way, and you could perhaps use the scripting interface to automate the extraction of individual components.

ihaveamac commented 1 year ago

For example, cartridge dumps vs eshop dumps. Ideally, we would be able to dump the same game from either source and end up with the same exact content (including matching checksums). However, this is not the case. Building a cia from a Zelda OOT cart does not produce the same result as one built from an eshop download. This is partly due to the lack of a digital manual in the cart, but the also because actual cxi files don't match either because of differing exheaders.

Game cards do not have tickets, so you can't dump it into a CIA that is identical to an eShop release. Similarly digital titles don't have game card-specific info like save encryption details. The difference in the extheader has to do with flags about where the title is loading from. (This also means the NCCH signature is different, so changing those flags still won't give you the same file 1:1.)

Even if a custom format is made you'd still need to dump from both sources if you want to preserve them.

d0k3 commented 1 year ago

@Pep-the-Dog - from the answers you have got from here, you may have guessed: what you want is not possible. CDN installable content (typically distributed as CIA) and cart dumps ("3DS" files) are not the same thing. Cart dumps contain data that's not in CDN content (for example the NCSD header and some more additional meta data), and CIAs contain tickets (among other stuff). Even if you'd go as far as going down to the least common denominator, which is the NCCH game file (contained in both, the cart dump and CDN content), you have in no way any guarantee that a NCCH file fetched from CDN is identical to the same game / region / version cart dump NCCH file.

In other words: yes, you have to live with 3DS games coming in two very different formats.

What you can do I already described on the Citra repo: pack the cart dump / CIA into a ZArchive container. To ensure good gains when compressing, you can decrypt the game files before archiving. Decryption, when done right, is fully reversible, meaning ZArchive files can be archive quality. However, as long as there is no interest on Citra's side, there's not much we can do. The puzzle pieces are already there.