[Feature] Native handling of PFS0 ("NSP") archives

leo60228 commented 11 months ago

Nintendo Switch digital titles consist of multiple NCA files, and are conventionally stored as PFS0 archives (a simple Switch-specific archive format) with the extension .nsp. oxyromon currently has no special handling for PFS0 archives, treating them as plain files. No-Intro has two DATs, one for the individual NCAs and one for NSPs. However, this has a problem: the exact layout of the PFS0 archive can vary depending on the tool used to create it, and these details are irrelevant for preservation. It would be preferable to at least have optional support for treating PFS0 files as an archive.

The PFS0 format is very simple. A Rust implementation already exists in the linkle crate.

leo60228 commented 11 months ago

I have a proof of concept implementation at https://github.com/leo60228/oxyromon/tree/pfs0, but it doesn't interact with NSZ, isn't configurable, and doesn't properly handle identical files existing in multiple games.

alucryd commented 11 months ago

That's a good start! I was waiting to see what format would be elected by No-Intro but since I couldn't find any Switch DATs I went for what I had on hand, my own NSP dumps and hand-written DATs. I was half expecting No-Intro to chose XCI for carttridges, it's cool that they went with NSP but as you said that's not very useful until these are made deterministically. So the only truly reproducible option right now is a DAT where games consist of multiple NCA files and the ability to parse PFS0 in oxyromon.

I'm guessing the switch DATs are still private?

leo60228 commented 11 months ago

This issue only applies to digital titles. Cartridges are datted as both "XCI" (a format only used for historical reasons that has the first 0x1000 bytes removed) and "FullXCI" (the "actual" XCI format that only recently gained tooling support). Digital titles are datted as both "NSP" (which is non-deterministic) and "CDN" (which consists of multiple NCA files, as you described). All of these are currently private. I can email the current DATs to you, if you want?

alucryd commented 11 months ago

That would be great, thank you! In the meantime I'll start looking into the updated tooling and have another go at dumping some games so I can at least add support for XCI (last time I checked I was only able to dump NSPs). Hopefully nsz will already support reversible XCZ on the new FullXCI format, if not I'll probably have a crack at a PR when I have some time. I should also be able to look into the digital realm afterwards.

I was planning on adding a way to import games without DAT files into the database at some point, so people can import their own dumps and be able to verify they don't get corrupted during conversion or via bit rot over time. PFS0 could be the first candidate for this (if we don't get NCA DATs in the meantime, or an NSP v2).

Immersion95 commented 11 months ago

I think nxdumptool is the tool used to generate the NSP for No-Intro, it would be great if oxyromon could also convert NSP to nxdumptool NSP format/padding. There is no tool on PC for that as it's a Switch homebrew.

leo60228 commented 11 months ago

nxdumptool NSPs change between updates. There is no canonical encoding for NSPs.

leo60228 commented 10 months ago

Oh, wait, apparently the nxdumptool rewrite is supposed to be deterministic, and the author made an (unfinished, based on commit messages) PC implementation that's intended to result in the same hash: https://github.com/DarkMatterCore/dom_xml_dataset_generators/blob/main/cdn2nsp.py

Immersion95 commented 10 months ago

Yes it is, please note that the new NSP format isn't included in the whole No-Intro NSP database yet according to the author

DarkMatterCore commented 5 months ago

There has been some work in the last few months to update the NSP entries from the No-Intro database to their deterministic variant. It's not yet 100% complete, but it's slowly getting there. The updated set will remain private for the foreseeable future, though.

As for the cdn2nsp script, it is designed to convert standalone CDN files and/or NSPs to the standardized NSP format generated by nxdumptool-rewrite, resulting in 1:1 matches all the time. I recommend cloning the custom-hactoolnet branch and using the cdn2nsp script with the latest hactoolnet CI/CD build (or build the latest commit yourself, whichever you prefer).

This is because the scripts from the custom-hactoolnet branch depend on changes made to hactoolnet that have not yet made it to a release build.

Hope that helps.

alucryd / oxyromon

[Feature] Native handling of PFS0 ("NSP") archives #100