search

ihaveamac / ninfs

FUSE filesystem Python scripts for Nintendo console files
MIT License
438 stars 18 forks source link

Nintendo DS ROM support #12

Closed ihaveamac closed 6 years ago

ihaveamac commented 6 years ago

Would be nice to browse DS and DSi games like GodMode9.

ihaveamac commented 6 years ago

aa230f4f3dc47ad52f1a94cdb035e0f7efbe897e implements this, needs more testing (only tried with 5 DS games so far, see if DSi also works, some special titles with no data dir, etc)

ihaveamac commented 6 years ago

Still need to implement sub-mounting.

@TurdPooCharger you mentioned on gbatemp that you wanted this, you can try it now if you want (install python, then this using pip; or wait until i can provide an exe if you can't). Until I add sub-mounts (so you can just open a cia and it shows the contents of the srl), you will need to do it a second time for the file inside

TurdPooCharger commented 6 years ago

Oh wow, you went and did it? Stoked! :smile:

@ihaveamac, I'll give a go and try to get back with results by week's end. Will need to re-familiarize myself with python, check if the three anti-virus programs I got running clashes with fuse-3ds, and look out for whatever weirdness pops up in Windows 8.1 (Windows 10 can stay the back burning with Linux-based OS).

ihaveamac commented 6 years ago

sub-mounts implemented in dcf2e2ccab69e8ee3e5c7986b75774d56679336f

ihaveamac commented 6 years ago

Feature should be finished enough to make a new release. Best games to try would probably be a few from early releases (2004, etc), mid-lifespan, and late-lifespan. Also should include DS, DSi-enhanced, DSi-exclusive cartridges, and DSiWare.

ihaveamac commented 6 years ago

v1.2b1 contains a standalone executable to make testing easier.

TurdPooCharger commented 6 years ago

Sorry I haven't gotten back to you earlier. Been doing some brake caliper rebuilding (got stuck with this dust boot that ripped :/), GBAtemp forum helping out other users (mostly SD card stuff), and trying to psych myself into reading fuse-3ds guide top-to-bottom before I went at it. ** OKAY, coffee being brewed and full attention to program. Let's see what's it got. :)

Edit 1 - Oh dang. I hit my first snag. Avast antivirus prevents install and sends a report back to examine the file. Will need to disable Avast if it decides to blacklist v1.2b1 pre-build to continue testing.

Edit 2 - Ah shoot. Double whammy. GBatemp is down. It's migrating to a new server. :( . Hey @ihaveamac, rainy check on testing for tomorrow Sunday? * No access to that fuse-3ds thread and tutorial; I'll need to check if GitHub has the guide here. Still waiting on Avast to give me the go ahead or disabling it.

gbatemp is down

ihaveamac commented 6 years ago

using fuse-3ds should be straightforward once you set up boot9/seeddb.

TurdPooCharger commented 6 years ago

After some hiccups waiting for GBAtemp to come back, Avast giving the clear, sleep from the ensuing hard crash of the coffee, boot9/seeddb retrieval, it's up and running.

fuse-3ds_installed

You have any particular requests in what to test for? I was thinking of fuse-3ds vs GodMode9 and the various other (misc. 3DS + NDS programs). Select games to test for by type: DSiWare (cia), .nds roms (DS, DSi-enhanced, DSi??), size (8, 16, 32, 64, 256 MB), game save type??, [USA/JPN/EUR,etc] region.

I will be doing files hash comparison to see if files output match to other extractors.

NDS/3DS editing tool inventory (in no particular order)

ihaveamac commented 6 years ago

I am mostly just concerned if all files from an nds/srl are identical to files extracted using ndstool/GodMode9/Tinke/etc. (first two lack proper Shift-JIS support, so filenames encoded with this will only appear properly in fuse-3ds and maybe some other tools). Things like save type are irrelevant to this. Format shouldn't have changed between any files of any year, but I don't know for sure, that's why this should be tested more.

Outside of that any tool should work with files directly inside a mount point. If any programs require administrator, do not use a drive letter as a mount point to use them with files directly inside.

ihaveamac commented 6 years ago

Some programs like VGMTrans are an order of magnitude slower when opening files inside mounts, rather than from an NTFS drive directly (e.g. 9s to 3m19s).

TurdPooCharger commented 6 years ago

Had to detour for milk pickup run as per my father's request. lol

Testing is underway. Since you seem keen on gunning for Tinke and GodMode9, I'm bench marking fuse-3ds against those two. Five nds roms were selected out of preference and (size.

Gonna hash check and tabulate the differences in their extractions on a spreadsheet.

ihaveamac commented 6 years ago

There's not a lot I can do to improve speed of tiny reads (cache path lookup, allow unsafe reads, etc). VGMTrans in particular reads files mostly 0x1000 at a time. At the very least it managed to work.

TurdPooCharger commented 6 years ago

File comparison was done through Duplicate Cleaner Pro 3.2.7 using Same Content (exact hash) and Same File Name search option.

Battery Test No.1 - fuse-3ds v1.2b1 vs GodMode9 v1.6.3-101 nightly

Title: 0560 - Cooking Mama (US) CRC32: 7FE24E59 Size: 32 MB Info: http://www.advanscene.com/html/Releases/dbrelds.php?id=0591 Differences:

Comment: fuse-3ds outputs banner.bin with the size of 8.93 KB while GodMode9 has it as 2.06 KB. Is fuse-3ds defaulting for all banners as the animated type?

Title: 3398 - Fire Emblem - Shadow Dragon (US) CRC32: 4CE55236 Size: 64 MB Info: http://www.advanscene.com/html/Releases/dbrelds.php?id=3488 Differences:

Comment: Here's where fuse-3ds shines through with its Shift-JIS support. Banner issue as already stated from Cooking Mama. No more mention of the banner for the latter games below.

Title: 5200 - Professor Layton and the Unwound Future (US) CRC32: B311E1EC Size: 256 MB Info: http://www.advanscene.com/html/Releases/dbrelds.php?id=5328 Differences:

Comment: This game was chosen for its size to see how fuse-3ds would fare. So far, so good.

Title: 5367 Golden Sun - Dark Dawn (US) CRC32: 508420E7 Size: 256 MB Info: http://www.advanscene.com/html/Releases/dbrelds.php?id=5569 Differences:

Comment: Chosen for the same reason as the Professor Layton game.

Title: Pokemon - Black Version 2 (USA, Europe) (NDSi Enhanced) [b] CRC32: 25BA3AA6 Size: 512 MB Info: N/A. Rom dump has incorrect header?? 2nd hand info source. Differences: none

Comment: This game was not only chosen for its size and DSi enhanced status, I wanted to expose fuse-3ds to how it handles a ROM dump that's considered bad. Here we see the two programs match everything including the banner. Pokemon Black 2's banner is animated.

I'll probably hold from further testing until notice. Off to Chipotle for dinner break!

ihaveamac commented 6 years ago

hardcoded banner size fixed in 90e9824af6ac9a36595b94c86086be267cf0f1fb

ihaveamac commented 6 years ago

v1.2b2 contains the banner size fix.

TurdPooCharger commented 6 years ago

@ihaveamac, I have a good feeling what Battery Test No.2, or round 2 vs GM9 will be like (will test this release regardless of that guess). The third test will deal with DSiWare cias.

What I'm curious is how it'll fare against Tinke. There were some differences spotted from quick glances between the two. I'll save this one for No.4.

Edit - I'll pick this up again tomorrow. Got some rear brake calipers to install tonight. :(

TurdPooCharger commented 6 years ago

Got my results in. Your previous v1.2b1 passed Avast inspection. I wonder why it thinks it's any different.

avast vs fuse-3ds_v1 2b2

Edit 1 - More info. avast_info

Edit 2 - I think I know what happened. In your v1.2b2* release, you mentioned GUI: Slightly more verbose GUI startup. One of the characteristic of filerepmetagen is annoying pop-ups. This is what probably tripped the inspection check.

Edit 3 - I'll see if I can get Avast to make an exception for fuse-3ds for the time being. I rather not fully disable my antivirus.

ihaveamac commented 6 years ago

Just add an exception or something. This isn’t really related to this issue.

TurdPooCharger commented 6 years ago

Battery Test No.2 - fuse-3ds v1.2b2 vs GodMode9 v1.6.3-101 nightly [.nds, SRL roms] Basically a repeat as Test No.1.

:+1: - Title: 0560 - Cooking Mama (US) :+1: * - Title: 3398 - Fire Emblem - Shadow Dragon (US) :+1: - Title: 5200 - Professor Layton and the Unwound Future (US) :+1: - Title: 5367 - Golden Sun - Dark Dawn (US) :+1: - Title: Pokemon - Black Version 2 (USA, Europe) (NDSi Enhanced) [b]

Comment: Every files match in cross comparison in both name and hash, * save for the text files in Fire Emblem with Japanese naming (they still match in hash).

Battery Test No.3 - fuse-3ds v1.2b2 vs GodMode9 v1.6.3-101 nightly [DSiWare cia roms] Games were acquired through that other shop and dumped into CIA format through GM9-v1.6.3-101.

Title: Shantae Risky's Revenge (USA) CRC32: 46BAAB4E TWL TitleID: 000300044B533345 Product Code: TWL-KS3E Size: 15.9 MB Differences: [Left: fuse-3ds // Right: GodMode9]

The WarioWare game is one of the very rare DSiWare titles that has an e-manual. fuse-3ds makes two copies of that e-manual. The decrypted.cfa was found in the decompile folder 0001.00000003. It is the same as 0001.00000003.ncch found one directory level back, which is the same as 0001.00000003.app from GodMode9.

TurdPooCharger commented 6 years ago

Battery Test No.4 - fuse-3ds v1.2b2 vs Tinke v0.9.2-dev [.nds, SRL roms]

Title: 0560 - Cooking Mama (US) Differences: What fuse-3ds has that Tinke is missing.

What Tinke has that fuse-3ds is missing.

What fuse-3ds has that Tinke is missing.

What Tinke has that fuse-3ds is missing.

What fuse-3ds has that Tinke is missing.

What Tinke has that fuse-3ds is missing.

What fuse-3ds has that Tinke is missing.

What Tinke has that fuse-3ds is missing.

What fuse-3ds has that Tinke is missing.

What Tinke has that fuse-3ds is missing.

ihaveamac commented 6 years ago

ftc probably contains extra files that aren't needed as much. I followed GodMode9's layout which doesn't include raw versions of the File Name Table or File Allocation Table.

TurdPooCharger commented 6 years ago

Just for kicks, I got two more test runs for your fuse-3ds to go up against: DSBuff and DSLazy. Both of them uses NDStool as its basis.

ihaveamac commented 6 years ago

"Of course, the .nds vs .app extension is explainable when Nintendo migrated DSi titles to the 3DS, they've changed this to match it more in line with the 3DS rom standards." <- using the nds extension was a personal choice to indicate it is not an NCCH. GM9 and Nintendo's firmware don't do this.

TurdPooCharger commented 6 years ago

Battery Test No.5 - fuse-3ds v1.2b2 vs DSBuff v1 Final [.nds, SRL roms]

Title: 0560 - Cooking Mama (US) Differences:

What fuse-3ds has that DSBuff is missing.

What DSBuff has that fuse-3ds is missing.

Title: 3398 - Fire Emblem - Shadow Dragon (US) Differences:

[Left: fuse-3ds // Right: DSBuff]

What fuse-3ds has that DSBuff is missing.

What DSBuff has that fuse-3ds is missing.

Title: 5200 - Professor Layton and the Unwound Future (USA) Differences:

[Left: fuse-3ds // Right: DSBuff]

What fuse-3ds has that DSBuff is missing.

What DSBuff has that fuse-3ds is missing.

Title: 5367 - Golden Sun - Dark Dawn (USA) Differences:

[Left: fuse-3ds // Right: DSBuff]

What fuse-3ds has that DSBuff is missing.

What DSBuff has that fuse-3ds is missing.

Title: Pokemon - Black Version 2 (USA, Europe) (NDSi Enhanced) [b] Differences:

[Left: fuse-3ds // Right: DSBuff]

What fuse-3ds has that DSBuff is missing.

What DSBuff has that fuse-3ds is missing.

Comment: I had a belief pitting these two in a comparison match up would have been a hands down win for fuse-3ds. While DSBuff does show its age in not having kept up with DSi/DSi enhanced games, it still holds up for the older NDS roms. Here the overlay folder and accompanying files appear again just like in Tinke.

There is something that warrants attention unless you've already know what this is and have accounted for it. There are discrepancies between the arm9.bin files with those last 12 bytes.

ihaveamac commented 6 years ago

no idea what's going on with the extra bytes, GodMode9 doesn't display them, ndstool does.

ihaveamac commented 6 years ago

https://github.com/devkitPro/ndstool/blob/313b0be6440d3087b6d433c3ea057605d2c95197/source/header.cpp#L585-L595

1cb275e04b342c86683855dc4933862854a28673 ARM9 footer will be read now according to what ndstool does

d0k3 commented 6 years ago

@TurdPooCharger - can you add the CRC32s of the rom dumps you tested?

TurdPooCharger commented 6 years ago

@d0k3, I was beginning to wonder when you'd pop up. :smiley:

Check the posts that have Battery Test No.1 - fuse-3ds v1.2b1 vs GodMode9 v1.6.3-101 nightly and Battery Test No.3 - fuse-3ds v1.2b2 vs GodMode9 v1.6.3-101 nightly [DSiWare cia roms] for the hashes.

Double check the arm9.bin footer for DSi, DSi enhanced, and DSiWare games too as DSBuff / DSLazy / NDSTool can't handle those so there isn't something for fuse-3ds & GodMode9 to cross compare against for these titles.

Anyway, you guys got the rest of this? I got some car brake calipers that needs some tending. If ever you get your hands dirty in rebuilding brake dust seals, use (Sil-Gylde) lube! General life tip, rubber tears when you force it in dry lol.

ihaveamac commented 6 years ago

I assume this could be considered complete enough. It has been tested with various games and works fine. Probably won't bother with extracting overlay files unless there's enough requests for it and/or it's not too complicated.

New release won't be put up until PyInstaller supports Python 3.7.0.

mariomadproductions commented 6 years ago

I would find overlays useful, especially if there was also a virtual folder for decompressed versions. Same with the ARM9.bin.

ihaveamac commented 6 years ago

Changed my mind: I'll put out another beta release using Python 3.6.6, but a stable release won't happen until 3.7.0 works.

About overlays: I'll see how difficult it is to read them.