Reading content0.game/romfs fails for Fire Emblem Fates - Birthright

[0xC0DEDEAD]

Tested on N3DS a9lh GM9 1.1.3 by - mounting either .3ds from the GAMECART C: drive or encrypted or decrypted dump from SDCARD 0: drive - attemting to open G:/content0.game/romfs exits to main drive menu and unmounts the image If I try to get directory info for content0.game/romfs or content0.game get - Analyze dir: failed!

[d0k3]

Okay. There are two parts to this problem.

The first is that the directories inside that RomFS contain too many entries, effectively blowing up the limitations set by the limited memory GM9 provides for the directory entry buffer. That is fixable.

The second is that some of these directories inside the RomFS include japanese symbols (yes, even on european and us versions). This may not be fixable, cause I need to keep FAT compatibility, and FAT has no standard way of supporting japanese symbols.

Thinking about it, but may not be fixable at all. Do you know of any other rom that has this problem? Also, may I point you here? ctrtool_unpack_CCI.bat will do the job for you, on a decrypted rom, when drag and dropped.

[0xC0DEDEAD]

I think its as you predicted, too many entries. I used ctrtool_unpack_CCI.bat and there were 26,877 files and 659 folders in romfs. Not sure if there are any Japanese symbols, cant think of an easy way to check.

None of the other carts I've dumped have a problem.

Isn't the FAT compatibility just needed for writing to the SD card? Displaying the file names is OTFDecrytion mounted as a virtual drive? If so would it be possible to list the files (are any Japanese symbols available in the font?) maybe in a differnt color if not transferable to the sd card?

[d0k3]

We don't have any symbols beside ASCII in the font table, meaning we can't display either. I'm still leaving this open, but most likely it won't be fixed, as this game apparently is a special case.

[ghost]

While it may be possible to extend GodMode9's max files per directory limit (currently 1024), for some titles this issue will still crop up. The FAT32 file system used by the 3DS has several limitations to be aware of:

• Maximum number of file entries in a single directory: 65,534
• Maximum number of file entries on the entire disk: 268,435,437
• Maximum file size: 4GB

In the modern day, each individual file can take anywhere between two to thirteen entries, depending on the length of the file name, so you don't necessarily need 65,534 or 32,767 files to hit the limit. In Stella Glow's case, the "sound/stream" folder in the romfs hits the limit with <16500 files due to long file names.

Unfortunately, this is a brick wall that can't be bypassed or climbed over, it's a limitation of the FAT32 file system itself; the only option for titles which exceed this limit is showing an error message explaining why the romfs cannot be opened.

[dogtopus]

I am experiencing the same issue with Conception2. In my case the game has 23744 entries (directories and files, accroding to wc -l of romfs listing produced by ctrtool) and ~70% of them are voice files (not very surprising) IMO there are probably other voice heavy games that may also have this issue

[ghost]

@dogtopus, an "easy" way to determine if a game contains a directory with enough files to hit the limit is to copy the romfs.bin from GodMode9 to your SD card, copy it to your PC, extract it with ctrtool, then try to copy the extracted directory to your 3DS SD card. If there is a directory that hits the limit, the transfer will fail at some point and throw an error code. On Windows, this error code is 0x80070052.

[dogtopus]

@Hikari-chin Just tried to extract the romfs to an empty fat32 image and it went through without any error

[Myriachan]

@d0k3 "This may not be fixable, cause I need to keep FAT compatibility, and FAT has no standard way of supporting japanese symbols." FAT long filenames are encoded as UTF-16.

[windows-server-2003]

Anyways, at least about 20*20 is needed for font size to display full Japanese chars. It's too big.

[Moire9]

@windows-server-2003 Well, if he is able to at least get them to show up, he can get them to display as something like [?], or something to denote that an unknown character is there. That's better than panicking when you find an unknown character.

[d0k3]

Yeah, @Myriachan is right. I think it's UTF-8 though. Displaying unknown chars as '?' is an option, though. There is no panicking when displaying these chars one way or another, btw,

[d0k3]

Alright, you wouldn't believe, but I've got something for you: https://f.secretalgorithm.com/E3xBI/godmode9-v1.6.0-16-g02f5abb8-20180315155453.zip

This build should correctly handle accessing and copying files and dirs with japanese (and other) symbols. It should work for both, RomFS and FAT. While GodMode9 can't display these filenames properly, copied files should be displayed properly on your PC or Mac, when you access the SD card from there.

Unavailable (= not displayable) symbols are right now replaced by '?' - I'm open for suggestions, tho.

The issue with the maximum number of displayable files remeains (working on it), but that really is only a display issue. If you copy a folder, all elements inside it (even if numbering more than 1024) will be copied.

Sooo... @0xC0DEDEAD, @dogtopus and @SirNapkin1334 - are you up for some testing?

[d0k3]

Okay, I tested myself, all looks fine. GodMode9 will never be able to actually display japanese chars and other non ASCII symbols, so the current handling is the best I can provide.

Except for these special symbols all being displayed as '?' and the display limitation (max 1024 files) now, there should be no further issues. I'm still willing to discuss the display limitation.

Closing this now. Further testing is appreciated, you may also reopen (this one or a new issue) if you feel this is not sufficiently fixed.

[Moire9]

@d0k3 I found a similar bug. My flashcart mimics a game with only a single ASCII character, which is Q. The rest is Japanese. (the Q must have some significance, since the icon shows a pixelly girl pointing to a large "Q") When viewing title info for the .nds file when unflashed, it shows only the Q, with all the Japanese simply missing, with a space in it's place. This makes the Q oddly positioned. With the newest commit, it does not fix this, and it still has the same lack of characters. I can send you the ROM to look at (all you have to do is "view title info").

Also, there's a PCX viewer, why not go a little further and add a BMP viewer, so you can look at your screenshots? Also, "snap???.bmp" is a bit vague, perhaps you could make it "[$DATESTAMP]_[$TIMESTAMP].bmp", or something along those lines.

[windows-server-2003]

Also, there's a PCX viewer, why not go a little further and add a BMP viewer, so you can look at your screenshots?

GM9's screenshot bmp format is very very simple, but there are some compression method for bmp which we should support too. I suppose they are complicated.

@d0k3 Each Japanese characters is replaced with 3 '?'s because it takes 3 bytes. Maybe you can fix ?

[d0k3]

Okay, reopening this so we can sort the remaining stuff out.

@SirNapkin1334 - noted, and yeah, we can fix this. Also, date-time names for screenshots are a great idea, thank you. BMP viewer, idk. You know these screenshots are actually larger than both screens combined? I'd need to add scrolling, which would be slow, etc... thinking about it, but as of now I'd rather not.

@windows-server-2003 - if we add compression to screenshots, we won't do so by using some badly supported BMP subformat but rather by introducing PNG. The files are not that big, though, so I'm undecided.

As for japanese characters being multiple symbols long - yeah, that' not pretty. It wouldn't be too hard to fix if it was a limited problem, but filenames in UTF-8 format turn up basically everywhere, we have functions for calculating the length of a string, for truncating strings (that would be especially nasty to handle), etc. Not pretty. If we'd convert each string before doing the respective operation, things would get notably slower. Also not good. What I could to is to represent these chars differently, so that it's 3 chars wide, but it's still apparent that it is only one char. Got any idea how to do this?

[Moire9]

@d0k3 I'll send you the flashcart's NDS to look at, since I just realized it's not actually the game, just stolen title info lol. Unfortunately, I'm now stuck because GM9 always freezes on [root] when I try to enter C: to get the ROM.

EDIT: Removing the flashcart and blowing in the slot makes it work again, but I think the former actually solved the issue, but I blew in it for good measure.

[Moire9]

@d0k3 I don't have a complete understanding of the character system, but what you could try to potentially do it set it that the "unknown" character is extra wide like the japanese, and contain a ? in the middle, but allow the two sides (which have nothing) to overlap with other characters, which would then display overtop/with the nothing, showing only that one character it was overlapping with. This is kind of a weird idea, I don't know if it would be possible or solve the problem without creating more problems, but it's an idea. The character would look somewhat like this:  ?  (note the oddly large spaces around the question mark).

[d0k3]

@SirNapkin1334 - well, that suggestion would make handling alternative fonts really difficult, almost impossible. I'm closing this now, but I'm open for other suggestions.