Lack of Japanese Character Support

[KevinX8]

All the homebrew I have tried that uses libnx has issues with reading files with Japanese characters off the sd card. As far as I can tell the issue stems from lack of non ASCII FS support on libnx.

[yellows8]

"with reading files " details?

[KevinX8]

Ftpd is unable to read any files with Japanese characters, they simply do not show up in a directory listing, and easyrpg player cannot read Japanese files either, I've spoken to the devs of rpg player and they narrowed down the problem to libnx aswell.

[CTCaer]

japanese **filenames***

[fincs]

Are you sure your filenames are encoded and parsed as UTF-8, and that your FTP client supports it? Please note that libnx merely passes through filenames to fsp-srv. There aren't any sort of variant service calls that differ in the filename encoding, the only encoding that fsp-srv purportedly supports is UTF-8.

[KevinX8]

The files are definetly encoded in utf-8 and my ftp client definetly also supports Japanese characters as it is the very one I used to place the files on the sd card. A sample of the name of one that ftpd and other homebrew fail to detect is "各フォルダの説明.txt" I checked the encoding and its definetly utf-8. Folders with japanese characters also fail to show up in all homebrew that uses libnx.

[KevinX8]

Turns out the fault is actually the switch, it does not encode files in UTF-8 but in SHIFT_JS instead and the only way to get around this is use archives.

[fincs]

After some further investigation by @hexkyz, we've found out that Nintendo's higher level code in the FS sysmodule assumes path names are UTF-8 and parses them as such, however something goes wrong when it passes down path names to the lower level internal FAT driver they're using, and it ends up misinterpreting them as having Shift-JIS encoding. This looks like a bug in Horizon OS. Unfortunately we don't know if Nintendo will ever fix this, or if we could even somehow report it to them :\