Song folders with non-ascii characters in their name are ignored without error on Windows

[DeinAlptraum]

Actual behaviour

Songs with non-ascii characters in their folder name are not picked up by USDX on my Windows install. E.g. usdb ID 28636 "Die Schlümpfe - Kleine grüne Geister" does not show up in-game for me, and also does not leave an error in the error log. Replacing the 'ü's by 'u's in just the folder name now produces the following error in the log:

ERROR:  Failed to open D:\karaoke\songs-copy\Die Schlumpfe - Kleine grune Geister\Die Schlumpfe - Kleine grune Geister.txt
ERROR:  AnalyseFile failed for "D:\karaoke\songs-copy\Die Schlumpfe - Kleine grune Geister\Die Schlumpfe - Kleine grune Geister.txt".

Note how the paths show even the txt file's names with 'u's in stead of 'ü's - although I haven't renamed these!

In other cases it just displays question marks, i.e. after renaming "Żywiołak - Studzianki" to "Zywiolak - Studzianki" I get:

ERROR:  Failed to open D:\karaoke\songs-checked\Zywiolak - Studzianki\?ywio?ak - Studzianki.txt
ERROR:  AnalyseFile failed for "D:\karaoke\songs-checked\Zywiolak - Studzianki\?ywio?ak - Studzianki.txt".

I also tried to let the directory traversal step print all the files it finds, here is part of the results:

"The Fox" and "Teenage Dirtbag" show what it looks like when it works, i.e. displaying the folder itself, but also all its contents. For the others, it just displays the folder, i.e. it doesn't find any files inside (before renaming the folders). Also notice how the Wolfgang Petry song is apprently somehow transliterated - the actual folder name is "Wir sind die Größten", not "Wir sind die Grosten".

If I rename both the folders and the files within to remove any non-ascii characters, they show up in-game and can be played without any problems.

This doesn't seem to affect everyone on Windows, other people have reported being completely fine. I can also play the exact same files (shared folder on a dual boot) without trouble on my Linux install.

Expected behaviour

The songs can be played without any folder/file renaming, and even if they can't, an error can be found in the log.

Steps to reproduce

Install USDX, then put any song with non-ascii characters in its name into the song folder. Start the game, see that the song doesn't show up.

Details

Provide some additional information:

• USDX version: 2024.3
• Operating System + version: Windows 11

[bohning]

I can only add that the problem doesn't occur on Mac either, so it's really Windows only.

[RumovZ]

I can confirm this. I've tested with Let 3 - Mama ŠČ (ESC 2023 Croatia) on 2024.5, Windows 10.

[barbeque-squared]

I cannot reproduce it the way @DeinAlptraum describes it. I can reproduce it with what @RumovZ commented. Specifically (at least on my system) it's the Č character that's causing the issue. I can also confirm that it does not cause an issue on Linux with exactly that same character.

The weird Š, eszet, o-umlaut, various dashes, none of those cause any issues for me, it's that C that's triggering it for me. You can just take any working txt (+accompanying files), add a Č somewhere in the txt filename, and tada it gives an error on AnalyseFile (I assume because it can't find the file).

NB: it's possible this character doesn't trigger it on German systems, but should hopefully fail more consistent on English systems.

What I find really weird is that that S and C are in the same Unicode block, but it's totally fine with the S but not the C.

[bohning]

@s09bQ5 If you are on Windows, could you maybe look into this issue?

[s09bQ5]

No Windows here.