Does Steam Tinker Launch generate INI files on Special K's behalf?

[Kaldaien]

Recently Special K's INI parser became more strict.

It has always been Unicode, and assumes that anything without a Byte Order Marker is UTF-8, and converts from UTF-8 to UTF-16 internally. In past releases it would silently allow characters in INI files that have no valid Unicode representation to succeed (even though the converted INI file is invalid) and just toss those characters out (dumb, I know 😛).

It seems the first known issue related to the stricter text encoding rules has something to do with this software. The user who reported the problem seems to have ghosted, and I am reviewing the source code, but cannot find anywhere that it actually writes out an INI file.

If I could locate that code (assuming it is even a thing), I would very much like to ensure it is using UTF-8 or, ideally, UTF-16LE. UTF16-LE uses more disk space, but it is faster to parse, always includes a Byte Order Marker and does not have to be converted internally in SK's INI parser.

It is also possible that WINE simply has a bug in MultiByteToWideChar (...), but many other users have run the new version of SK on Steam Deck without issue. I need to narrow down the possible failure points, so wanted to ask directly if Steam Tinker Launch generates INI files and if so, where can I review that code at?

[Kaldaien]

Yes, evidently it does, I've looked closer now. It appears Steam Tinker Launch doesn't understand Unicode text files, which is going to be a problem :-\

[Compatibility.General]
UsingWINE=true
ÿþ

 [ C o m p a t i b i l i t y . G e n e r a l ] 

 A s y n c I n i t = t r u e 

It's prepending this UsingWINE=true stuff to a UTF-16LE INI file, which means it's destroying the Byte Order Marker.

The prepended stuff is UTF-8 or ANSI (can't tell cause there aren't any non-Latin characters), then we see the Byte Order Marker, which doesn't translate to valid Unicode characters, so SK's INI parser bails out.

---

I'd say

1) Do not add UsingWINE=true to SK's INI file, all versions of SK since May of last year can auto-detect this 2) Something needs to be done about handling Unicode INI files, you can't go adding text that's the wrong encoding to the beginning of an INI file, especially when doing so erases the Byte Order Marker that's there to indicate the text encoding in the first place.

[sonic2kk]

Apologies for the delay, I have been taking a mental health break from public projects, but will attempt to clarify a little here and hopefully get a resolution. :slightly_smiling_face: Thank you for taking the time to investigate what's going on and giving some useful insights!

This sounds like a nasty issue. If I'm understanding correctly, the issue here i, SteamTinkerLaunch is creating an INI file with the wrong encoding. It has been a hot minute since I've touched the SpecialK-related functions, and much of it was initially written almost 3 years ago now by a previous maintainer, but I will do my best to shed some light here!

SteamTinkerLaunch creates a blank INI file using the Special K DLL name, if it does not already exist, using touch. This is done in prepareSpecialKIni.

Once the INI file is created, the function then calls a writeValueToIni function to write out some values to the file.

This whole process uses Bash to create and write to the file.

• touch is used to create a blank file at the specified path.
• sed is used to update an existing entry (https://github.com/sonic2kk/steamtinkerlaunch/blob/6fc2a35a9661a2ecbeb6820a9573f769927bbde8/steamtinkerlaunch#L10086).
• If some content/section does not exist, it echos it out into the file (https://github.com/sonic2kk/steamtinkerlaunch/blob/6fc2a35a9661a2ecbeb6820a9573f769927bbde8/steamtinkerlaunch#L10096-L10100).

It seems like when files are created by Bash (either with touch or with something like { echo "this is some text" } >> myfile.txt), they are US-ASCII by default. I tested this on my Desktop PC with my locale as en_GB.UTF-8, and on my Steam Deck with my locale as en_US.UTF-8. When creating text files with touch examplefile.txt, file -i (and encguess) both report that it is US-ASCII.

The SpecialK INI files that I have on my system are reported as UTF-8, but I guess this may have been done by SpecialK in the past, if was more lenient in parsing files, it would've gotten through to the stage where it converted them?

However, Bash should preserve the encoding of a file it processes. For example, writing out to the SpecialK INI files that I have on my system, they are kept as UTF-8. I also tested creating a blank file with Vim and using :e ++enc=utf-16le, and the file encoding was set correctly, and Bash was still able to write out to it (using echo and also with sed), and the encoding did not change in my quick tests.

Therefore, I wonder if when we create a blank SpecialK INI, we could use a command to convert the blank file to UTF-16LE. STL could do the following:

• After touching a blank SpecialK INI, convert the encoding to UTF-16LE (should be safe to do with a blank file).
• Check the SpecialK INI encoding and if it is not UTF-16LE, backup the existing file and re-create a fresh one as UTF-16LE.

But I'm not sure if this will work, since I get the impression that when STL writes out to the file in your tests, it borks it entirely:

It's prepending this UsingWINE=true stuff to a UTF-16LE INI file, which means it's destroying the Byte Order Marker.

To me this sounds like writing to the file that is already UTF-16LE is not working properly, which is quite odd and something I'd like to try and replicate.

It appears Steam Tinker Launch doesn't understand Unicode text files,

I would like to understand this better what this means, to understand what needs fixing. If this is still related to writing out to the INI file, then that's fine, but if there's something else here I want to make sure we're on the same page.

Usually, things like this are up to the system to process properly. If a system locale is not set to use UTF-8, things can be parsed incorrectly and lead to crashes (which was historically common for Arch users that didn't configure their system properly, speaking from experience of myself with my first Arch install and dealing with bug reports here). One good example with SteamTinkerLaunch is with a change I made to how game names are parsed from Steam. Well over a year ago now a change was made to process "real" game names as they are shown in the Steam library, including with Unicode characters. But if a system does not have a UTF-8 locale configured correctly, this can lead to a crash.

This was even an issue on SteamOS 3.3, but Valve have cleared it up as far as I know.

It would be interesting to know if this user was perhaps having a locale issue, but as they have ghosted, it is hard to know (very common theme I deal with here too, but thanks for trying to follow up regardless, I appreciate it).

Do not add UsingWINE=true to SK's INI file, all versions of SK since May of last year can auto-detect this

Awesome! This can be removed then. I had even left a comment in the code that SpecialK may automatically detect this now too, but no one reported back to confirm.

---

In short, STL is creating files using bash, with no explicit work going on to set/alter encodings. And the story is much the same with writing to files. Encodings should not be getting changed by Bash in my experience, and my hope is that we can fix this by changing the encoding after we touch the INI.

I would like to test on my side to see what's going on. Would using a SpecialK nightly with a fresh game be enough to reproduce this issue? That should create an INI with the wrong encoding and SpecialK should be unable to parse it. Although I'm not sure what the symptoms of being unable to parse it would be, would it be a game crash, or is there some logging I should look for?

That way, if I can see the problem is causes, I can try testing if my proposed change of ensuring the INI that STL generates is UTF-16LE will work.

EDIT: Updated links to point to latest commit instead of directly to master.

[sonic2kk]

Did a quick test with some test files and iconv -t utf-16le -o ininame.ini ininame.ini converts files. Most usages I can see specify an encoding to go -from, but it doesn't seem necessary, although if it is we could parse it from file and fall back to encguess.. It seems safe on some SpecialK INIs that I had which were UTF-8, but I'm still cautious of converting people's INI files like this, it may be safer to just back them up and then run this command on the SPEKINI name after touching it.

But again, that may not solve the core problem, if simply writing to an existing SpecialK INI with UTF-16LE causes issues too...

EDIT: There is an untested branch with an initial implementation of what I envision for enforcing UTF-16LE in SpecialK INI files (spek-ini-encoding-change). It may serve as a step to solve this issue.

[sonic2kk]

It's been a couple of months and no further issues have been reported here. I'll keep the branch around in case it's needed if this becomes a widespread issue in future.