After many trials I came to the same conclusion as Goldberg and probably everyone else.
Using the wide char version of std::wstring is definitely avoidable and useless, it screws up the encoding so bad and other standard libraries like std::filesystem don't even handle it well.
The regular std::string is multi-byte capable and perfectly fitting for that case
Using the ANSI versions of Win32 APIs (ex: GetModuleFilenameA) is certainly avoidable and actually cannot handle multi-byte strings (at least by default)
The wide char versions of the Win32 APIs can work with multi-byte strings, but they are utf-16 encoded
The only viable option is to always work with std::string in your application and convert to std::wstringonly before calling the wide char version of the APIs
After getting a wide-char (utf-16) encoded string back from Win32 APIs, always convert them back to narrow/single byte (utf-8) encoded strings (std::string) and continue with your application
After 2-3 years I've re-discovered why Goldberg made the 2 functions utf8_encode and utf8_decode :/
There's actually no other way around this problem on Windows. Unicode encoding has a lot of bad history and compatibility problems, both in C++ runtime and in Win32 APIs. It is agonizing.
This PR fixes all of that + the original problem where the overlay language wasn't being changed:
Compare the language value of steam_settings/achievements.json as case-insensitive, for some reason the supported_languages schema contained the language value as latam (small case), but the achievements schema contained the language value as LATAM (upper case), hence the lookup always fails and the emu defaults to either English or the first available language if English isn't an option
Compare language in the overlay as case-insensitive
In generate_emu_config script make sure each line of the supported languages is lowercase
Introduce a helper function to convert between utf-8 and utf-16 via the library utfcpp and use them everywhere possible + use them in-place of utf8_encode() and utf8_decode()
Introduce a new object-based synchronized logger, without dependency on emu specific stuff, now shared between cold client loader and the emu code base, also could be used later anywhere if needed
Rewrite cold client loader code to use SimpleIni library like the emu + use multi-byte std::string for all paths + use the new logger
Fix a recurring problem where logs are never generated when the game/app path contains non-Latin chars, now it could handle anything ( hopefully :/ ), tested on a path like this C:\test\命定奇谭ğğğğğÜÜÜÜ:
By moving the game files to that path and pointing at the game folder back from C:\test\
By moving everything inside C:\test\命定奇谭ğğğğğÜÜÜÜ, both the emu log file STEAM_LOG.txt and cold client loader log file steamclient_loader_x64.exe.log were generated successfully and the data looked properly encoded
Also tested dll injection, and of course the dll itself was inside C:\test\命定奇谭ğğğğğÜÜÜÜ
Improve the usage of std::string_view, passing them as const ref is useless since they don't allocated anything and cheap to copy (no allocations)
Refactor the code a little to extract common stuff
After many trials I came to the same conclusion as Goldberg and probably everyone else.
std::wstring
is definitely avoidable and useless, it screws up the encoding so bad and other standard libraries likestd::filesystem
don't even handle it well.std::string
is multi-byte capable and perfectly fitting for that caseGetModuleFilenameA
) is certainly avoidable and actually cannot handle multi-byte strings (at least by default)std::string
in your application and convert tostd::wstring
only before calling the wide char version of the APIsstd::string
) and continue with your applicationAfter 2-3 years I've re-discovered why Goldberg made the 2 functions
utf8_encode
andutf8_decode
:/ There's actually no other way around this problem on Windows. Unicode encoding has a lot of bad history and compatibility problems, both in C++ runtime and in Win32 APIs. It is agonizing.This PR fixes all of that + the original problem where the overlay language wasn't being changed:
steam_settings/achievements.json
as case-insensitive, for some reason thesupported_languages
schema contained the language value aslatam
(small case), but the achievements schema contained the language value asLATAM
(upper case), hence the lookup always fails and the emu defaults to either English or the first available language if English isn't an optiongenerate_emu_config
script make sure each line of the supported languages is lowercaseutfcpp
and use them everywhere possible + use them in-place ofutf8_encode()
andutf8_decode()
cold client loader
code to useSimpleIni
library like the emu + use multi-bytestd::string
for all paths + use the new loggerC:\test\命定奇谭ğğğğğÜÜÜÜ
:C:\test\
C:\test\命定奇谭ğğğğğÜÜÜÜ
, both the emu log fileSTEAM_LOG.txt
and cold client loader log filesteamclient_loader_x64.exe.log
were generated successfully and the data looked properly encodedC:\test\命定奇谭ğğğğğÜÜÜÜ
std::string_view
, passing them as const ref is useless since they don't allocated anything and cheap to copy (no allocations)Closes #239