String encodings continue to be one of the hardest things
When we get a string for an item name from the game, it might be
in one of several encodings:
ascii
Latin-1 1252, aka ISO-8859 most likely the 9 variation
UCS2, which is almost like UTF-16 LE except it's fixed width
We must take this varied input and turn it into valid UTF-8 strings
for Rust usage & logging, and for display in the HUD via Imgui. We
must also not disturb any already-valid strings by mangling them while
attempting to decode them into a modern encoding.
So, we do the following. First, use chardet::detect() to guess
at an encoding. If it's utf-8 already or one of the ISO-8859s,
we can decode immediately and return. If it is not one of those,
we make the assumption that it is UCS2. The string might be
almost-correctly detected as UTF-16le if it uses the second byte
for any characters, but if the characters it is holding are plain
old ascii, it'll be reported as ASCII and that is wrong. So we guess,
and then decode the UCS2.
Broke the string manglers out into their own file and wrote some
basic tests. Changed the helper wrappers to use cstr null-terminated
version for item name handling, since those come from the game with
null termination. If I discover the null termination is also
double-width I guess I'll have a bug to fix.
Put mcm-meta-helper to work on the task for which it was invented.
Added better logging for when fmtstr fails on huditems.
String encodings continue to be one of the hardest things
When we get a string for an item name from the game, it might be in one of several encodings:
We must take this varied input and turn it into valid UTF-8 strings for Rust usage & logging, and for display in the HUD via Imgui. We must also not disturb any already-valid strings by mangling them while attempting to decode them into a modern encoding.
So, we do the following. First, use
chardet::detect()to guess at an encoding. If it's utf-8 already or one of the ISO-8859s, we can decode immediately and return. If it is not one of those, we make the assumption that it is UCS2. The string might be almost-correctly detected as UTF-16le if it uses the second byte for any characters, but if the characters it is holding are plain old ascii, it'll be reported as ASCII and that is wrong. So we guess, and then decode the UCS2.Broke the string manglers out into their own file and wrote some basic tests. Changed the helper wrappers to use cstr null-terminated version for item name handling, since those come from the game with null termination. If I discover the null termination is also double-width I guess I'll have a bug to fix.
Put mcm-meta-helper to work on the task for which it was invented.
Added better logging for when fmtstr fails on huditems.