[BUG] Japanese text displayed as garbled characters

[onimitch]

Ashita Version (Type /ashita in-game to get this.) v4.1.0.2

FFXI Version (Type /version in-game to get this.) Latest retail. (/version didn't work in-game).

Windows Version (ie. Start > About Your PC) Windows 10 Home 22H2 OS build 19045.3448 64bit

Describe The Bug In IMGUI components at least, Japanese text is not displayed correctly. It appears as garbled characters.

Steps To Reproduce Option 1: Run client in Japanese. Install tCrossBar (https://github.com/ThornyFFXI/tCrossBar) and open the bindings window to see a list of spells/abilities. They will all have garbled characters. Alternatively, install HXUI (https://github.com/tirem/HXUI) and cast a spell to see the cast bar. The name appears in the bottom left and it will be garbled.

Option2: It might be quicker to reproduce by simplying grabbing a spell name from AshitaCore:GetResourceManager and using index 1 (2 in lua) to force the Japanese name rather than english. I assume you don't even need to change the client language if you do this.

Screenshots See below. The label field is displaying the Japanese text. The Action drop down is displaying the English. The macro field is also using Japanese in the quotes.

Additional Information Obviously I don't know how Ashita is coded internally, but I did fix a similar issue in Thorny's tRenderer (used for tCrossBar and tHotBar) by simply adding a setLocale call, which ensures char -> wchar are parsed correctly (e.g when you call swprintf_s or similar method).

https://github.com/ThornyFFXI/tRenderer/pull/1/files

#include <locale.h>
// Ensure we convert japanese text correctly when we call swprintf_s
setlocale(LC_ALL, "");

It'd be really interesting to see if that's all that is needed to fix it for IMGUI as well. It feels like a long shot but I was surprised it worked for tRenderer, so we might be lucky.

I don't mind testing a version of Ashita with that added to verify if it works.

FWIW I'd also be happy to contribute to the development and investigate myself, but I understand you don't know me + the source is closed so I appreciate you probably don't want to do that.

[atom0s]

This is now addressed and fixed. It will be included in the next update of Ashita which is planned to be released soon. This does come with some caveats though.

1. ImGui does not include a font, by default, that supports Japanese glyphs. Ashita also changes the default font used to a different one (Agave) which also does not support Japanese glyphs. Because of this, a secondary bundled font has been added to support the Japanese glyph ranges. The font we picked is NotoSans CJK - Regular as it is a high quality font, includes all needed glyphs to support the Japanese text used by the game, and is free/open source. You will be required to set this font yourself using something like the ImGuiStyle addon for the time being until we add a feature later to set the default font in a different manner.

2. ImGui only accepts UTF8 encoded strings. However, the Japanese strings stored within the DAT files used by the game are ShiftJIS encoded. This means you will need to convert the strings to proper UTF8 before passing them to ImGui in order for them to display properly. At this time, we do not plan to include a means to do this conversion within Ashita. An addon library may be included at a later date but is not considered a priority at the moment. This is something you can find examples of around the web though and is pretty easy/straight forward to do.

3. Addons cannot define new fonts that make use of these glyphs for now. While addons do have the ability to load additional fonts from disk/memory, they currently do not have access to mark the font to be loaded with the proper glyph ranges to support Japanese text. This means that unless you code your own plugins, addons will not be able to load additional Japanese fonts for the time being. You will be stuck using the NotoSans font otherwise. (This is planned to be added, and may be added within the next update.)

[onimitch]

Amazing, thanks. Looking forward to checking out in the next release.