Incorrect fonts for Simplified Chinese and Traditional Chinese

flower-elf commented 2 weeks ago

Describe the bug?

Currently, the game uses Japanese fonts to display these languages, which makes some characters appear very strange. Could change the font to noto-sans-sc (for Simplified Chinese) noto-sans-tc (for Traditional Chinese) to resolve this issue?

To Reproduce

Use zh-cn or zh-tw Localization,and then you will see the problem.

Expected behavior

When using these localizations, fonts should display correctly, not Japanese glyphs

Screenshots

Correct Fonts: photos

in game: photos

Resonite Version Number

Beta 2024.5.4.678(Steam Store)

What Platforms does this occur on?

Windows, Linux, Android / Quest

What headset if any do you use?

Samsuang MR+

Log Files

No logs.Because it's a display problem.

Additional Context

No response

Reporters

Resonite Username:flower_elf Discord:雾雨花精灵(flower_elf)

Frooxius commented 1 week ago

Unfortunately I'm not familiar enough with Chinese languages, so I don't know how to identify the problem just from switching the locale, I will need more information.

Have you tested this issue with the fonts you specified? Does that actually resolve the problem?

The fonts however are global, it's not something that's loaded based on the locale. I'm a bit confused why would the glyphs map to Japanese characters if they are Chinese and how could the system know that it needs to actually query Chinese version of the font for these instead of Japanese.

E.g. there can be Chinese text displayed even if locale is set to something different and vice versa- it should still be able to display those properly as well.

Are you able to include the actual glyphs as well in the text? You only provided pictures, so I can't copy them and do any sort of testing or analysis.

Also additionally, please provide actual Resonite version, not "latest", because I don't know what "latest" is at the time you made this issue.

ko-tengu commented 1 week ago

The issue here is specifically Han Unification and haunts a lot of unicode projects. The most correct answer is to use specific regional fonts, such as the provided Google examples, with specific culture codes.

https://en.wikipedia.org/wiki/Han_unification

Frooxius commented 1 week ago

Hmm... This makes this issue pretty tricky, because like I mentioned above, our font system is more global - it assumes that distinct characters will have distinct codepoints, so they can be looked up appropriately in the fonts based on that.

If there's two different characters that map to the same codepoint and the only difference is the font, that makes it a bit trickier to handle, as we'll need to make some more substantial changes on how the font system is working.

flower-elf commented 1 week ago

Unfortunately I'm not familiar enough with Chinese languages, so I don't know how to identify the problem just from switching the locale, I will need more information.

Have you tested this issue with the fonts you specified? Does that actually resolve the problem?

The fonts however are global, it's not something that's loaded based on the locale. I'm a bit confused why would the glyphs map to Japanese characters if they are Chinese and how could the system know that it needs to actually query Chinese version of the font for these instead of Japanese.

E.g. there can be Chinese text displayed even if locale is set to something different and vice versa- it should still be able to display those properly as well.

Are you able to include the actual glyphs as well in the text? You only provided pictures, so I can't copy them and do any sort of testing or analysis.

Also additionally, please provide actual Resonite version, not "latest", because I don't know what "latest" is at the time you made this issue.

The version number has now been provided on the issue, sorry about that.

These problematic characters use the same codepoint(differing only in glyphs), so I don't have a way to provide them as text, they vary with the system's regional settings.

As @ko-tengu said.The root of this problem is Han Unification. The current effective solution is to load specific fonts for specific languages (or globally allow users to use custom fonts)

And uhh... how do I load specific fonts within the game that will allow me to show the fixes?

ko-tengu commented 1 week ago

You can drop a TTF file into the game window, and then use it as the font for a text field by dropping it into the Font field in the inspector.

https://wiki.resonite.com/Component:Text

shiftyscales commented 1 week ago

Based on the Frooxius' description, this does not sound like a bug, but rather a shortcoming/limitation of the current system. So I will reclass this issue as an enhancement/feature request.

Could you provide some additional context on the impact this issue has to the users, @flower-elf @ko-tengu?

E.g. does the display of the incorrect characters drastically alter the meaning of the text, or the average users' ability to parse and understand what is trying to be said?

ko-tengu commented 1 week ago

Yeah, it's occasionally different meanings and even under more charitable circumstances it makes it much harder to read, it's like if someone made a font that randomly replaced the letter q with a 9 or an o because they kinda look the same.

Frooxius commented 1 week ago

Are there any other ways to deal with this kind of situation in the practice, other than just using user's system locale?

How does it work if you for example go to say a Japanese Wikipedia and Chinese Wikipedia. Is Chinese Wiki going to show wrong symbols to users who have their OS locale set to Japanese? And vice versa?

I'm trying to think how to approach this, if we could perhaps embed some extra information to the strings to "hint" at the language it's in and prioritize font selection based on that.

Anteness commented 1 week ago

Hi. To figure out the point, I need to quote a feedback thread here from another company's product.

https://feedback.vrchat.com/localization/p/use-hanji-characters-that-match-the-set-language
https://feedback.vrchat.com/localization/p/chinese-simplified-words-type-font-issue For some reason, there is overlap in Unicode for a large number of Japanese & Chinese glyphs, they are hard to path/redirect with a LUT. ( I have a simple research base on GB2312 with 6969 words, about 10-15% of them will met this issue) In the end they did a font swap for Chinese language, for example: when there is a word sharing two type font in Japanese/Traditional Chinese/Simplified Chinese:
If user prefer language is JP, keep it on JP type font;
If user prefer language is SC, render it as SC character. This may result font character mistakes in some mixed Japanese and Chinese text, but for most users it will work as expected.

Frooxius commented 1 week ago

Hmm that solution seems less than ideal to me. Is that really the only approach there is to this problem?

How do websites handle it? For example Wikipedia?

Anteness commented 1 week ago

Probably is the easiest one. I have no idea about how Wikipedia deal with this issue, and as I said there is hard to swap these words one-by-one. (10% for 6969 words means you have over 696 words to find and swap them and it's really very hard to build a table for such this number of word charaters )

ko-tengu commented 1 week ago

Webpages generally handle it by having a culture code embedded in the page and optionally embedded before problematic characters, and having your web browser and OS determine the correct font to use for what language is being requested, the large block example in the middle of the wikipedia article is an example of that.

Yellow-Dog-Man / Resonite-Issues