Add native support for fcitx, include temporary CJK font.

[GrimPixel]

Fcitx is not supported yet.

[drwhut]

What is Fcitx?

[GrimPixel]

It is an input method framework to support typing Chinese, Japanese, Korean. I pressed its shortcut in the game, and it doesn't activate.

[drwhut]

After some research, I found out that fcitx is a form of "IME input".

It looks like it only works in Godot on macOS so far. It looks like Windows and Linux support won't be coming before Godot 4.0 sadly: godotengine/godot#53658

[drwhut]

Looking at the issue mentioned above, IME has just been merged into Godot, so it will be ready for the 4.0 release. So once Tabletop Club uses Godot 4.x, we can close this issue 🙂

[ImBearChild]

I tried using Fcitx5 to type some Chinese characters in this game, but they just don't render, invisible. I can use Ctrl + A and Ctrl + C to copy those invisible and paste them in some other application, where they can got rendered properly. So we can prove that the IME part works well.

Maybe we need some CJK font to make them rendered properly. I'm not sure this is caused by missing font or not, since I'm not familiar with Godot Engine, but there is an article points out how to fix this problem on Godot Engine. [1]

Adobe and Google provides a set of open source CJK fonts called Source Han Sans.[2] And if you consider Source Han Sans is oversized, another open source font called WenQuanYi is also a good choice.[3]

By the way, I also submit some Chinese translation (the "game" part is 100% now) on weblate. But it's meaningless if we can't render Chinese character.

[1] https://godotengine.org/qa/3540/unable-to-enter-the-chinese [2] https://fonts.adobe.com/fonts/source-han-sans-simplified-chinese [3] http://wenq.org/en/

[drwhut]

I tried using Fcitx5 to type some Chinese characters in this game, but they just don't render, invisible. I can use Ctrl + A and Ctrl + C to copy those invisible and paste them in some other application, where they can got rendered properly. So we can prove that the IME part works well.

Interesting... out of curiosity, what operating system are you running?

Maybe we need some CJK font to make them rendered properly. I'm not sure this is caused by missing font or not, since I'm not familiar with Godot Engine, but there is an article points out how to fix this problem on Godot Engine.

Yeah, this is correct. Funnily enough, this very thing was discussed recently in #279.

Adobe and Google provides a set of open source CJK fonts called Source Han Sans.[2] And if you consider Source Han Sans is oversized, another open source font called WenQuanYi is also a good choice.[3]

This is the tricky bit when it comes to including CJK support in the game. I would prefer to use Source Han Sans simply for future-proofing (for when Japanese and Korean translations are added), but the font size for the "main" variant is about 10x bigger than the game itself 😱

One idea I am considering is to make it so when a CJK language is picked in the options menu, a pop-up comes up offering the game to automatically download Source Hans Sans if it has not already been downloaded. This would unfortunately require an internet connection for those who want to use CJK languages, but it would also mean the final build sizes stay relatively small. I might potentially make a separate issue for this to see what people's opinion are on this.

By the way, I also submit some Chinese translation (the "game" part is 100% now) on weblate. But it's meaningless if we can't render Chinese character.

Thank you so much! ❤️ Hopefully the translations will be included in the game sooner rather than later, I just need to decide the best course of action before potentially commiting huge font files to the git repo.

[ImBearChild]

what operating system are you running?

I'm using Fedora Linux 38 (Kinoite) and flatpak version of this game.

but the font size for the "main" variant is about 10x bigger than the game itself

Since relaying on system builtin fonts is not very reliable, at least we can try to make fonts files smaller. Instead of shipping huge original form (those otf or ttf files) of Soure Han Sans, we can use woff2 format. Woff2 is smaller and better for size sensitive case. ^woff And Godot support this format.^support_woff And official release of Source Han Sans contains a woff2 file. ^release Just unzip this file, and you will find it at Variable/WOFF2/OTF/SourceHanSans-VF.otf.woff2. It's just 12.9 MB, not that big. The release file is big, because it include different formats and variations we don't need at all.

I would prefer to use Source Han Sans simply for future-proofing (for when Japanese and Korean translations are added)

And using WenQuanYi Zen Hei is not a bad idea. It's a Chinese font, but it support Korean and Japanese characters too! Following picture shows some Chinese, Korean and Japanese characters rendered with WenQuanYi Zen Hei. And it's smaller than Source Han Sans even in the format of woff2. I put an attachment here, it's 4.3 MB unzipped for most common CJK characters. Its size is acceptable. WenQuanYi Zen Hei contain less characters than Source Han Sans, some special symbols and rarely used characters are missing. But since we're not printing a publication here, a little imperfection is acceptable (at least it's okay for myself).

WenQuanYiZenHeiMono.zip

This would unfortunately require an internet connection for those who want to use CJK languages

Well, this may cause issues for some users in mainland China. This is complex to explain, but in short, China government have the access and bandwidth limited for people who wants to download something from other country. Common infrastructures like Github and Google CDN are some kind of banned in mainland China too. ^the-filewall So it will be extremely slow or even impossible to download original form of Source Han Sans without a proxy. And proxy support for this game is another topic, right?

In conclusion, it's possible to include a small font file (4 - 12 MB) in a 70 MB sized game. 😃 And thank you for making this good game. My friend and I really had a good time in it.

[drwhut]

Instead of shipping huge original form (those otf or ttf files) of Soure Han Sans, we can use woff2 format. Woff2 is smaller and better for size sensitive case. 1 And Godot support this format.

Oh interesting, I had no idea this format existed! Thank you for bringing this to my attention, will give this a try when testing the Chinese translations.

And using WenQuanYi Zen Hei is not a bad idea. It's a Chinese font, but it support Korean and Japanese characters too! Following picture shows some Chinese, Korean and Japanese characters rendered with WenQuanYi Zen Hei. And it's smaller than Source Han Sans even in the format of woff2. I put an attachment here, it's 4.3 MB unzipped for most common CJK characters.

4.3 MB?? Well, this puts the whole issue of font sizes to bed 🤣 I'll have a look into the kinds of characters that aren't included, but I guess if a translation down the line includes an unsupported character, it could be patched in from another font (i.e. make a font with one character)?

Well, this may cause issues for some users in mainland China. This is complex to explain, but in short, China government have the access and bandwidth limited for people who wants to download something from other country.

Ah yeah, I hadn't considered the Great Firewall of China... luckily, if the WenQuanYi font is suitable, this won't be an issue.

In conclusion, it's possible to include a small font file (4 - 12 MB) in a 70 MB sized game. 😃 And thank you for making this good game. My friend and I really had a good time in it.

Agreed 👍 Glad you enjoyed the game, and thank you for sharing your insight!

[ImBearChild]

I'll have a look into the kinds of characters that aren't included

I don't think we will encounter those missing characters in Chinese. Back when Fedora was Fedora Core (I mean 10 years ago), WenQuanYi is the almost standard Chinese font of all Linux distributions, because it was the only open source Chinese sans font we can use. And it is capable for daily use. It contains more than 27842 Chinese characters (in 4 MB!), having GB2312 (mainland China standard ^gb2312) and Big5 (Taiwan standard ^big5) covered, while The Table of General Standard Chinese Characters only contains 8105 of those characters. ^std_chn_table I can sure that all existing Chinese translation strings are good with WenQuanYi.

What I'm not sure is Korean and Japanese characters, but after using WenQuanYi on several articles from Korean Wikipedia and Japanese Wikipedia, I found no missing ones. So this problem is not very likely to happen.

In fact, WenQuanYi also contains some Latin characters and Cyrillic ones. Not all of them but some common ones. So it's possible to use WenQuanYi as only font in this game (If you want fonts in game looks more uniformed,and current Latin font and Cyrillic font have notable size and thickness different). I can confirm that all Russian translation in base game and default assets packs can rendered with WenQuanYi Zen Hei Mono. And "Ǎ, Ě, Ǐ, Ǒ, Ǔ" (that missing in Cabin ^missing_cab) are also included.

Firefox is a handy tool when checking font support. Its developer tool will tell user what kind of font is using on the page. So we can modify CSS to use single font, and see if there's second font shows in the list as fallback. If there's no fallback, the font is capable for character on the page.

it could be patched in from another font (i.e. make a font with one character)?

Yes, although this situation is not likely going to happen, It's possible to use tools like FontForge ^font_forge to deal with it. We do not have to create a new font file, instead, we could copy missing characters from other fonts and paste it into our own version of WenQuanYi. And Making a new font is possible though.

What's more, you can directly use the attached version of WenQuanYi above. I download official WenQuanYi font and used FontForge to convert it into WOFF2 format. It's purely organic. 💚

By the way, the friend that I mentioned above now have the Simplified Chinese translation of Default Assets Pack done. With all Simplified Chinese parts in game are done here, we can use OpenCC to convert strings in Simplified Chinese to Traditional Chinese, so that we do not need to make Chinese translation twice.

[GrimPixel]

About that font topic, the problem of “Han Unification” exists. In short, multiple glyphs are assigned a same code, so there is only one variant available. Examples are https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs

For other writing systems, there is one font for all languages; for Han, if Source Han Sans are not the choice, then there would be fonts from different families, which means visual inconsistency.

There is a list of fonts: https://polyglotclub.com/wiki/Language/Multiple-languages/Culture/FLOSS-Fonts

Godot should be supporting system fonts now: https://docs.godotengine.org/en/latest/tutorials/ui/gui_using_fonts.html#doc-using-fonts-system-fonts https://github.com/godotengine/godot-proposals/issues/306

[drwhut]

As far as I understand it, the best way to resolve the unification problem would be to use a variant font, but unfortunately Godot only supports variant fonts from 4.0 onwards - it may be some time before Tabletop Club is able to take advantage of them.

So for the time being, it may be best to include a temporary font just so that all of the necessary characters can at least be rendered, the smaller the font size the better. The glyphs will be wrong for some languages, but it's better than them not showing up at all I would say, and this will eventually be fixed when Tabletop Club uses Godot 4.0. Although it's worth pointing out I can't really judge if the inconsistent glyphs would be too glaring for native speakers, I would have to get people's opinion on this.

Something else I saw that might be worth mentioning is that while the .woff2 format is supported in Godot 3.5, it is not supported in Godot 3.4 - so depending on what format the font files are distributed in, it may not be possible to include a highly-compressed font file with the game before v0.2.0.

[ImBearChild]

As far as I understand it, the best way to resolve the unification problem would be to use a variant font.

Did you mean "variable font" ?[^vf]

Something else I saw that might be worth mentioning is that while the .woff2 format is supported in Godot 3.5, it is not supported in Godot 3.4

Web Open Font Format 1 (.woff) is supported in Godot 3.4^woff1-support. It's also a compressed format, compared to TTF and OTF. Theoretically, it's possible to convert Source Han Sans from WOFF2 to WOFF 1, but Fontforge seems not work well with Source Han Sans (no matter what format, Fontforge just says this file is damaged 😢 ). Maybe some other tools can do this job.

So for the time being, it may be best to include a temporary font just so that all of the necessary characters can at least be rendered, the smaller the font size the better.

WenQuanYi is good in this situation, I guess. 5.9 MB uncompressed for WOFF1, 10.8 MB uncompressed for TTF. Font files here^wqy-zip, not very big.

I can't really judge if the inconsistent glyphs would be too glaring for native speakers, I would have to get people's opinion on this.

Inconsistent glyph is not a big problem for native speakers of these languages. Basically, these multiple glyphs refer to same character. For example：

Upper line is Japanese glyph, lower line is Chinese glyph. They're similar. And people in CJK area are already able to handle these difference. We can understand them by comparing them with locally used glyph and guessing its meaning with context. It's not a big deal since we got a working brain on our shoulder, and for whom without a working brain, they won't play Tabletop Club, so it's not a problem at all.

We (I mean Chinese players here) usually do not expect CJK fonts working perfectly in games, and even in open source operation systems. Just a month ago, Fedora replace original Source Han Sans with new variable font format ^fedora-crazy , and breaks almost every Qt applications, making those application show wrong font type, but still readable (Qt do not support variable font now ^no-qt, but those fedora guys totally ignore people who using Qt and KDE and push this to end users).

So if big projects like fedora can mess up with CJK font, you can too. CJK font stuff is complicated, it is understandable to leave some flaws here. It's good to make it perfect, but imperfection is acceptable.

Games like Minecraft probably never do this right (Minecraft is using Bitmap font converted from GNU Unifont without considering variant, obviously not perfect), but nobody complains about CJK font. X4:Foundation even will not render CJK font when selecting non-CJK language (Thinking about loading saving from Chinese Players, and see only a lot of question marks).

Believe me, using curved line font in game and handle variant properly is some kind of advanced technology for game developers. Without this technology, those games still get sold. Not a big deal, really.

[^vf]: A variable font (VF) is a font file that is able to store a continuous range of design variants. https://en.wikipedia.org/wiki/Variable_font

[GrimPixel]

Hanazono can display them well. HanaMinA covers several standards of common characters. It is 22.8 megabytes.

[ImBearChild]

Hanazono can display them well. HanaMinA covers several standards of common characters.

Did you mean Hanazono Mincho (花園明朝) ? It is a serif font (明朝体，宋体 in Chinese), not sans (黑体). I don't know should we combine CJK serif font with existing Latin sans font or not. 🤣

[GrimPixel]

文泉驿正黑 is under GPL2 license, which doesn't fit here. 文泉驿微米黑 is 5.2 megabytes. It is small enough and looks more polished than 文泉驿正黑 on high-resolution screens. Its download address is https://sourceforge.net/projects/wqy/files/wqy-microhei/0.2.0-beta/. Its problem is, it doesn't support many letters with caron. Source Sans Pro, Inter, Nunito, among others, have better diacritics support. Noto Sans supports at least 806 langauges.

[GrimPixel]

Ah, I just noticed that the title is changed. Fcix is not working on the AppImage version, still.

[drwhut]

There... is no AppImage version? :thinking: Do you mean the standalone build?

[GrimPixel]

Oh, yes, it feels like AppImage but it isn't.

[ImBearChild]

文泉驿正黑 is under GPL2 license, which doesn't fit here.

Why not? I'm not a lawyer, but as far as I can tell, GPL2 is safe when you just bundle it with your application. It's not the core part of Tabletop Club can can be replaced if needed. So many Linux distribution put Wenquanyi Zenhei (文泉驿正黑) in their software collection and remote repo, and not causing all parts of the system going into GPL.

If Wenquanyi Zenhei (文泉驿正黑) is not possible, Droid Sans is a decent replacement. It's not very big, and licensed under Apache License. 10 MB uncompressed for all language (at least all language in android). Chinese part of this font is contained inside DroidSansFallbackFull.ttf.

google-droid-fonts-20200215.zip

[drwhut]

Why not? I'm not a lawyer, but as far as I can tell, GPL2 is safe when you just bundle it with your application.

The GPL is strictly a copy-left license, meaning if you include it in something the "something" has to also be at least copy-left, which the MIT license isn't (it's more permissive than the GPL). If it was licensed under the LGPL, then that would be OK, since it makes an exception for being used as part of another project.

Ah, I just noticed that the title is changed. Fcix is not working on the AppImage version, still.

I have a theory as to why it works on my system, but not yours. I've noticed in both Tabletop Club and Final Fantasy XIV that when I use fcitx to type in Japanese, a small pop-up appears on the bottom-left of my screen containing the current "progress" of text. When I press enter, the pop-up disappears and the text appears in the game. I think this is something that Fedora has added on by default, because when I try the same thing in Fedora's default text editor, it does the character conversion within the application itself.

So my theory is that your system does not have this generic pop-up that mine does, which does not allow you to use fcitx in applications that do not natively support it. Looking at the issue I mentioned near the start of this thread, it looks like support has been added to Godot 4.0, so it may be some time before fcitx is supported natively in Tabletop Club.

[ImBearChild]

The GPL is strictly a copy-left license

As far as I know, GPL allows developer to "aggregate" it. Only ship it with a program or product won't cause GPL infection. Obviously, hardcoding font matrix into code will make your program GPL, but it's possible to ship it with your program and use it as it is. Commercial Linux distributions can ship it with non-GPL software or even non-opensource software.

Developers of Wenquanyi have written an faq for this situation:

如果字体仅仅是程序可以使用的一种外部字体之一，可有可无，没有包含在程序的其他的数据当中，那么在其他条件都满足的情况下，合成的程序则可以有独立的授权声明。
Translated: If that font is one of external fonts program can use, and it's optional, not contained inside other data of this program, with all of other requirements met, combined program can use independent license.

So I don't think that using a GPL font will cause license issue. It's not combined into the code as a binary blob, just a part of file we can use later, clearly separate. Another example is NVIDIA Linux Kernel driver, working with GPL code under the same memory space, with some distributions ship it in their installation media. But people's definition of "external", "data" and "separate" may vary ... it's also reasonable to stay away from it ,if you want to avoid those potential legal problem. I'm clarifying my opinion here, and we can just not use Wenquanyi.

Wenquanyi MicroHei and Wenquanyi Zenhei are both GPL. Wenquanyi MicroHei is derived from Droid Sans Fallback, and Droid Sans Fallback is licensed under Apache License (from Android). In my opinion, using Droid Sans is a acceptable option, since it won't cause license issue and relatively small.

[GrimPixel]

It's not the problem of the pop-up, but is that fcitx can't be activated: even after pressing the shortcut of activation or clicking on its icon on the panel.

文泉驿正黑 suits low-resolution screens as its strokes are overall not concentrated. It is aethetically inferior comparing to 文泉驿微米黑 on high-resolution screens. The license of 文泉驿正黑 is at http://wenq.org/wqy2/index.cgi?ZenHei_FightingState_README The license of 文泉驿微米黑 is at http://wenq.org/wqy2/index.cgi?MicroHei_BigBang_README

[drwhut]

I've just noticed that they've added a font exception in their GPL license:

Appendix B. 

GPL with font embedding exception:

http://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#FontException

As a special exception, if you create a document which uses this
font, and embed this font or unaltered portions of this font into 
the document, this font does not by itself cause the resulting 
document to be covered by the GNU General Public License. This 
exception does not however invalidate any other reasons why the 
document might be covered by the GNU General Public License. If you 
modify this font, you may extend this exception to your version of 
the font, but you are not obligated to do so. If you do not wish to 
do so, delete this exception statement from your version.

EDIT: After further investigation, this may only apply to documents in the sense of Word, Excel, etc. rather than programs.