Help wanted | Adding Korean language support

[OctopusET]

Hi, I'm trying to add support for Korean language. I have almost done with translate strings into Korean, so I'm trying to build and test it on my Pinecil V2. But I'm getting these errors (Here's my fork build log: https://github.com/OctopusET/IronOS/runs/8073453776?check_suite_focus=true)

Traceback (most recent call last):
  File "/build/source/source/../Translations/make_translation.py", line 1396, in <module>
    main()
  File "/build/source/source/../Translations/make_translation.py", line 1353, in main
    language_data = prepare_language(lang_, defs_, build_version)
  File "/build/source/source/../Translations/make_translation.py", line 600, in prepare_language
    sym_list, sym_lists_by_font, font_map = get_sym_list_and_font_map(text_list, fonts)
  File "/build/source/source/../Translations/make_translation.py", line 458, in get_sym_list_and_font_map
    font_maps = get_font_map_per_font(text_list, fonts)
  File "/build/source/source/../Translations/make_translation.py", line 410, in get_font_map_per_font
    font12_line = get_cjk_glyph(sym)
  File "/build/source/source/../Translations/make_translation.py", line 244, in get_cjk_glyph
    glyph: Glyph = cjk_font()[ord(sym)]
  File "/usr/lib/python3.10/site-packages/bdflib/model.py", line 401, in __getitem__
    return self.glyphs_by_codepoint[key]
KeyError: 12434

I think this is because font file doesn't include Korean characters (Hangul). Font description says it also has Korean font, but there's no Korean font when I checked with font viewer. Or it does have Korean font and I might be done something wrong.

There's another Korean bitmap fonts like (https://sourceforge.net/projects/baekmuk/files/) This could be used to fix this issue.

What do you think about this problem? Thanks.

[River-Mochi]

are you able to pull that Korean Bitmap from Sourceforge into your personal Fork and compile? does it work?

[OctopusET]

I tried build before, Build was failed because that Korean bitmap font does not contain other fonts(Japanese and Chinese). But I haven't try build only Korean. I think it will work, so I will try build and share you results.

[River-Mochi]

we don't know very much but might be able to help a little since we did a different translation a long time ago. potato may be able to at least know which character is being rejected.

for the Sourgeforce, you need to @Ralim to see if he can add that into the IronOS for you to use.

[River-Mochi]

I tried build before, Build was failed because that Korean bitmap font does not contain other fonts(Japanese and Chinese). But I haven't try build only Korean. I think it will work, so I will try build and share you results.

at the very top of the TranslationEditor.html it lets you import other fonts, try to import all of them. but Ralim would know more how this works

[OctopusET]

Can you have live chat rn? Sorry I was doing something I couldn't read your response. And I tried with TranslationEditor.html but it seems not working, so I did translate manually. You could find my translation on lang-ko branch in my fork. And I'm using pinecilV2. Thanks

[OctopusET]

Fonts that might can be used for. Some READMEs might be in Korean, if you want translation. Let me know.

https://github.com/RanolP/dalmoori-font | Apache 2.0 https://github.com/hurss/fonts/tree/master/bdf | MIT License These repo has 5 fonts. https://github.com/quiple/galmuri | This font supports CJK, and there are 7pt and 9pt versions, but it's SIL Open Font License 1.1 https://cactus.tistory.com/193 | It's called 'DungGeunMo' and it's public domain. It does not have any github page. And it's distributed as only otf and ttf. So I converted to bdf. DungGeunMo BDF.zip https://sourceforge.net/projects/baekmuk/files/ | BSD license (Alternative link: https://ctan.org/pkg/baekmuk) https://ctan.org/pkg/unfonts-core | GPLv2 license.

[River-Mochi]

just keep documenting everything here. Ralim is in Australia time. he will see it

[Ralim]

Okay, so I think that yeah the font is missing the code points you will need. If the fonts can be converted to bdf, its relatively simple to swap them out when trying this. Not all of these will look good when rendered at the low resolution (or fit).

Also I cant read any Korean at all so I'm going to be fairly useless at testing. 😁 So I'll play it by ear and go with what you say

But, to make save you a bunch of leaning the mess of the python:

Merge the branch https://github.com/Ralim/IronOS/tree/1397-testing into your branch. Then put your font to test as test.bdf in the Translations folder and add the font korean to the json definition file instead of cjk:

"fonts": [
        "ascii_basic",
        "korean"
    ],

That should then cause it to generate using that bdf file for you to be able to test it, and you can swap in and our bdf files to try different options.

---

Obviously at the end we wouldn't call the font test.bdf but we can fix that up later on 😁

[discip]

Is Korean not already supported as stated here?:

https://github.com/Ralim/IronOS/blob/8cee506ede0ab03c4e01d79e17bdc33e0c2ad51c/Translations/wqy-bitmapsong/README_original#L70-L71

[Ralim]

Some of the korean characters are missing (it throws errors on build as they are missing). The system is throwing errors that the symbols cant be found in that range. So suspect either bdf file is missing them or its a mismatched documentation :/

[discip]

I've tried to implement it in my fork as well, unfortunately without success, and as you mentioned, I'm not able to read Korean glyphs either. Otherwise, my idea would have been to determine the non-matching ones.

[Ralim]

Given the error is for a number in the middle of the range they say is supported I'm not super sure whats up. Will see if other fonts work, if they wont will need to dig more on this font. But ill take the easy win of a new font if it works.

[OctopusET]

I tried build with several fonts, but it fails. Error logs:

Traceback (most recent call last): File "/build/source/source/../Translations/make_translation.py", line 1421, in main() File "/build/source/source/../Translations/make_translation.py", line 1378, in main language_data = preparelanguage(lang, defs_, build_version) File "/build/source/source/../Translations/make_translation.py", line 621, in prepare_language sym_list, sym_lists_by_font, font_map = get_sym_list_and_font_map(text_list, fonts) File "/build/source/source/../Translations/make_translation.py", line 479, in get_sym_list_and_font_map font_maps = get_font_map_per_font(text_list, fonts) File "/build/source/source/../Translations/make_translation.py", line 431, in get_font_map_per_font font12_line = get_korean_glyph(sym) File "/build/source/source/../Translations/make_translation.py", line 254, in get_korean_glyph return get_bdf_glyph(sym, korean_font()) File "/build/source/source/../Translations/make_translation.py", line 40, in korean_font return bdfreader.read_bdf(f) File "/usr/lib/python3.10/site-packages/bdflib/reader.py", line 125, in read_bdf font[key] = _unquote_property_value(values) File "/usr/lib/python3.10/site-packages/bdflib/reader.py", line 65, in _unquote_property_value return int(value) ValueError: invalid literal for int() with base 10: b'Galmuri7' make: [Makefile:535: Core/Gen/Translation.KO.cpp] Error 1 make: Waiting for unfinished jobs....

[Ralim]

Weird, I did test it with one of the bdf files from your second link and it compiled and ran (but looked cut off)

That would make me wonder if the file isnt bdf that your feeding in?

[OctopusET]

Only second link fonts are working. Now build works. I think I should investigate why other fonts are not working.

BTW, there's a bunch of warning

"/usr/lib/gcc/arm-none-eabi/11.2.0/../../../../arm-none-eabi/bin/ld: warning: /usr/lib/gcc/arm-none-eabi/11.2.0/../../../../arm-none-eabi/lib/thumb/v7-m/nofp/libg_nano.a(lib_a-memmove.o) uses 2-byte wchar_t yet the output is to use 4-byte wchar_t; use of wchar_t values across objects may fail"

Is it normal behavior? I'm building it with docker.

[Ralim]

Yep you can ignore those warnings.

Hmm okay, I didnt realise I happened to use the one that works by accident. I can try and look at the others tomorrow

[OctopusET]

Right side of fonts are cut off because font is too big. But I can say it's readable (as a native speaker). Great!

[OctopusET]

It should be printed like "고급 설정" but right side each of the characters are gone.

[discip]

Maybe @alvinhochun is willing to help with this. I think I am not saying too much in crediting him for implementing CJK fonts as such.

[alvinhochun]

I had wondered when this day will come...

Well, for starters, the character size for the large font is 16x12 but for our purpose we need a 12x12 (9pt) Hangul font. I am not sure if WenQuanYi Bitmap Song is supposed to include them but they are apparently not there in the 9pt bdf font file.

Best is to find another font that is compatible with GPLv3. If the file name says 16 it is probably a 16x16 font which will get cut off as you see. Assuming all your characters are inside the Hangul Syllables block and the font contains all of them, it should just work.

After that you may want to fine tune the glyph placement for your font (the current offset is optimized for WenQuanYi Bitmap Song).

[alvinhochun]

Also, given that the screen only has enough space to print 8 of 12x12 characters, Since Korean uses whitespace as word separator unlike Chinese and Japanese, if you do run out of room with menu items, you might want to look into implementing half-width space for this purpose.

Alternatively, if you can manage some non-trivial coding work you can try a 8x8 Hangul font for use with two-line display. The small font for western languages is only 6x8 so we cannot just handle a 8x8 Hangul font like normal -- it will require changes to the font printing functions.

[OctopusET]

Thanks for great advice. Firstly, I will try to find 9pt Korean fonts that compatible with GPLv3. and can build. I don't know why other fonts are not able to build.

And half-width space looks really interesting to implement.

[OctopusET]

Some STARTPROPERTIES on bdf fonts weren't strings that surrounded with double quotes, and they caused build error. After I fixed bdf fonts files, build works. I tried "Galmuri" 9pt font, and it works great. Still fonts are kinda small, and we need to find GPLv3 compatible fonts. Furthermore, as alvinhochun said, there are some ways to improve readability.

[discip]

@OctopusET Have you managed to get this to work? And if so, are you planing on creating a PR for this?

[OctopusET]

I tried some, but I couldn't do it yet. I've been busy for weeks, I might be able to restart work next month but I can't promise.

[discip]

So there's hope, that this will be finished some time. 😊

[River-Mochi]

@OctopusET are you able to contact me Pine64 chat (I am in Discord)? I have an idea to help with korean language and I see you have a Pinecil V2 which is perfect. you may/may not like idea but I think it will work.

[OctopusET]

I'm trying to restart this work. Looks like a lot of things related to the translation have changed.

First, I will try to implement only the 12x12 font with half-width spaces (for compatibility with the other CJK translations). Then I will try to implement the 8x8 font (with dalmuri font). However its readability is not that good.

I will keep update on this topic

[hijae]

If fonts are an issue, I think you might want to try this one. https://kldp.org/node/134250 AGPL v3 License, BDF type are also available.

[OctopusET]

Thank you for the information