Better CJK support

[AJenbo]

This is for tracking issues noticed with displaying the Chinese translation, some of these may affect other languages as well.

• [ ] https://github.com/diasurgical/devilutionX/issues/3548
• [ ] https://github.com/diasurgical/devilutionX/issues/3549
• [ ] https://github.com/diasurgical/devilutionX/issues/3538
• [x] All symbols in the small fonts have been left aligned, but CJK symbols should be centered as they are not variable width
• [x] Item prefix blue string is not exposed to translators, change it to fmt to allow translators to reorder it as well ({:s} {:s} -> {1}{0} = "Golden Plate" -> "PlateGolden")
• [x] Line breaks can still be limited for languages that rarely have separator symbols (`,.,?,!,　,，,、,。,？, and！`). https://github.com/diasurgical/devilutionX/pull/3438 https://github.com/diasurgical/devilutionX/pull/3528 https://github.com/diasurgical/devilutionX/pull/3541
• [x] Store / talk text overlap. https://github.com/diasurgical/devilutionX/pull/3461
• [x] https://github.com/diasurgical/devilutionX/issues/3547 https://github.com/diasurgical/devilutionX/pull/3584
• [x] https://github.com/diasurgical/devilutionX/issues/3603
• [x] https://github.com/diasurgical/devilutionX/issues/3621
• [x] Re render font in high quality
• [x] Add texture to small font

[AJenbo]

@tytannial this is the best place to report issues and as well as report any that you find.

[AJenbo]

Please see https://github.com/diasurgical/devilutionX/pull/3171 for the chosen solution to long lines not being split automatically.

[AJenbo]

@tytannial {0} {1} -> {0}{1} and {0} of {1} -> {1}{0} has been applied, please verify.

[tytannial]

Quest review progress

Diablo Quest

• [ ] Poisoned Water Supply
• [x] Skeleton King
• [x] The Butcher
• [ ] Ogden's Sign
• [ ] Gharbad the Weak
• [ ] Magic Rock
• [x] Arkaine's Valor
• [ ] Halls of the Blind
• [ ] Zhar the Mad
• [ ] Black Mushroom
• [ ] Anvil of Fury
• [x] Warlord of Blood
• [x] Lachdanan
• [x] Chamber of Bone
• [ ] Archbishop Lazarus
• [ ] Diablo

Hellfire quests

• [ ] The Defiler
• [ ] Farmer's Orchard
• [ ] Grave Matters
• [ ] The Jersey's Jersey*
• [ ] Little Girl*
• [ ] Na-Krul
• [ ] Reconstructed Note

[glebm]

The 。 in the small font is misaligned:

[AJenbo]

@tytannial @nsm53project @bubio please have a look at the new CJK fonts: https://github.com/diasurgical/devilutionx-assets/releases/download/v2-dev1/fonts.mpq

Note that you will need the latest test build for the font width to be correct for the small size.

The 。 in the small font is misaligned:

@glebm Should be fixed now :)

[bubio]

@AJenbo It's looking really good! However, there seems to be some overlap.

[AJenbo]

However, there seems to be some overlap.

@bubio

This is separate from the font it self and is being worked on in https://github.com/diasurgical/devilutionX/pull/3461

For the dialogs the text needs to have ZWSP (or other break points) injected to allow the render to properly wrap the text. See https://github.com/diasurgical/devilutionX/pull/3438 for how that was handled for Chinese.

[glebm]

The new big Japanese font looks nice!

There is something odd about the spacing around katakana ("ケイン" below):

      ケインと話す

[tytannial]

I think one of the font table need adjust...

地 回 器 在 生 堂 祝

[nsm53project]

Something different :) And small font looks pretty good!

It would be good to reduce the letter spacing in dialog font

Like this.

[AJenbo]

Something different :)

What am I supposed to notice with the marked ones?

And small font looks pretty good!

It would be good to reduce the letter spacing in dialog font

Like this.

Is Korean not full width like Chinese?

[nsm53project]

What am I supposed to notice with the marked ones?

Shadowing effect? I don't know the exact term. Anyway, It's a little different from the other letters, but it's not a big deal :)

Is Korean not full width like Chinese?

Korean(phonogram) uses fixed width or variable width in some cases. There are some differences from Chinese(ideogram). The current font looks like 'T h a n k g o o d n e s s y o u ' v e r e t u r n e d !' to Koreans.

[NikoVP]

Shadowing effect? I don't know the exact term. Anyway, It's a little different from the other letters, but it's not a big deal :)

I have the files so I can fix it easily as well the same effect in the Chinese blocks. I'll pass the files to @AJenbo when ready. @nsm53project can you paste/write here the problematic symbol here so I can identify which block it belongs to.

[qndel]

군 괴 from "Thank goodness you've returned" ogden speech

[AJenbo]

I see that full width isn't necessarily the same across all symbols. It's more to be taken as meaning they have a square appearance. I'll investigate the width

Korean(phonogram) uses fixed width or variable width in some cases.

How would I know when it should be fixed or variable width?

[AJenbo]

The new big Japanese font looks nice!

There is something odd about the spacing around katakana ("ケイン" below):

ケインと話す

I'm not following, i guess this isn't just down to different letter spacing. What range are these symbols? Should they be variable or fixed width for the group?

When I compare them they appear to have the same width as Chinese, which should be how they are rendered by the engine:

[nsm53project]

How would I know when it should be fixed or variable width?

Except in special cases(like 8bit classic games), variable width is usually used.

[AJenbo]

What you are showing here is actually that Korean symbols are always a fixed width: The only reason the lines end up with a different length is the punctuation marks, which are not part of the Korean symbol range.

From what I can gather elsewhere, the Korean symbols are fixed width, but narrower then Chinese symbols. I'll realign AC00-D7A3 (Hangul Syllables) separately from other symbols and hopefully that will give the desired result.

[AJenbo]

@nsm53project does this spacing look correct to you?

Before After

[nsm53project]

The only reason the lines end up with a different length is the punctuation marks, which are not part of the Korean symbol range.

I didn’t explain properly. I should have explained the difference between full and half width, fixed and variable-width.

In Korean fonts, Korean symbols: full width / fixed or variable width (mostly fixed width) punctuation marks: half width / fixed or variable width (mostly variable width)

[nsm53project]

Before After

Nice! It would be good to reduce it a bit more.

I want this feeling. from the Witcher 3 Korean ver.

[AJenbo]

Can you write the text so that I can put it in game and compare.

The thing is that if I make it any more narrow then some letters combinations will start to overlap

[nsm53project]

Here it is :)

잊었어? 반 아드의 전시회에서부터였지. 네가 지겨워 미쳐 버릴 것 같다며 그걸 시작했잖아. 그때 난 네 유머 감각이 형편없으면서도⋯ 이상하게 사랑스럽다고 생각했어.

[glebm]

The new big Japanese font looks nice! There is something odd about the spacing around katakana ("ケイン" below):

ケインと話す

I'm not following, i guess this isn't just down to different letter spacing. What range are these symbols? Should they be variable or fixed width for the group?

When I compare them they appear to have the same width as Chinese, which should be how they are rendered by the engine:

The space between these katakana characters is really large, or perhaps the characters are too small. Compare to the Ubuntu rendering. /cc @bubio

[glebm]

Here is another example from the spell book: スピリット

[AJenbo]

The alignment looks correct, so they are either more narrow or there is additional spacing where you are taking this screen shot from.

Ours: https://github.com/diasurgical/devilutionx-assets/blob/main/assets/fonts/12-30.pcx?raw=true

Original: https://unifoundry.com/unifont/png/plane00/uni0030.png

As I understand it it should be rendered with the same width as the general CJK symbols, so unless you are suggesting that we narrow the spacing of them then I don't see what can be done here.

For better comparison I would do a line of Katakana and one line of dense Chinese symbols below it so that we get a better comparison of how both are spaces in the same font.

[AJenbo]

@nsm53project

Here it is :)

잊었어? 반 아드의 전시회에서부터였지. 네가 지겨워 미쳐 버릴 것 같다며 그걸 시작했잖아. 그때 난 네 유머 감각이 형편없으면서도⋯ 이상하게 사랑스럽다고 생각했어.

Alright I adjusted things a bit. It's pushing the width a bit, so there are technically a chance that some letter could overlap at this point. So I would like you to give thing an extra careful look to see if things appear correctly in terms of horizontal overlapping.

https://github.com/diasurgical/devilutionx-assets/releases/download/v2-dev2/fonts.mpq (you will also need the latest test build)

[AJenbo]

@tytannial

地 回 器 在 生 堂 祝

This should also be fixed in the latest render:

https://github.com/diasurgical/devilutionx-assets/releases/download/v2-dev2/fonts.mpq (you will also need the latest test build)

[AJenbo]

@bubio

However, there seems to be some overlap.

Things should also start to look good for Japanese at this point:

(I don't know exactly which of Pepins lines you where showing to have issues, so let us know if there is still something there)

https://github.com/diasurgical/devilutionx-assets/releases/download/v2-dev2/fonts.mpq (you will also need the latest test build)

[NikoVP]

To me it seems that the Korean can even take one pixel less without overlapping.

[bubio]

I think it's pretty good. The katakana margins are also a bit wide, but I don't mind that too much.

[glebm]

@AJenbo Ah, the issue in Unifont it seems. Katakana characters are smaller than Hiragana and Kanji for some reason :| The new big font looks good!

[nsm53project]

@AJenbo

[tytannial]

复

[AJenbo]

Please give a description of what you think should be changed about it

[tytannial]

Please give a description of what you think should be changed about it

This character size does not match other. 😖

[AJenbo]

This appears to be in issue in Noto font. If we correct it manually there is a good chance that it will be reverted the next time we re render the font.

[glebm]

@tytannial @AJenbo Regarding 复, this is not a bug in Noto. This is an instance of Han unification. The font is the Japanese version, where this character does look like that. This will be resolved with #3538