Add Ukrainian alphabet

[andrewjswan]

Add Ukrainian alphabet to Fonts

https://en.wikipedia.org/wiki/Ukrainian_alphabet

[trip5]

Thank you for opening this issue but I realize now that I've run into a brick wall as far as BDF files are concerned...

I've been digging into the code of ESPHome and Pillow (the Python library that converts fonts into bitmaps that ESPHome can use) and I finally found the reason that ESPHome can't work with the unicode characters I've added into the font. I had assumed it was ESPHome's fault but...

There's legacy code in Pillow that dates back to 2003 or earlier (!) that means in a bitmap font, anything above character 255 can't be converted. I'll refer you to this post here, if you're interested: https://github.com/python-pillow/Pillow/issues/5124

But there is a solution, too. The next step is actually to convert these BDF files to TrueTypeFont with embedded bitmaps. I'll see what I can do and hopefully you'll see an update soon enough. In the meantime, try using a TTF font with the characters you need...?

[andrewjswan]

It is very strange that TTF fonts work without problems, but BDF fonts do not, although pillow is used in both cases. I am of course ready to try TTF fonts, or support Issue to fix the bug with Unicode in BDF fonts.

[trip5]

I've been playing around and will continue to do so... I've already found that's easy enough to convert a BDF to a TTF file that contains only bitmaps, making it unusable in Windows but perfect for Unicode in the clock.

The problem I'm encountering is I'm actually unfamiliar with baselines and heights and how Fontforge works with them.

Saving the BDF as a TTF file is also basically a one-way street. They're unusable to edit since Fontforge seems to consider a TTF without scalable fonts as empty.

As for Pillow supporting Unicode BDF files? That post is 3 years old. The only way it's going to get fixed is for someone to dig into that section of the Pillow code and fix it themselves... and even then, I tried updating my ESPHome docker to the newest Pillow (10.0.1) and ESPHome told me to downgrade back to 9.5.0... so the new code might take awhile to get into ESPHome. It took them until just this summer to even update Pillow at all. The version they were using previously was something like 4 or 5 I think (many years old).

In my adventure inside the ESPHome docker container, I also discovered that the default install of Pillow doesn't have something called "RAQM" which allows Pillow to work with bidirectional text, and according to the notes I found in the code, support for "non-latin characters" which....

I'm trying Arial right now (yeah the one that comes with Windows) and I can't seem to get the clock to display Unicode characters from Home Assistant either. And Arial has a huge number of characters so... this issue might be bigger than I thought.

So... If you have some time, perhaps it might be an issue on my end... do you wanna give it a shot?

Use a font you know has Ukrainian characters (I'm very certain Arial Regular has almost unicode characters except the CJK set) and make sure your glyphs section has the characters you want to test (ESPHome defaults to a very small set of roman characters if you don't specify this). Send the new config to your clock and see if you can get it to display Ukranian. I usually use a Home Assistant service call for an icon screen but whatever...

Let me know what you discover. I'll be heading back into my modified container later and see if adding RAQM support works.

[andrewjswan]

Use a font you know has Ukrainian characters (I'm very certain Arial Regular has almost unicode characters except the CJK set) and make sure your glyphs section has the characters you want to test (ESPHome defaults to a very small set of roman characters if you don't specify this). Send the new config to your clock and see if you can get it to display Ukranian. I usually use a Home Assistant service call for an icon screen but whatever...

On graphic displays, displaying Ukrainian letters with TTF fonts happens without problems. Do you think there is a difference in ESPHome, which display is used for output? Matrix or Graphics or Led matrix based?

[trip5]

I'm laughing a bit right now at my misunderstanding of that issue I linked to over at the Pillow Github. I genuinely thought it referred to the ASCII limit. Partly, it does.

The pillow font file created by the conversion is limited to 255 characters... but the font descriptors used in BDF don't match what Pillow is looking for so it fails to convert correctly. I'll try converting the BDF to PIL using a utility (but I'm not sure ESPHome will recognize a PIL file as a valid font file)... or using just using TTF.

As for the Arial test, I had 2 problems: my glyphs was going over the 255 character limit and I was calling the test using the wrong font... I was able to get my clock to display some Ukrainian characters (ЗЖ) and my Unicode won character (₩). So I have full confidence I can add more Unicode characters... as soon as I figure the conversion workflow. I think I have to fix the baseline of existing characters, then I'll get on with creating Cyrillic characters.

[andrewjswan]

I would help, but I opened the BDF in fontforge, tried to open any character to edit, but fontforge showed me an empty window and suggested drawing the character from scratch, I got frustrated and didn't try any further. :)

[trip5]

Oh my. I've had no such issue with Fontforge. Usually I just open the BDF directly, edit it, then export it again.

In any case, looks like I've got a working TTF and even discovered the reason why Pillow version 10.0.0 is not being added to ESPHome... my test string "₩123ЗЖ789" was appearing on my clock as "₩12 3 ЗЖ78 9 " as if double-byte characters are interfering with something... No such weirdness with a default ESPHome container (Pillow 9.5.0). It functions perfectly. Looks like I've figured out how to remove the baseline for my X fonts, too.

Stay tuned.

[andrewjswan]

Oh my. I've had no such issue with Fontforge. Usually I just open the BDF directly, edit it, then export it again.

Reinstalled again, it seems to be working. I just need to understand how to work with it and whether there is a restriction in the height of characters of 8 points.

[trip5]

Well, technically there's no restriction in height... actually characters can go waaay beyond their set margins... For the 6-series fonts, a lot of the accented characters actually start below the baseline so they can use the full 6 pixels available, if that's what you mean. Baseline actually doesn't matter much for single-line clocks... it might be a problem if you need to display 2 lines at once. It would look all mashed together.

Actually I've been putting in some serious work on the BDF files to make sure they actually follow proper standards and get converted to TTF correctly. I've managed to remove the baseline for the X-series fonts and I made sure they all can convert to TTF correctly...

I've begun work on the Cyrillic characters, though I'm unfamiliar with how the characters should appear... I mean for example an A (English or Cyrillic) can be made as narrow as 3 columns - I mean, they're basically the same shape! But for example, I'm not sure if Д actually needs to have those little marks below the baseline as they appear here and on https://en.wikipedia.org/wiki/De_(Cyrillic). And I wonder if Ж can be curved a bit to keep it a bit narrower. So, for now I'll work on the Chunky8 font and get you to take a look at it and make some changes (if that's OK with you) and then I take a shot at Chunky6 and see if there's any issues.

By the way, something I had to learn the hard way was when saving a BDF file again, choose "No Outline Font" & "BDF" & "AGL for New Fonts" and you'll get a pretty decent output... though the new file might have to be renamed.

[trip5]

I've added a file in the 8-series if you can take a look at it... see if I got most of the Cyrillic characters right. Let me know. I'll finish out the rest of them later (mostly accented characters I think).

[andrewjswan]

I'm not sure if Д actually needs to have those little marks below the baseline as they appear here and on https://en.wikipedia.org/wiki/De_(Cyrillic). And I wonder if Ж can be curved a bit to keep it a bit narrower.

8

6

To make it look the same, you can make the letters И and Й a little wider, or leave И the regular width and make Й wider, like:

[andrewjswan]

As possible options other letters, but do as you like :)

[andrewjswan]

I've added a file in the 8-series if you can take a look at it... see if I got most of the Cyrillic characters right. Let me know. I'll finish out the rest of them later (mostly accented characters I think).

I'll try it tonight. Should I try the BDF? Or TTF? Font? Thanks!

[trip5]

They're the same. You can use the BDF to look at them in Fontforge or put the TTF to your clock. Thanks for the pics. That actually helps a lot. I'll do a little update right now.

[trip5]

BTW the pics you're showing me here is pretty much what I'll do for Chunky8X - and helps a lot. For now, its based on Chunky8 where there's still a baseline and only some ticks and tails below that... It's updated now. If you take a look at them, you might laugh... I stopped at that point because I realized I was working on "traditional" Cyrillic characters, which according to Wikipedia are no longer in use, lol. I've probably already added enough to cover Ukrainian, Russian, Belarusian... maybe others?

[andrewjswan]

maybe others?

I don't know, but if I were you I would start with your native alphabet, if I understand correctly, it's not English.

If you take a look at them, you might laugh... I stopped at that point because I realized I was working on "traditional" Cyrillic characters, which according to Wikipedia are no longer in use, lol.

It's a shame to waste time on something that is no longer in use.

BTW the pics you're showing me here is pretty much what I'll do for Chunky8X - and helps a lot.

If you need more help, write. :)

[andrewjswan]

put the TTF to your clock

This font? Did I get that right? https://github.com/trip5/Matrix-Fonts/blob/main/8-series/MatrixChunky8-Cyrillic.ttf

[trip5]

That's the one.

[andrewjswan]

There are slight problems with some of the letters, maybe I didn't connect the font correctly?

  - file: fonts/MatrixChunky8-Cyrillic.ttf
    id: default_font
    glyphs:  |
      ! "#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz°|АаБбВвГ㥴ДдЕеЄєЖжЗзИиІіЇїЙйКкЛлМмНнОоПпРрСсТтУуФфХхЦцЧчШшЩщЬьЮюЯя
    size: 8

My test string:

Аа Бб Вв Гг Ґґ Дд Ее Єє Жж Зз Ии Іі Її Йй Кк Лл Мм Нн Оо Пп Рр Сс Тт Уу Фф Хх Цц Чч Шш Щщ Ьь Юю Яя Ёё Ъъ Ыы Ээ

I tried tweaking, some of the letters ( Д Ж Л М У Ц Щ Ы Э б д ж л м у ф ц щ і ї І Ї Є Э) . And removed the tails hanging down from the letters. They were cut off at the letters, I don't know if they interfere or not, but if they don't interfere you can leave them at the letters - Д Щ Ц ф ц щ

But I don't know how to save to TTF correctly. font.zip

[andrewjswan]

Ґ ґ added ... MatrixChunky8-Cyrillic.sfd.zip

[trip5]

I'm comparing the changes you made. Yup, I'm definitely not familiar enough with Cyrillic to be doing this alone. Thank you.

Namely, I didn't realize the У character isn't the same as y. But lol I can even see it here as I type. The missing pixel fits the Light version better... I feel the Latin y and Cyrillic У basically have the same shape just with different height.

Some of the changes I'm curious about...

Are the curves necessary in certain characters?

Your example pic above shows a "chunky" way of doing Ж and then we can put the little curve you did in the Light version of the font. Would that work?

Also Э and Є... the ones you fixed seem more suitable for the Light version of the font. For the Chunky version, which would be better? Something like the Euro symbol (similar, not same) which keeps the curve (4 pixels wide)? Or even just the same shape as the number 3 (3 pixels wide)? I guess I'm wondering how necessary that curve is.

By the way, I see you like the serif on the i-related letters. Yes, I like that and I should put it back in the Latin i's as well. I think I removed it at some point for space but it really does look better on the Chunky font.

[andrewjswan]

Your example pic above shows a "chunky" way of doing Ж and then we can put the little curve you did in the Light version of the font. Would that work?

It's the same, but with the ticker Ж looks more readable. But in this case you can do it both ways, but you can't do it in М, because there is T, which in capital letters is sometimes spelled like Ш upside down.

As for the У, it's not bad on the screen, but it shouldn't be a Ч. :)

Also Э and Є... the ones you fixed seem more suitable for the Light version of the font. For the Chunky version, which would be better? Something like the Euro symbol (similar, not same) which keeps the curve (4 pixels wide)? Or even just the same shape as the number 3 (3 pixels wide)? I guess I'm wondering how necessary that curve is.

I am not an expert in fonts, I like most of all your font Chunky (though I do not know what it is), but when you display information, some characters want to lighten, they look heavy compared to their neighbors. So I made changes that would not seem alien when viewing.

Э and Є are completely mirror symbols, they are similar to the sign € only one stripe in the middle, and does not protrude on the other side, simply a semicircle, and from its center a straight line.

[andrewjswan]

You can leave it like I did, or you can change it to something like this:

[andrewjswan]

If anything, write what symbols you are interested in I will try to draw all possible variants in the right size.

[trip5]

changes that would not seem alien when viewing

That's exactly what a font designer thinks, I think. I'm hardly an expert. A lot of this project has had constant changes. I kind of feel bad thinking someone downloaded the font months ago and hasn't seen the improvements because they didn't check my github.

I like your your Є as you show here... but it feels like a "Light" version.

How does this one look for Chunky?

[trip5]

Sorry for the delays. Lol, curves are always annoying when there's only a 7 pixel height to work with.

Oh and I never answered you about the tails. Since this is an 8 pixel font, one row is reserved for tails. So we can keep them for now like this:

When moving this to Chunky8X and LightX, I will definitely follow your idea because the X fonts have no room for tails (or the letter simply gets raised)... your file you attached earlier shows me how to fix that nicely... unlike some of the non-English Latin letters which absolutely need the accent below the baseline so the letter had to be raised up.

By the way, is this OK for Chunky?

And I can do this one for Light...

BTW it's strange to me that I have 8 fonts on this repo... I only use Chunky8X and Light8X on my own clocks. I have an older Mini Awtrix in the Living Room and my wife complained that other fonts were hard to read.

[trip5]

Oh lol and I just got what you mean with М and Ш... so I can use an upside-down W as Cyrillic М. Zero confusion. Actually, I'll out the inverted W for Latin too...

[andrewjswan]

How does this one look for Chunky?

Super!

[andrewjswan]

Oh and I never answered you about the tails. Since this is an 8 pixel font, one row is reserved for tails. So we can keep them for now like this

Good!

[andrewjswan]

By the way, is this OK for Chunky?

It's not great, it doesn't really fit the display of the letter Л

[andrewjswan]

Л Sorry, Missed the boundaries :)

[trip5]

Got it. I put it in exactly like that. I've also added a bunch of probably-not-Ukrainian characters. The Cyrillic set is pretty similar in size to the Latin-1 set. I might finish it up over the weekend then get Cyrillic into the rest of the 8 set.

Please tell me this character is uncommon: Ksi (Ѯ, ѯ) lol. Wikipedia says it's an old character. It's impossible to render this decently on 8 pixels. On 6 it's going to look like a blob, I'm sure.

I'm about to turn in for the night... some last-minute changes to the i's and widening the H's... everywhere. Narrow H (Latin or Cyrillic) looks too narrow.

[andrewjswan]

Please tell me this character is uncommon: Ksi (Ѯ, ѯ) lol. Wikipedia says it's an old character. It's impossible to render this decently on 8 pixels. On 6 it's going to look like a blob, I'm sure.

It's unused, and has been for a very long time.

I.e. it is not needed, the Ukrainian alphabet has 33 characters, the Russian alphabet has the same.

I'm about to turn in for the night... some last-minute changes to the i's and widening the H's... everywhere. Narrow H (Latin or Cyrillic) looks too narrow.

I thought you were in Republic of Korea and thought it was almost morning..... :)

[andrewjswan]

I think this would be better.

[trip5]

Haha. It was 7am. I am worse than a night owl sometimes.

Yup. It does look better. I forgot it's not a Latin N, it's a vowel in Cyrillic. Number 3 & 4 are short vowels, right? A dot or a 2 pixel bar can replace a grave accent... and shrink the width of и. I'm applying that rule to the others.

[trip5]

There are slight problems with some of the letters, maybe I didn't connect the font correctly?

  - file: fonts/MatrixChunky8-Cyrillic.ttf
    id: default_font
    glyphs:  |
      ! "#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz°|АаБбВвГ㥴ДдЕеЄєЖжЗзИиІіЇїЙйКкЛлМмНнОоПпРрСсТтУуФфХхЦцЧчШшЩщЬьЮюЯя
    size: 8

My test string:

Аа Бб Вв Гг Ґґ Дд Ее Єє Жж Зз Ии Іі Її Йй Кк Лл Мм Нн Оо Пп Рр Сс Тт Уу Фф Хх Цц Чч Шш Щщ Ьь Юю Яя Ёё Ъъ Ыы Ээ

Sorry I forgot about this. I'll test out the Cyrillic on my own device now... Thanks for the examples... I can just copy and paste and see with my own eyes.

As for this...

But I don't know how to save to TTF correctly.

Here's how to save a BDF (because a TTF becomes unusable): ...and yes, it tends to add a -8 to the filename because that's the point size... I'm constantly deleting & renaming files when saving BDF.

When it asks BDF Resolution, select "Guess" and it works perfectly.

And here's how to save a TTF useable by ESPHome:

I've re-adjusted the M again on all fonts, including the Cyrillic one (it's obvious now, I'm old enough to remember DOS). I'll re-upload after I test the Cyrillic. It could be a problem with my baseline (again).

[andrewjswan]

Number 3 & 4 are short vowels, right? A dot or a 2 pixel bar can replace a grave accent... and shrink the width of и. I'm applying that rule to the others.

They're different letters, roughly like: ㅈ ㅊ And the lettering is about the same as I drew above.

[andrewjswan]

I've re-adjusted the M again on all fonts, including the Cyrillic one (it's obvious now, I'm old enough to remember DOS). I'll re-upload after I test the Cyrillic. It could be a problem with my baseline (again).

DOS and I remember perfectly well, so you can update the BDF/TTF in the repository I'll try to save it in TTF and test it on my screen.

[10der]

@andrewjswan hi, what's a tools name for converting BDF files?

а що за тулза для конвертації BDF? дяк!

[andrewjswan]

what's a tools name for converting BDF files

Fontforge

[10der]

what's a tools name for converting BDF files

Fontforge ще раз вельми дякую!

[trip5]

Lol. How do you know Korean letters? Yup, those are soft J (voiced) and soft Ch (unvoiced) in Korean.

And yeah it's Fontforge. Uhhh... I'm getting an error from ESPHome that my TTF has an invalid pixel size... umm... lol.

My instructions may be wrong. I did it before and now I screwed it up... working on it!

[andrewjswan]

an invalid pixel size

Add size: 8 to yaml

How do you know Korean letters?

The kid is studying Korean philology at university :)

[trip5]

That's neat. The Korean alphabet is kind of unique and super-easy to learn. Korean language is not easy (for me).

And lol at my rookie mistake. With TTF files you MUST specify the size of the font in the yaml, whereas BDF you shouldn't. It works just fine. But you're right. The tails are missing completely. That's weird. It probably has something to do with the way the BDF file is interpreted by Fontforge then that improper configuration is converted wrong.

I actually had to dive into the actual BDF specs and fix some things just so it would work at all... I'll keep working on it. It might have something to do with the BDF underline height.

[andrewjswan]

I'll be waiting, once it's ready, I'm ready to check it out.

[10der]

@andrewjswan cool! wow

[trip5]

Oh, that looks pretty good! With you, this looks really good. I'll tell you what. I'll sync the Github now and you can see even better. Some of them looks really good. Cyrillic is now in the main Chunky8 and Light8 files. There are slight differences, but especially in rounded characters. Especially take a look at the different Дд, Єє, Жж, Зз, Іі, Її, Лл and tell me what you think.

Especially the Іі & Її in the Light font. Does the Arial-style work here?

Sorry delay... I accidently deleted a character. In an effort to remove the old Cyrillic, I just wiped a bunch of characters lol... Ґґ and some others had to be put back in.

And I figured out what was going on. The BDF 8 size files need a yoffset of 8 but the TTF files need a yoffset of 7.

  default_font_id: default_font
  default_font_yoffset: 7
  special_font_id: special_font
  special_font_yoffset: 8

So, adding that note to the readme... But I'm thinking I should just get rid of anything below the baseline and change the height of all files to match the actual number of rows available. Thinking...

Anyways, syncing the Github now. Take a look.

[trip5]

Or maybe I should just ditch the idea of baselines completely and focus only on the full height of 6 or 8 pixels.

Hm.

[andrewjswan]

cool

WIP :)

[andrewjswan]

Or maybe I should just ditch the idea of baselines

I think the tails can be left in the font, if they are not displayed, it won't be a problem, the font is readable.