search

lvgl / lv_font_conv

Converts TTF/WOFF fonts to compact bitmap format
https://lvgl.io/tools/fontconverter
MIT License
183 stars 79 forks source link

Desired features #1

Closed puzrin closed 5 years ago

puzrin commented 5 years ago

Later:

puzrin commented 5 years ago

@kisvegabor

Need comments about optional features of https://github.com/littlevgl/lv_utils#run-the-font-converter-offline.

Are those really needed?

kisvegabor commented 5 years ago

I answered some questions (and added new ones) in #2

As usually, should allow read options from file.

Does it mean a config file?

Need comments about optional features ...

puzrin commented 5 years ago

monospace: very important but easy to implement, just add the specified number to lv_font_t

I don't like like doing things without understanding reasons and goals. Especially when i suspect that things may be wrong :).

What is this option for?

Could you explain details?

scale, baseline_shift: they also might be required in some cases.

E.g. I changed the baseline of a * character because it looked ugly in a password filed. The same way the user can zoom on a character.

Something is wrong with it. Change of separate character is very bad idea. Fonts are designed to be consistent. I suspect one of this:

The only valid case i know - merge 2 fonts with different baselines, sometime may need to adjust baseline of second font. But not sure this should be done manually

I don't decline to port "some legacy options for frictionless migration". But we should sort out such things to deprecate in future (if we can't drop those immediately).

In other words... may be user can just take modern font and never use discussed options :) ?

built_in: useful to have all bpps and select later with a define

I still don't understand. Why 4 runs of convertor from makefile not enough? You can include 4 resulting files and guard those via #ifdef-s as you wish

symbol: still required to generate a LittelvGL compatible symbol font. They have some special attributes.

I did not found such option at all. Could you give a link what are you talking about?

kisvegabor commented 5 years ago

I don't like like doing things without understanding reasons and goals. Especially when i suspect that things may be wrong :).

Very good! :)

What is this option for?

In monospace fonts i and W still can have different "useful" width and they should be horizontally aligned. To do this a fix width needs to be known.

Something is wrong with it. Change of separate character is very bad idea. Fonts are designed to be consistent.

In my case, the * was fine when used as * in a normal text but looked ugly when used in password input. There was too much bottom padding. Probably the * was not designed to be used that way.

I still don't understand. Why 4 runs of convertor from makefile not enough? You can include 4 resulting files and guard those via #ifdef-s as you wish

It's also possible. But then we need an option to set the filename and the font name separately. E.g. the font is called my_font_20 but the files are:

In addition, having one file is more convenient. It can be even very long because ideally you will never open it.

I did not found such option at all. Could you give a link what are you talking about?

It seems I've forgotten to update the README with that. Sorry for the confusion, I called it built-in-sym in the PHP script

puzrin commented 5 years ago

In monospace fonts i and W still can have different "useful" width and they should be horizontally aligned. To do this a fix width needs to be known.

Probably i could not explain my question good enougth. I mean, if we need monospase result, we can take monospace TTF font and extract all required metrics as we do for other fonts. Technically, there are no difference between monospace and proportional vector fonts (ttf/woff/...)

Is something wrong with that?

In my case, the * was fine when used as * in a normal text but looked ugly when used in password input. There was too much bottom padding. Probably the * was not designed to be used that way.

Well, if that's the only known case, there should be tons of bullet chars in unicode. I suggest try to find something better-looking there or in FontAwesome. I hope it will be easy with new convertor.

If you have more cases - let me know.

It's also possible. But then we need an option to set the filename and the font name separately. E.g. the font is called my_font_20 but the files are:

  • my_font_20_1bpp
  • my_font_20_2bpp
  • etc

In addition, having one file is more convenient. It can be even very long because ideally you will never open it.

We need to discuss this. I'd like to keep convertor's interface simple. IMHO joining multiple fonts into "catalog" is not convertor's task - too specific thing.

I don't see advantage of keep all in single file, but anyone can join multiple files into one with cp command. Just one additional line in makefile.

It seems I've forgotten to update the README with that. Sorry for the confusion, I called it built-in-sym in the PHP script

Seems this option appeared because old convertor had no multiple sources support. Also, there are lot of popular fonts in npm registry - no problems with install. I think it's safe to drop that option.

kisvegabor commented 5 years ago

Probably i could not explain my question good enougth. I mean, if we need monospase result, we can take monospace TTF font and extract all required metrics as we do for other fonts. Technically, there are no difference between monospace and proportional vector fonts (ttf/woff/...)

It's better if you can extract the intended width of a monospace font from the font file itself.

Well, if that's the only known case, there should be tons of bullet chars in unicode. I suggest try to find something better-looking there or in FontAwesome. I hope it will be easy with new convertor.

You are right. I've chosen the fastest solution.

If you have more cases - let me know.

These options were also used to fix some issues of the PHP ttf renderer functions manually if required. I hope the Node.js libraries will work better :) I so we can drop the base_shift and scale options.

We need to discuss this. I'd like to keep convertor's interface simple. IMHO joining multiple fonts into "catalog" is not convertor's task - too specific thing.

I still see the all-in-one approach reasonable. If I want a font with one bpp (e.g. only 4 bpp) just convert it and copy into my project. I don't need to deal with "enabling it with a define". It just works. It's quite convenient and I'd like to keep this ease of integration. I need a define only if I create an all-in-one font.

If the converter doesn't know about the other bpp-s it needs to add a define guard to every file and the fonts need to enabled by the user manually in every cases.

Seems this option appeared because old convertor had no multiple sources support.

No, it's because the built-in symbols need to be composed in a special way: https://github.com/littlevgl/lv_utils/blob/master/font_conv_core.php#L220-L252

puzrin commented 5 years ago

I still see the all-in-one approach reasonable. If I want a font with one bpp (e.g. only 4 bpp) just convert it and copy into my project. I don't need to deal with "enabling it with a define". It just works. It's quite convenient and I'd like to keep this ease of integration. I need a define only if I create an all-in-one font.

If the converter doesn't know about the other bpp-s it needs to add a define guard to every file and the fonts need to enabled by the user manually in every cases.

Probably i don't know something about lvgl internals, still can't understand advantages of generating one file directly from convertor. As far as i understand, we only have to make different struct names to allow join multiple files into one without conflicts.

I don't object against "having all in single file". I just wish to keep this out of convertor.

No, it's because the built-in symbols need to be composed in a special way: https://github.com/littlevgl/lv_utils/blob/master/font_conv_core.php#L220-L252

I could not understand what is special there. Could you explain? New convertor has options to pick glyphs from additional font and change codes anyhow. Is it not enougth?

If you mean difference between sparse/continuous store formats - IMHO this should be autodetected without any options.

kisvegabor commented 5 years ago

Files can be "joined" via Makefile.

Yes, it is possible but we can not expect the user to use makefiles for an everyday task. A lot of people (including me) uses an IDE and just hits the Build button.

You can have C "wrapper", including all separate files, with additional guards to manage fonts as you need.

If I understood well, it would require to include c files into an other c file. If so it should be avoided because including C files is very hacky.

I can imagine a generic and flexible solution: add a CLI option like --guard which will add
#if FONT_NAME == <font_bpp> to the file.

I could not understand what is special there. Could you explain?

Mainly two things:

I thought these can be done with your suggested "remap" feature too but we need to use thse LV_SYMBOL_GLYPH_FIRST/LAST defines because their value depends on LV_USE_UTF8 settings. (without UTF8 supprot the symbols are mapped from 0xC0)

puzrin commented 5 years ago

Yes, it is possible but we can not expect the user to use makefiles for an everyday task. A lot of people (including me) uses an IDE and just hits the Build button.

Makefile is only an example. There are lot of alternatives - scons, cmake, bat-files and so on. There are should be something configurable for build.

How do you plan to run font convertor with arguments? You still have to store "config" (list of options) somewhere, and allow it to be editable. The same for lv_i18n. How do you plan to share those recipes when multiple users works on the same project, and prefer different ide-s?

PS. FYI, linux CLI should be available in latest Win with ease. There should be no problem to run Makefile there.

If I understood well, it would require to include c files into an other c file. If so it should be avoided because including C files is very hacky.

I don't know details. My position is - use existing standard instruments (just for example - makefiles or C macro processor), instead of writing custom code (which makes font convertor architecture overcomplicated)

I can imagine a generic and flexible solution: add a CLI option like --guard which will add

if FONT_NAME == to the file.

This can be generalized to --template option

BTW, if you expect user to have node.js installed, you should have available all packages via npx. For example, you can run lodash template.

Also, you can have separate package to join multiple fonts into one (technically - with built-in template).

Mainly two things:

  • unicode_fist/last = LV_SYMBOL_GLYPH_FIRST/LAST
  • although the symbols are spare in FontAwesome LittlevGL remaps them into a continuous set. So lv_font_get_..._continuous callback should be assigned.

I thought these can be done with your suggested "remap" feature too but we need to use thse LV_SYMBOL_GLYPH_FIRST/LAST defines because their value depends on LV_USE_UTF8 settings. (without UTF8 supprot the symbols are mapped from 0xC0)

May be we could unify this somehow? Anyway, 6.0 will have a lot of breaking changes. Probably, separate issue required to discuss.

I still don't know all details, but my position is:

So lv_font_get_..._continuous callback should be assigned.

Don't know what is this about. Can this be a part of template as in lv_i18n()? It generates data + small functions. Do we need something special for fonts?

kisvegabor commented 5 years ago

Makefile is only an example. There are lot of alternatives - scons, cmake, bat-files and so on. There are should be something configurable for build.

Sure, there are "tools" to do it. But such a common thing should work out of the box without requiring the use of Makefiles or others.

How do you plan to run font convertor with arguments? You still have to store "config" (list of options) somewhere, and allow it to be editable. The same for lv_i18n. How do you plan to share those recipes when multiple users works on the same project, and prefer different ide-s?

If you have more complex project (more people, more build options etc) you should create the environment for it. We should make possible to use build tools with the font converter but the simple "copy-paste to the project" should work too.

Guard

This can be generalized to --template option

I think templates are overkill in this case. An option -g --guard can do what we need. Easy to use and easy to understand.

Symbols I got an idea about a very different approach. There will be an option to run the converter using a config file, right? Then we can provide some default config files for:

It would have some advantages:

I'm not sure it is planned or nor not but the CLI flags should overwrite the config file options to make possible to set your own font for example.

Don't know what is this about. Can this be a part of template as in lv_i18n()? It generates data + small functions. Do we need something special for fonts?

It's much simpler in this case. In i18n every language has very specific functions to get plurals. These functions could be a myth for the user and we can expect to understand it and deal with them. In case of font there are 2 option:

  1. If the font is continuous use the ..._continuous_... functions
  2. If you provided a list of character use the ..._sparse_... functions

So here is not required to store these functions in every font file.

puzrin commented 5 years ago

If you have more complex project (more people, more build options etc) you should create the environment for it. We should make possible to use build tools with the font converter but the simple "copy-paste to the project" should work too.

I can repeat the same question. Why user can copy-pase single shell line but can not copy-paste 2-3 shell lines :) ?

I think templates are overkill in this case. An option -g --guard can do what we need. Easy to use and easy to understand.

As far as i understand, generated C file should include things like #if USE_LV_FONT_DEJAVU_10 == 1 and there is no need to make those optional. Useful data to pass is "font name" (DEJAVU_10), to be used in names.

I got an idea about a very different approach. There will be an option to run the converter using a config file, right? Then we can provide some default config files for:

  • different scripts/languages/ranges
  • symbol fonts
  • and for what comes later

...

If project uses argparse, you can run <script_name> @file_with_args. Then arguments will be loaded from file the same way as if those were typed in CLI string. You can try with lv_i18n.

Of cause, it's obvious to provide files with presets in lvgl docs.

I'm not sure it is planned or nor not but the CLI flags should overwrite the config file options to make possible to set your own font for example.

I'm not sure it is planned or nor not but the CLI flags should overwrite the config file options to make possible to set your own font for example.

I'd recommend to keep things simple and modify default config templates for project needs, instead of doing nested configs. But user can do things like this:

lv_font_conv @default_config @additional_config <and_more_cli_options>
  1. If the font is continuous use the ...continuous... functions
  2. If you provided a list of character use the ...sparse... functions

Not a problem.

But why not just boolean flag (continuous/sparse) in font header (struct)? That's just an internal format details, like compression algorythm, bpp and so on.

embeddedt commented 5 years ago

@puzrin

(puzrin) Files can be "joined" via Makefile. (kisvegabor) If you have more complex project (more people, more build options etc) you should create the environment for it. We should make possible to use build tools with the font converter but the simple "copy-paste to the project" should work too. (puzrin) Why user can copy-pase single shell line but can not copy-paste 2-3 shell lines :) ?

Note: I'm a bit confused on exactly what the topic is here, so please correct me if I'm not making any sense. :)

I think the point @kisvegabor is making is that the existing approach with keeping all of the various BPP formats inside a single C file works better for many users. We shouldn't require them to run a script in order to join the C files when they could just be joined to begin with.

Having a C wrapper works, but why bother with splitting them and then making a wrapper when we could just avoid splitting them in the first place?

Why user can copy-pase single shell line but can not copy-paste 2-3 shell lines :) ?

They aren't copying and pasting a shell line; they are copying and pasting one C file instead of 4 or 5, unless I'm mistaken.

puzrin commented 5 years ago

Note: I'm a bit confused on exactly what the topic is here, so please correct me if I'm not making any sense. :)

The root of problem is, existing font structure use some "old-fashioned" approaches, and i don't like to propagate those from old script to new one. So we try to find compromise - how to get desired result and keep new convertor internals nice.

I think the point @kisvegabor is making is that the existing approach with keeping all of the various BPP formats inside a single C file works better for many users. We shouldn't require them to run a script in order to join the C files when they could just be joined to begin with.

My personal opinion is, making modern projects without build system is like making projects without git. In theory, someone can, but that looks horrible even for "helloworld" :).

But i respect everyone's right to do any things, if i'm not touched :). So we discuss possible solution instead of saying "i will never do that".

Having a C wrapper works, but why bother with splitting them and then making a wrapper when we could just avoid splitting them in the first place?

I said different thing. I said we should try to avoid add "font catalog manager" into font convertor core. Because that's a different abstraction layer. And tried to suggest alternate solutions (may be, not very good).

embeddedt commented 5 years ago

My personal opinion is, making modern projects without build system is like making projects without git. In theory, someone can, but that looks horrible even for "helloworld" :)

I don't agree completely. I think it depends on the level of complexity that the project has, and how portable it needs to be.

For example, LittlevGL consists purely of a collection of source and header files. There is no dependence on any existing build system, and it can easily be integrated into anyone's project. This is one of the strengths about it that I really like. There are other projects that require you to make use of their specific build system, which makes them harder to integrate into an existing product.

I understand that using scripts can make things easier, more flexible, and cleaner, but I think portability and ease of import is an important issue to consider.

puzrin commented 5 years ago

I suggested earlier one of possible solution - create additional node.js wrapper like lv_font_conv_all_bpp. It's as portable as lv_font_conf

My point is:

I hope, everyone agree that i don't wish to make things bad, and i don't propose to throw problems from font convertor to LVGL sources.

kisvegabor commented 5 years ago

I hope, everyone agree that i don't wish to make things bad, and i don't propose to throw problems from font convertor to LVGL sources.

Sure, we see that. :+1:

As @embeddedt has pointed out one of the best things in LittelvGL is the ease of integration because you can integrate it into any build system. So the result fonts should work out of the box, similarly to the other parts of the library.

Let's discuss the "all bpp" question first and open a new issue for "built-in symbols".

A lot of things were being said, so just to see things clear: what's the problem with an --bpp 1/2/4/8/all option which behaves similarly to the current version?

puzrin commented 5 years ago

so just to see things clear: what's the problem with an --bpp 1/2/4/8/all option which behaves similarly to the current version?

Common intro :)

When we write software, a key point is to make a good data flow. This means:

If we have good data flow => code will be fine with high probability.

Why i don't like --bpp all

It changes output data to multi-dimentional "without need". That means more complicated testing and "not symmetric" writers for different output formats.

What is "ease of use/integration" for LVGL?

We should declare very pricise, what this mean.

IMHO, if we can achieve the same effect with "multiple files" - goal will be reached.

embeddedt commented 5 years ago

It changes output data to multi-dimentional "without need". Looks more like "ability to switch bpp with single directive". If advanced user wish to make custom font - he knows exactly what is needed, and don't need switch bpp on the fly via #define at all.

They could still want to do that.

In this case, the bpp option isn't just for display compatiblity. It affects the visual quality of the font. For example, a 1 bpp font is not antialiased at all, while 8 bpp has a full alpha channel and looks the best. The user could potentially wish to change the visual quality on the fly.

Using a method where they can just change the #define allows them to test the different options faster than reconverting the font.

puzrin commented 5 years ago

They could still want to do that.

Using a method where they can just change the #define allows them to test the different options faster than reconverting the font.

IMHO this request is very artificial, BUT... user can copy the same "C wrapper" from LVGL, create 4 variants and switch those via #define, as LVGL built-ins.

Situation is a bit different. In LVGL we MUST provide variants with easy access, because we don't know user's needs. When user does customs font - he already knows his hardware and requirement for font smoothness.

embeddedt commented 5 years ago

I like a universal solution. In my opinion the way built-in fonts are handled and the way custom fonts are handled should be similar. It makes things simpler.

puzrin commented 5 years ago

I like universal solutions too (when possible), but don't like to pay with bad data for it.

We have use case, when blind apply of "universal principle" may be evil (like premature optimizations). In such case, correct approach is to split for sub-cases and analyse each independently, then join results back.

In https://github.com/littlevgl/lv_font_conv/issues/1#issuecomment-476680525 i tied to do such split/join, to explain how to do things more simple without loosing useful features.

embeddedt commented 5 years ago

@puzrin

We have use case, when blind apply of "universal principle" may be evil (like premature optimizations). In such case, correct approach is to split for sub-cases and analyse each independently, then join results back.

I understand.

Right now there's a lot of information and discussion spread throughout all the comments. For clarity, could you outline, from beginning to end, the exact process a user would follow to convert some_font.ttf to LittlevGL format and utilize it in their project? I just want to get an idea of exactly how much work would be involved for them.

puzrin commented 5 years ago

I did not dived deep into LVGL API, and can not provide code samples. Existing lv_font_conv skeleton is in dev branch, and you can play with it's --help.

Also, feel free to ask me anything about options.

Case 1. User wish custom fonts

I expect, user knows exactly bpp, size, codepoints and so on. Because it does that for known project/hardware.

Then:

Case 2. LVGL developpers wish to provide built-in fonts

Goal - provide something "ready-to-use" for people who don't wish to dive into details.

Result: a bit more font files (but nobody cares, because autogenerated), but still the same control via #define, as in lvgl 5.x.

embeddedt commented 5 years ago

i'd suggest to create make/sh/bat file to store all configs and update all at once

The only major potential issue I see is that we need a script setup that is going to seamlessly work across all platforms without requiring users to be installing lots of tools.

User write code to attach fonts to LVGL runtime.

That currently consists of three steps:

  1. Declare the font in a header or source file: LV_FONT_DECLARE(my_font_struct);
  2. Attach it to a style: style.text.font = &my_font_struct;
  3. Set the style on an object.

Would the workflow be similar?

puzrin commented 5 years ago

The only major potential issue I see is that we need a script setup that is going to seamlessly work across all platforms without requiring users to be installing lots of tools.

I think not. Let's split it into sub-cases:

  1. User wish to use LVGL out-of-box. Then he does not need any scipting, because we already provide working solution. From user's point of view, all interface/features will the exactly the same. Internals will be different, but user don't cares about internals.
  2. User wish to make custom font. Then node.js should be installed. It will fork the same way on all platforms. Fonts initicalisaton will be custom anyway, but this C code is also the same for all platforms.

That currently consists of three steps:

  • Declare the font in a header or source file: LV_FONT_DECLARE(my_font_struct);
  • Attach it to a style: style.text.font = &my_font_struct;
  • Set the style on an object.

Would the workflow be similar?

I think yes. The only difference - by default (for custom font) he will not be able to "magically" switch bpp. But if he add all bpp variants + copy wrapper from LVGL, then it will be possible (personally, i think this is abnormal, but doable).

embeddedt commented 5 years ago

When converting fonts, you said that:

  • User runs lvgl_font_conf for each required combination and store results in separate dir
    • i'd suggest to create make/sh/bat file to store all configs and update all at once

That script wouldn't be written in JavaScript, would it?

puzrin commented 5 years ago

That script wouldn't be written in JavaScript, would it?

This is ONLY for case when user wish to switch custom font bpp "on the fly". Since such request is very specific, we say "do it as you wish, according with your preferences to your built system".

In real world, i think, user will never do this, and just run lv_font_conv for known [font, size, bpp].

Note, if we consider Material Design, user will have to create fonts for multiple sizes. It will have to automate this somehow anyway. This probability is much higher than multi-bpp case.

puzrin commented 5 years ago

In other words, "bpp selection on the fly via #define" is attribute of default LVGL built in fonts. It's required, no doubts. But we should not propagate it to custom fonts.

So, instead of "how to switch bpp on the fly" we can solve "how to switch LVGL built-in fonts bpp on the fly". That's completely different thing.

kisvegabor commented 5 years ago

So in summary:

  1. Generate fonts with a specific bpp without define guards.
  2. In case of built-in fonts have all 4 bpp variants as 4 files and add the guard manually. (As they are changing very rarely we could even add the defines manually. 10 minutes of work.)

If so, then it's ok for me. As @puzrin pointed out it's not a real-life scenario to continuously switch between bpp-s. It should be only an experimental thing to see which one looks well on a given display. If the user is curious about the bpp variants of a font, he can just replace the file to see the difference.

Have I understood well? :)

puzrin commented 5 years ago

Have I understood well? :)

Yes, just a small note about:

Generate fonts with a specific bpp without define guards.

Probably we could add some guard to simplify making wrapper for 4 font variants and keep existing notation. That will not affect stand-alone usage anyhow. Details can be postponed to final moment, when we start integration with LVGL sources.

kisvegabor commented 5 years ago

Probably we could add some guard to simplify making wrapper for 4 font variants and keep existing notation. That will not affect stand-alone usage anyhow. Details can be postponed to final moment, when we start integration with LVGL sources.

I've mentioned a --guard flag earlier. Do you mean something like this?

puzrin commented 5 years ago

I've mentioned a --guard flag earlier. Do you mean something like this?

You mentioned this feature as option, but i mean is as alwais existing.

Probably that's not needed. If we argree multi-bpp feature is needed for LVGL core only, then we can do wrap in Makefile (it's used only by developpers, not by users).

puzrin commented 5 years ago

Just for info. If node.js is acceptable, and Makefile NOT welcome, there is nice imitation to describe make-like targets in JS + popular unix shell commands :)

https://github.com/shelljs/shelljs/wiki/The-make-utility

I would not promote this as ideal solution, but it's nice.

puzrin commented 5 years ago

Dumper sandbox ready. Could you check glyph images quality? Is it appropriate or something need fixes?

Example without install:

npx littlevgl/lv_font_conv#dev --font <ttf/woff_path> -r 20-0x10ffff -s 16 -b4 -o <output_folder>

Example with install:

npm i littlevgl/lv_font_conv#dev -g
lv_font_conv --font <ttf/woff_path> -r 20-0x10ffff -s 16 -b4 -o <output_folder>
kisvegabor commented 5 years ago

Probably that's not needed. If we argree multi-bpp feature is needed for LVGL core only, then we can do wrap in Makefile (it's used only by developpers, not by users).

I suggest dropping it for now. In a previous version (a year ago) I made it mandatory but very inconvenient to always enable it if new font comes. If the user doesn't want a font he can delete it or exclude from the build.

Just for info. If node.js is acceptable, and Makefile NOT welcome, there is nice imitation to describe make-like targets in JS + popular unix shell commands :)

Node.js is fine because finally, it will run in an online converter too. For example, Python also could work as a CLI tool but I wouldn't promote it because we can't turn it to an online converter.

Dumper sandbox ready. Could you check glyph images quality? Is it appropriate or something need fixes?

I've tested it and worked well. The quality look good as well.

puzrin commented 5 years ago

Close - brainstorming complete. Continue in font features/spec issues.