Add unicode font support

sago35 commented 4 years ago

Unicode font support
- Tested with jisx0208 (Japanese) font only
TinyGo font files can be created from BDF fonts (tinyfontgen)

sago35 commented 4 years ago

TODO

[x] Create a tinyfontgen command to create a tinyfont from a BDF font
[x] Convert to new format font (freemono / freesans / freserif / gophers / proggy)
[x] Update examples
[x] ~~Update comments in font files / Approx. XXXX bytes~~
[x] Determine which of the following options to use (DrawChar(), WriteLine(), ...)
1. ~~Use [] byte as argument~~
  - ~~I think this option is second best because of compatibility.~~
2. ~~Use [] rune as an argument~~
  - ~~I think strings are better.~~
3. Use string as argument
  - Compatibility is broken, but I think the best. I think string is better than []rune.
[x] Update README
- warn users about the incompatibilities
[x] Fix YAdvance is wrong when input is emoji (tinyfontgen)
[x] Add the function to output all characters included in bdf (tinyfontgen)
[x] Draw tofu (or nothing) when there is no font corresponding to the code point (tinyfont)
[x] Deprecate Rune2index to include rune in Glyph to reduce memory size.
[x] Reduce the number of calls to GetGlyph() to improve processing efficiency
[x] Update comments
[x] gofmt
[x] Fix magic number

Not for this PR??

[ ] ~~Publish a program that converts the original tinyfont.Font to the new tinyfont.Font~~
[ ] Set the function's argument to the Fonter interface instead of the Font struct
[ ] ~~Merge master branch to rune branch.~~
[ ] Add jisx0208 & notoemoji as fonts, and add example #10

sago35 commented 4 years ago

Changed Font structure. I don't know if this is appropriate. However, a structure like map[rune]Glyph is needed for Unicode support.

conejoninja commented 4 years ago

Thanks a lot!. This is a step in the right direction to support a wide range of languages. A new format is fine as long as there's a way to generate/convert fonts to it The only thing I don't like is that the interface will change, and make existing code incompatible ([]byte -> []rune), but I think it's for the better.

sago35 commented 4 years ago

the interface will change, and make existing code incompatible

I think it is bad to break compatibility. Is there a better way?

sago35 commented 4 years ago

Added tinyfontgen CLI. You can freely create tinyfont font files from bdf. For now, bdf font CHARSET_REGISTRY only supports ISO08859 and JISX0208 (japanese).

Usage
https://github.com/sago35/tinyfont/blob/rune/cmd/tinyfontgen/README.md

sago35 commented 4 years ago

Current status : tinygo + tinyfont + notoemoji https://gist.github.com/sago35/e833c76d9e5ccd71e5cc8d8c2229b9f8

sago35 commented 4 years ago

Now I updated all fonts included in tinyfont. Of course, fonts created with tinyfontgen can also be used at the same time.

sago35 commented 4 years ago

@conejoninja

The only thing I don't like is that the interface will change, and make existing code incompatible ([]byte -> []rune), but I think it's for the better.

[] byte was changed to string. The reason is to make it a more natural function prototype. Compatibility is broken, but I think string is the most natural.

conejoninja commented 4 years ago

I pushed and change the base branch to dev, so we can test it once merged, do a proper release and warn users about the incompatibilities. I took a look and it's a great job, a few details to polish, but really great :+1:

deadprogram commented 4 years ago

This is very exciting work, thank for contributing @sago35 looking forward to seeing it land.

sago35 commented 4 years ago

Update comments in font files / Approx. XXXX bytes

How did you calculate Approx. XXXX byte? Do you need comments?

conejoninja commented 4 years ago

I don't really know how to calculate it, they were left there from the original GLXFont files. I personally would remove them.

sago35 commented 4 years ago

The size of the new and old fonts was compared. Since the key of map has become rune, the amount has increased.

The conversion and size calculations were performed using the following Go CLI. However, this CLI has not been generalized enough to be included in tinyfont. https://github.com/sago35/tinyfont/commit/57f7dd5822f3750adf2991507f89b9bf5d0433ef

fontname	org	new	diff	%
freemono.Regular12pt7b	2128 byte	2506 byte	+378 byte	117.76 %
freemono.Regular18pt7b	3757 byte	4135 byte	+378 byte	110.06 %
freemono.Regular24pt7b	6326 byte	6704 byte	+378 byte	105.98 %
freemono.Regular9pt7b	1512 byte	1890 byte	+378 byte	125.00 %
freemono.Bold12pt7b	2398 byte	2776 byte	+378 byte	115.76 %
freemono.Bold18pt7b	4481 byte	4859 byte	+378 byte	108.44 %
freemono.Bold24pt7b	7465 byte	7843 byte	+378 byte	105.06 %
freemono.Bold9pt7b	1668 byte	2046 byte	+378 byte	122.66 %
freemono.BoldOblique12pt7b	2634 byte	3012 byte	+378 byte	114.35 %
freemono.BoldOblique18pt7b	4924 byte	5302 byte	+378 byte	107.68 %
freemono.BoldOblique24pt7b	8303 byte	8681 byte	+378 byte	104.55 %
freemono.BoldOblique9pt7b	1835 byte	2213 byte	+378 byte	120.60 %
freemono.Oblique12pt7b	2375 byte	2753 byte	+378 byte	115.92 %
freemono.Oblique18pt7b	4182 byte	4560 byte	+378 byte	109.04 %
freemono.Oblique24pt7b	7120 byte	7498 byte	+378 byte	105.31 %
freemono.Oblique9pt7b	1650 byte	2028 byte	+378 byte	122.91 %
freesans.Regular12pt7b	2637 byte	3015 byte	+378 byte	114.33 %
freesans.Regular18pt7b	4827 byte	5205 byte	+378 byte	107.83 %
freesans.Regular24pt7b	8132 byte	8510 byte	+378 byte	104.65 %
freesans.Regular9pt7b	1818 byte	2196 byte	+378 byte	120.79 %
freesans.Bold12pt7b	2854 byte	3232 byte	+378 byte	113.24 %
freesans.Bold18pt7b	5171 byte	5549 byte	+378 byte	107.31 %
freesans.Bold24pt7b	8811 byte	9189 byte	+378 byte	104.29 %
freesans.Bold9pt7b	1898 byte	2276 byte	+378 byte	119.92 %
freesans.BoldOblique12pt7b	3203 byte	3581 byte	+378 byte	111.80 %
freesans.BoldOblique18pt7b	5939 byte	6317 byte	+378 byte	106.36 %
freesans.BoldOblique24pt7b	10115 byte	10493 byte	+378 byte	103.74 %
freesans.BoldOblique9pt7b	2132 byte	2510 byte	+378 byte	117.73 %
freesans.Oblique12pt7b	3030 byte	3408 byte	+378 byte	112.48 %
freesans.Oblique18pt7b	5619 byte	5997 byte	+378 byte	106.73 %
freesans.Oblique24pt7b	9479 byte	9857 byte	+378 byte	103.99 %
freesans.Oblique9pt7b	2037 byte	2415 byte	+378 byte	118.56 %
freeserif.Regular12pt7b	2507 byte	2885 byte	+378 byte	115.08 %
freeserif.Regular18pt7b	4554 byte	4932 byte	+378 byte	108.30 %
freeserif.Regular24pt7b	7678 byte	8056 byte	+378 byte	104.92 %
freeserif.Regular9pt7b	1748 byte	2126 byte	+378 byte	121.62 %
freeserif.Bold12pt7b	2659 byte	3037 byte	+378 byte	114.22 %
freeserif.Bold18pt7b	4941 byte	5319 byte	+378 byte	107.65 %
freeserif.Bold24pt7b	8515 byte	8893 byte	+378 byte	104.44 %
freeserif.Bold9pt7b	1830 byte	2208 byte	+378 byte	120.66 %
freeserif.BoldItalic12pt7b	2906 byte	3284 byte	+378 byte	113.01 %
freeserif.BoldItalic18pt7b	5406 byte	5784 byte	+378 byte	106.99 %
freeserif.BoldItalic24pt7b	8913 byte	9291 byte	+378 byte	104.24 %
freeserif.BoldItalic9pt7b	1978 byte	2356 byte	+378 byte	119.11 %
freeserif.Italic12pt7b	2652 byte	3030 byte	+378 byte	114.25 %
freeserif.Italic18pt7b	4801 byte	5179 byte	+378 byte	107.87 %
freeserif.Italic24pt7b	8247 byte	8625 byte	+378 byte	104.58 %
freeserif.Italic9pt7b	1831 byte	2209 byte	+378 byte	120.64 %
gophers.Regular121pt	40665 byte	41039 byte	+374 byte	100.92 %
gophers.Regular14pt	1257 byte	1631 byte	+374 byte	129.75 %
gophers.Regular18pt	1623 byte	1997 byte	+374 byte	123.04 %
gophers.Regular22pt	2091 byte	2465 byte	+374 byte	117.89 %
gophers.Regular32pt	3371 byte	3745 byte	+374 byte	111.09 %
gophers.Regular58pt	9859 byte	10233 byte	+374 byte	103.79 %
proggy.TinySZ8pt7b	1057 byte	1435 byte	+378 byte	135.76 %

sago35 commented 4 years ago

@conejoninja

I have a question.

I treated the CRLF (0x0A or 0x0D) as a line break, is that good? It might be better to implement the handling of line breaks in a library that uses TinyFont.

https://github.com/sago35/tinyfont/blob/rune/tinyfont.go#L163

sago35 commented 4 years ago

@conejoninja @deadprogram It's now ready for everyone to try. It would be nice if you could try tinyfontgen while trying the example source.

I would like you to correct the English in README.md.

tinygo build commands

tinygo build -o app.uf2 -size short -target microbit ./examples/epd
tinygo build -o app.uf2 -size short -target pybadge ./examples/hub75
tinygo build -o app.uf2 -size short -target pybadge ./examples/pybadge
tinygo build -o app.uf2 -size short -target pyportal ./examples/pyportal

tinyfontgen example https://github.com/sago35/tinyfont/tree/rune/cmd/tinyfontgen

tinyfontgen --font _font\mplus_bitmap_fonts-2.2.4\fonts_e\mplus_f12r.bdf --output .\examples\pyportal\mplus_f12r.go --fontname Mplusf12r --verbose
tinyfontgen --font _font\mplus_bitmap_fonts-2.2.4\fonts_e\mplus_f12r.bdf --output .\examples\pyportal\mplus_f12r.go --fontname Mplusf12r --verbose --yadvance 32

# Merging two fonts
tinyfontgen --font _font\ayu20gothic\10x20grkm.bdf --font _font\ayu20gothic\k20gm.bdf 世界 --output .\examples\pyportal\ayu20gothic2.go --fontname Ayu20gothic --verbose

I tested it with the following fonts. Some fonts fail to judge YAdvance, so you need to fix it with --yadvance option.

M+ BITMAP FONTS 2.2.4
- https://mplus-fonts.osdn.jp/mplus-bitmap-fonts/download/index.html
- https://osdn.net/projects/mplus-fonts/downloads/5030/mplus_bitmap_fonts-2.2.4.tar.gz/
- mplus_f12r -mplus-fxd-medium-r-semicondensed--12-120-75-75-c-60-iso8859-1
- mplus_h12r -mplus-hlv-medium-r-normal--12-120-75-75-p-60-iso8859-1
eosrei/twemoji-color-font
- https://github.com/eosrei/twemoji-color-font
- Use otf2bdf for conversion
  - ./otf2bdf.exe -o TwitterColorEmoji-SVGinOT.bdf TwitterColorEmoji-SVGinOT.ttf
Noto Emoji
- https://www.google.com/get/noto/#emoji-zsye
- Use otf2bdf for conversion
- Note: Noto Color Emoji does not work

sago35 commented 4 years ago

There is a little todo in the source. I will respond in the future.

sago35 commented 4 years ago

@conejoninja The PR is now complete. Please review.

https://github.com/tinygo-org/tinyfont/pull/2#issuecomment-606085233

sago35 commented 4 years ago

Removing RuneToIndex has made the font size a little smaller.

fontname	org	new	diff	%
tinyfont.Org01	939 byte	1127 byte	+188 byte	120.02 %
tinyfont.Picopixel	848 byte	1036 byte	+188 byte	122.17 %
tinyfont.Tiny3x3a2pt7b	810 byte	998 byte	+188 byte	123.21 %
tinyfont.TomThumb	2335 byte	2741 byte	+406 byte	117.39 %
freemono.Regular12pt7b	2128 byte	2316 byte	+188 byte	108.83 %
freemono.Regular18pt7b	3757 byte	3945 byte	+188 byte	105.00 %
freemono.Regular24pt7b	6326 byte	6514 byte	+188 byte	102.97 %
freemono.Regular9pt7b	1512 byte	1700 byte	+188 byte	112.43 %
freemono.Bold12pt7b	2398 byte	2586 byte	+188 byte	107.84 %
freemono.Bold18pt7b	4481 byte	4669 byte	+188 byte	104.20 %
freemono.Bold24pt7b	7465 byte	7653 byte	+188 byte	102.52 %
freemono.Bold9pt7b	1668 byte	1856 byte	+188 byte	111.27 %
freemono.BoldOblique12pt7b	2634 byte	2822 byte	+188 byte	107.14 %
freemono.BoldOblique18pt7b	4924 byte	5112 byte	+188 byte	103.82 %
freemono.BoldOblique24pt7b	8303 byte	8491 byte	+188 byte	102.26 %
freemono.BoldOblique9pt7b	1835 byte	2023 byte	+188 byte	110.25 %
freemono.Oblique12pt7b	2375 byte	2563 byte	+188 byte	107.92 %
freemono.Oblique18pt7b	4182 byte	4370 byte	+188 byte	104.50 %
freemono.Oblique24pt7b	7120 byte	7308 byte	+188 byte	102.64 %
freemono.Oblique9pt7b	1650 byte	1838 byte	+188 byte	111.39 %
freesans.Regular12pt7b	2637 byte	2825 byte	+188 byte	107.13 %
freesans.Regular18pt7b	4827 byte	5015 byte	+188 byte	103.89 %
freesans.Regular24pt7b	8132 byte	8320 byte	+188 byte	102.31 %
freesans.Regular9pt7b	1818 byte	2006 byte	+188 byte	110.34 %
freesans.Bold12pt7b	2854 byte	3042 byte	+188 byte	106.59 %
freesans.Bold18pt7b	5171 byte	5359 byte	+188 byte	103.64 %
freesans.Bold24pt7b	8811 byte	8999 byte	+188 byte	102.13 %
freesans.Bold9pt7b	1898 byte	2086 byte	+188 byte	109.91 %
freesans.BoldOblique12pt7b	3203 byte	3391 byte	+188 byte	105.87 %
freesans.BoldOblique18pt7b	5939 byte	6127 byte	+188 byte	103.17 %
freesans.BoldOblique24pt7b	10115 byte	10303 byte	+188 byte	101.86 %
freesans.BoldOblique9pt7b	2132 byte	2320 byte	+188 byte	108.82 %
freesans.Oblique12pt7b	3030 byte	3218 byte	+188 byte	106.20 %
freesans.Oblique18pt7b	5619 byte	5807 byte	+188 byte	103.35 %
freesans.Oblique24pt7b	9479 byte	9667 byte	+188 byte	101.98 %
freesans.Oblique9pt7b	2037 byte	2225 byte	+188 byte	109.23 %
freeserif.Regular12pt7b	2507 byte	2695 byte	+188 byte	107.50 %
freeserif.Regular18pt7b	4554 byte	4742 byte	+188 byte	104.13 %
freeserif.Regular24pt7b	7678 byte	7866 byte	+188 byte	102.45 %
freeserif.Regular9pt7b	1748 byte	1936 byte	+188 byte	110.76 %
freeserif.Bold12pt7b	2659 byte	2847 byte	+188 byte	107.07 %
freeserif.Bold18pt7b	4941 byte	5129 byte	+188 byte	103.80 %
freeserif.Bold24pt7b	8515 byte	8703 byte	+188 byte	102.21 %
freeserif.Bold9pt7b	1830 byte	2018 byte	+188 byte	110.27 %
freeserif.BoldItalic12pt7b	2906 byte	3094 byte	+188 byte	106.47 %
freeserif.BoldItalic18pt7b	5406 byte	5594 byte	+188 byte	103.48 %
freeserif.BoldItalic24pt7b	8913 byte	9101 byte	+188 byte	102.11 %
freeserif.BoldItalic9pt7b	1978 byte	2166 byte	+188 byte	109.50 %
freeserif.Italic12pt7b	2652 byte	2840 byte	+188 byte	107.09 %
freeserif.Italic18pt7b	4801 byte	4989 byte	+188 byte	103.92 %
freeserif.Italic24pt7b	8247 byte	8435 byte	+188 byte	102.28 %
freeserif.Italic9pt7b	1831 byte	2019 byte	+188 byte	110.27 %
gophers.Regular121pt	40665 byte	40851 byte	+186 byte	100.46 %
gophers.Regular14pt	1257 byte	1443 byte	+186 byte	114.80 %
gophers.Regular18pt	1623 byte	1809 byte	+186 byte	111.46 %
gophers.Regular22pt	2091 byte	2277 byte	+186 byte	108.90 %
gophers.Regular32pt	3371 byte	3557 byte	+186 byte	105.52 %
gophers.Regular58pt	9859 byte	10045 byte	+186 byte	101.89 %
proggy.TinySZ8pt7b	1057 byte	1245 byte	+188 byte	117.79 %

sago35 commented 4 years ago

memo:

NotoEmoji

https://www.google.com/get/noto/#emoji-zsye

Noto is a trademark of Google Inc. Noto fonts are open source. All Noto fonts are published under the SIL Open Font License, Version 1.1. Language data and some sample texts are from the Unicode CLDR project.

fontname	size
NotoEmoji-Regular-12pt.go	3140 byte
NotoEmoji-Regular-16pt.go	4560 byte
NotoEmoji-Regular-20pt.go	6462 byte

NotoSansCJKjp-Regular

fontname	size
NotoSansCJKjp-Regular-12pt.go	1030823 byte

note: It's too big if we bundle all the glyphs.

conejoninja commented 4 years ago

Thanks for your work, it's truly amazing. :clap: :clap: :clap:

I'm merging this now, and we could add new fonts in separate PRs.

conejoninja commented 4 years ago

Just one issue I saw now, in line 48 https://github.com/tinygo-org/tinyfont/blob/dev/tinyfont.go#L48

font *Font is not being use, we could remove it, right?

sago35 commented 4 years ago

font *Font is not being use, we could remove it, right?

That's right. I've created a PR (#6) to delete because I don't need it

tinygo-org / tinyfont

Add unicode font support #2