Publishing TTF files for releases

[WhyNotHugo]

Hi! I have a pipeline that builds ttf files two twemoji. It builds Twemoji as a TrueType font. In particular, it is usable on modern Linux (either desktop or mobile) as well as other Unix-like distributions.

Mozilla has twemoji-colr, but that's in a font-format designed by Adobe and Mozilla, which basically only work on Windows, Firefox and some very rare applications. Major toolkits like Qt and GTK do not support that format.

I wrote these about two months ago and the builds are used downstream by Alpine Linux packages. I didn't see much hope in up-streaming these to Twitter's repo since, well Twitter is undergoing changes.

I'm not sure what your plans are for publishing artefacts, but perhaps this can serve as a reference for building truetype files for tags/releases.

The build script is a somehow of a hack, and leverages Noto's build system for its emoji fonts and a lot of its scripts. It's a mix of the Fedora recipe and an old ArchLinux recipe.

image: alpine/edge
packages:
  - make
  - graphicsmagick
  - cairo-dev
  - nototools
  - pngquant
  - zopfli
  - rsync
environment:
  VERSION: 14.0.2
sources:
  - https://src.fedoraproject.org/rpms/twitter-twemoji-fonts.git#78a8615f22996e173b6489b004210f6302cfa456
  - https://github.com/twitter/twemoji.git#v14.0.2
  - https://github.com/googlefonts/noto-emoji.git#ac1703e9d7feebbf5443a986e08332b1e1c5afcf
tasks:
  - patch: |
      cd noto-emoji
      patch -p1 -i ../twitter-twemoji-fonts/noto-emoji-use-system-pngquant.patch
      patch -p1 -i ../twitter-twemoji-fonts/noto-emoji-build-all-flags.patch
      patch -p1 -i ../twitter-twemoji-fonts/noto-emoji-use-gm.patch
  - prepare: |
      cd noto-emoji
      mv NotoColorEmoji.tmpl.ttx.tmpl Twemoji.tmpl.ttx.tmpl
      sed -i 's/Noto Color Emoji/Twemoji/; s/NotoColorEmoji/Twemoji/; s/Copyright .* Google Inc\./Twitter, Inc and other contributors./; s/ Version .*/ '$VERSION'/; s/.*is a trademark.*//; s/Google, Inc\./Twitter, Inc and other contributors/; s,http://www.google.com/get/noto/,https://github.com/twitter/twemoji/,; s/.*is licensed under.*/      Creative Commons Attribution 4.0 International/; s,http://scripts.sil.org/OFL,http://creativecommons.org/licenses/by/4.0/,' Twemoji.tmpl.ttx.tmpl
      cd ../twemoji/assets/72x72/
      for png in *.png; do
          mv $png emoji_u${png//-/_}
      done
  - build: |
      cd noto-emoji
      make EMOJI=Twemoji EMOJI_SRC_DIR=../twemoji/assets/72x72 FLAGS= BODY_DIMENSIONS=76x72
  - setup-ssh: |
      echo "anchor.whynothugo.nl ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE2WacYpYEP5vvevx4NXQn3Ktd53pBkZR/bpPoS3Wkyg" > $HOME/.ssh/known_hosts
  - rsync: |
      cd noto-emoji
      install -Dm 644 Twemoji.ttf artefacts/twemoji.ttf/$(date '+%Y-%m-%d_%H-%M')/Twemoji-$VERSION.ttf
      rsync -r artefacts/ rsync@anchor.whynothugo.nl:./

[12Me21]

I'm pretty sure COLRv0 is the most widely supported color font format, i have tested it myself on linux (i.e. FreeType), windows, ios, in several web browsers, etc. (of course, it's still good to make the font available in other formats, regardless)

[boywithkeyboard]

Hey @WhyNotHugo!

I have write access to the NPM organization and I'd suggest that we should publish this as a separate package - maybe @twemoji/ttf or @twemoji/font?

If possible, I'd also suggest to include .woff2 files.

[WhyNotHugo]

I think @twemoji/ttf is best; @twemoji/font is ambiguous since there's different font formats.

Can you push a raw ttf just by itself to npm (or a tarball with just a ttf)? Doesn't it validate that what you're publishing actually looks like an npm package?

[boywithkeyboard]

A NPM package must only have a package.json (and should have a license and readme).

[boywithkeyboard]

I already messaged @jdecked on both Twitter and Discord to give me maintainer permissions to this repository here to manage the issues/pull requests. Sadly, she's pretty inactive on both.

[Soup-64]

This would be great for the AUR packaging of Twemoji since it currently depends on fedora's packaging of Twemoji to get it in a .ttf format for installation, and that is still using the now dead original repo. This would also allow that package to directly use this repo rather than depending upon another packaging.

[boywithkeyboard]

@Soup-64, as I already mentioned, I don't think we should further make the core NPM package larger, rather create separate packages for various usecases.

There are already @twemoji/api (this repository) and @twemoji/svg, so it'd be imo a good idea to publish it under @twemoji/font and aim to provide it in various popular formats (mainly ttf and woff2).

[boywithkeyboard]

@WhyNotHugo could you please create a demo repository and send me the link?

[12Me21]

There aren't really multiple font formats in use currently, everything is just opentype (.ttf, .otf) with various different tables for storing glyph data. (and .woff/.woff2 which are container formats for storing compressed opentype files) I assume this uses either CBDT or sbix tables to store the glyph images, so it should be named twemoji/cbdt or twemoji/sbix or something like that (to differentiate between other color font tables like COLR and SVG)

[boywithkeyboard]

@12Me21 right. We should name it @twemoji/font as the abbreviations you mentioned most definitely don't mean anything to most people.

So, we just generate a TTF file and then create a WOFF(2) file from it.

There are already some libraries out there for converting these files, so the second step should be quite easy.

[WhyNotHugo]

On Thu, 30 Mar 2023, at 14:59, Samuel Kopp wrote:

@Soup-64 https://github.com/Soup-64 i already mentioned, I don't think we should further make the core NPM package larger, rather create separate packages for various usecases.

I think Soup was suggesting publishing the ttf file directly (eg: attaching it to GitHub releases?) instead of uploading to npm at all. Uploading to npm requires creating the package, packaging it so that it can be downloaded, unpackaged and package.json discarded.

On Thu, 30 Mar 2023, at 15:33, Samuel Kopp wrote:

@WhyNotHugo could you please create a demo repository and send me the link?

Link to the build script's repo is in my original post: https://git.sr.ht/~whynothugo/twemoji.ttf

[boywithkeyboard]

@WhyNotHugo @Soup-64 I just chatted with @jdecked and she said that there's already something planned regarding this behind the scenes.

[WhyNotHugo]

Alright, nice. Feel free to use the above set of scripts if they're any use.

Ping me when you have anything concrete; I'd like to confirm that the font format is usable on desktop applications (gtk, Qt, foot, and a few others). That's really the main use case for distributions downstream.

There's a few different ways that coloured emoji can by encoded into ttf files, and so far I know of one that's rarely supported and one that almost universally supported, but I've no idea about the others.

[mavit]

Assuming these Emoji don't use gradients, COLRv0 built with nanoemoji is the way to go for TTFs, these days. Have a look at the OpenMoji project for example GitHub automation. I hope to get this into Fedora at some point, but it's a question of finding the time to package all of the prerequisites.

[WhyNotHugo]

I quickly tested the COLRv0 TTF fonts from the OpenMoji project:

They render okay in foot, although fonts render rather small compared to twemoji:

On alacritty, the flag is missing. I'm not sure if the font is missing it, or it's a rendering issue.

There's twemoji/CBDT on alacritty for reference (note that flags don't render properly; that's an alacritty bug).

On GTK applications, it seems to render okay. On Qt applications, this font does not render. There's just a blank space.

The current font being packaged in Alpine, ArchLinux and Fedora is CBDT, and works in all the above (including GTK and Qt).

[ez-me]

Android uses CBDT, so for root users, or custom roms it could be useful.

I'm not sure exactly how Windows handles Emoji, besides their documentation stating that they support all major methods (Google NotoColorEmoji has a separate CBDT file that says Windows Compatible)

[Sav22999]

Any news about this topic? Will you publish ttf fonts as well?

[WhyNotHugo]

TTF files are available here: https://artefacts.whynothugo.nl/twemoji.ttf/2023-12-05_16-23/

These are not official builds. The build manifest used to generate these is the same as the previous one mentioned above.

These are CBDT, and work in most Linux desktop applications (Qt, GTK, a few terminals, etc).

[ez-me]

These are CBDT, and work in most Linux desktop applications (Qt, GTK, a few terminals, etc).

Could you add the EMOJI_WINDOWS in the make command? The file that's created by default isn't compatible. Or maybe replace it entirely, as both seem to work fine on Linux desktop (plus Android).

[Gontier-Julien]

I think that this would be an awesome milestone to have before the Unicode 16 is drop ^^

[literon36]

Twemoji needs a release as a ttf since using the up to date twemoji on a Linux distribution outside of arch / fedora is quite difficult. There used to be this project https://github.com/13rac1/twemoji-color-font/ but it hasn't seen a release since Unicode 14...

[jdecked]

I don't have the time right now to maintain a TTF font (really, to maintain a build pipeline for one), but I've pinned this issue so hopefully anyone looking for a font can use OP's helpful build instructions.

If anyone is willing to maintain such a build pipeline, I'm happy to promote the resulting font file/repo in the README.

[CoelacanthusHex]

Hi, for COLR version you can try https://github.com/mozilla/twemoji-colr/pull/74

[WhyNotHugo]

TTF builds of 15.1.0 are available here: https://artefacts.whynothugo.nl/twemoji.ttf/2024-06-06_17-55/Twemoji-15.1.0.ttf

[literon36]

@WhyNotHugo thanks for the ttf builds!

If anyone is willing to maintain such a build pipeline, I'm happy to promote the resulting font file/repo in the README.

I think the sourcehut repo / the files on the website would fit @jdecked's description of promotion... I'd suggest adding the links somewhere at the top of the page so people just looking for the font can find it directly.

Also... couldn't the build pipeline be automated via GitHub actions? Artifacts could then be easily downloaded via https://nightly.link/

[WhyNotHugo]

Also... couldn't the build pipeline be automated via GitHub actions?

It could be, but I fail to see any value in porting something from an open source CI into Microsoft's proprietary one.

The builds themselves happen on an Alpine image is really just some lines of shell script that glue together scripts from different sources: https://git.sr.ht/~whynothugo/twemoji.ttf/tree/main/item/.build.yml They should work on any distribution which provide the necessary dependencies (which are pretty ubiquitous AFAIK).