Many emoji variation sequences are not recognised

Unicode Emoji 5.0 added variation sequences for a few dozen emoji-default characters because they map to entities in the Wingdings/Webdings sets. Twemoji still does not recognise those, which has three consequences:

They cannot be forced to display as text style with VS-15.
If they are entered with VS-16 (which is not required, but valid), then they take up 4 characters rather than 2 because the variation selector is counted separately.
Entering them with VS-16 results in an empty image being inserted after the emoji with VS-16 as its alt text, at least on TweetDeck. This image causes a small but visible gap between the emoji and the following text.

The affected characters are listed in these two tweets:

None of these should be substituted with images, but so far only ♀︎, ♂︎, and ⚕︎ are handled correctly.

You should be comparing from a different website. Twitter, like Github and Facebook, often substitute emoji as images, rather than using the image font. Twitter specifically avoids rendering certain glyphs/emoji with VS16 for graphical representation, and keeps them as text, even when you provide VS16:

In practice, Twitter emojifies bare -EP +EPSq, and ignores ©®™ even with FE0F. - Source Tweet

Beyond that, whatever is handling the font rendering has varying support/behaviour. Firefox and Chrome are somewhat different for example, and this also varies between the same browser across platforms afaik.

If you'd like to do a comparison of how rendering is working with and without Variation Selectors(since each one has their own default presentation style without the added codepoint), Wikipedia has a nice little table here. Here's a list of all the variants, and here's a test page.

⚕︎ btw is rendering as an emoji for me. Try viewing them on a site like emojipedia, select the emoji in the copy text field, inspect, and disable the font-family CSS, on Chrome that should render it as whatever your system default would(note it has the VS16 codepoint). Not sure if this is reliable on github, but here you go ♀︎(VS15 - &#x2640&#xFE0E, copy/paste that into html p tag if needed)♀️(VS16 - &#x2640&#xFE0F). You can verify the codepoints by copy/pasting tothis site.

I've not noticed Chrome to respect Variation Selectors, Firefox tends to do a better job of it. You can see some comparisons across my Linux desktop and Android with Chrome and Firefox here.

According to that emojipedia link btw, the Female Sign emoji is meant to have VS16 support, and presumably only that with Twemoji.

I checked on Firefox with it's embedded version, it will render the colour/emoji presentation(VS16) in the tab bar and input fields(all CSS styles for font-family on the page disabled) it seems to respect VS15/VS16. The text variant is rendered with DejaVu Serif. Actual text content, like in a paragraph tag is all in text form with DejaVu Serif(or any font that you want to override with such as an emoji one). Right click context menu is also respecting VS16.

Chrome on the other hand is doing it differently, text content is rendering all in emoji(Noto Color Emoji), and the text input field is rendering Liberation Sans(fallback for Arial defined by Chromes user agent stylesheet). Looking into that further, Chromes defaulting to sans-serif font when no font is specified, while Firefox is going for serif, firefox also has an input field font via user agent assigning "Noto Sans"(dev tools report this, it's probably from the system rather than hard coded in browser stylesheet).

Have you got any fonts that are behaving the way you want, that respects Variation Selectors? I don't personally know enough about fonts, but I am doubtful this is something the fonts themselves can fix(unless packaging both variants into the font?)

I was under the impression that the bit of code that replaces emoji characters with images on Twitter web and TweetDeck was part of the Twemoji library rather than being unique to those respective sites. If that isn’t the case, I will file the same bug report on the correct project again.

This has nothing to do with fonts – TweetDeck and the Twitter web client do not use fonts to display emoji; they use embedded images. The trouble is that a few dozen variation sequences that were added to the Unicode standard in 2017 are completely ignored by the responsible script. It is currently impossible to prevent characters like U+1F378 🍸 COCKTAIL GLASS from being substituted with a Twemoji image on Twitter, even though <U+1F378, U+FE0E> has been a valid variation sequence for two years now.

And because these newer variation sequences are not recognised, this also breaks Twitter’s tweet length calculation. Every RGI emoji is supposed to be counted as two characters. Emoji like U+2648 ♈ ARIES are in fact counted as two characters, both with and without VS16: U+2648 alone counts as two and the entire sequence <U+2648, U+FE0F> counts as two as well. But if you write the perfectly valid emoji sequence <COCKTAIL GLASS, VS16>, that will count as 4 because Twitter does not know that this sequence exists.

TL;DR: AFAIK, the emoji do not support rendering both variants on those platforms? In which case the difference you'r experiencing is by those platforms(Twitter/TweetDeck) using the twemoji JS library which looks to see if a codepoint is here, and if it is, replace the text with an image element in the HTML. Otherwise leave the content as text.

@charlotte-buff Ok... but if you think it really is the fault of this font, then you should verify by trying Twemoji elsewhere on the web, not on platforms that convert emoji into images to present as they're no longer presenting the fonts(some emoji font use SVG, others embed bitmaps/images but they're treated like text still in HTML).

🍸 🍸︎ 🍸️ The following is wrapped in inline code block which should avoid Github converting to image. It is the base codepoint(default emoji presentation), followed by VS15 version, and then VS16.

AFAIK, you can't mess with the CSS by setting the Twemoji font, otherwise they'll all render with the assigned font regardless of what Variation Selector is used. Your system or browser should handle it as long as the initial font used doesn't have the emoji from what I've seen.

For me Chrome renders them all as the same font. Firefox is showing it's browser emoji font(in "about:config") which is set to Twemoji. The middle one with VS15, is rendered with a different font(a recently installed Twemoji based on this repo, which for some reason has glyphs at 50% of the size, lot of padding around the emoji/glyph). So Firefox is technically correct, I'd just need to prioritize a black/white font like Noto Sans Symbol2, and then it should behave correctly.

If that isn’t the case, I will file the same bug report on the correct project again.

I do not use either, but the twitter examples you provided, you can inspect the HTML to see that the colour presentations have been replaced with actual image elements. Even if they look the same as what Twemoji emoji look like, they're no longer using the font, it's just Twitter wanting the emoji to render the same style consistently to all users to avoid the font problems that can happen. If they're doing extra handling to sometimes treat glyphs differently from others(like the linked tweet confirming some are forced as text), then that's specific to their website.

This has nothing to do with fonts – TweetDeck and the Twitter web client do not use fonts to display emoji; they use embedded images.

Right. So why are you raising an issue about such here then? This is a project for an emoji font by twitter. It does not behave in the same way(replacing characters/emoji with HTML img elements).

The trouble is that a few dozen variation sequences that were added to the Unicode standard in 2017 are completely ignored by the responsible script.

Perhaps it's related to the special handling already mentioned. If so, it might not have been updated since, but that has nothing to do with this font afaik.

even though <U+1F378, U+FE0E> has been a valid variation sequence for two years now.

If you'd like to test it off Twitter/TweetDeck, you can do so here, the cocktail emoji is on the 2nd row. It is valid and working fine for me.

this also breaks Twitter’s tweet length calculation.

Twitter bug then. Calculating the length of text can be confusing/hard, the article shows how languages themselves can differ with what they calculate. But it's also probably related to special handling by Twitters code, not this font, which alters the length it contributes.

I was under the impression that the bit of code that replaces emoji characters with images on Twitter web and TweetDeck was part of the Twemoji library rather than being unique to those respective sites.

Maybe. I am only commenting from the perspective of the font itself. Your original issue wasn't clear enough on that concern, that it seemed like you were raising issue with how the font worked, not any supplementary library that Twitter and third-party clients might use.

Having a quick look, here's the exclusion method mentioned on the README(at least for v1 of the API). It looks like the related code you're interested in is twemoji.parse(), although that method seems to state it will ignore any characters with VS15 for image replacement. The Regexp it refers to doesn't look like there is any reason for it not to handle the cocktail emoji any differently than others.

Ok I've figured it out for you. You did not confirm if using VS16 would provide emoji/image variants of the text representations that you said do present as text with VS15, and I don't use those platforms to be able to verify my understanding, but I'd say I'm on the money and you've not verified the other way around?

It seems that twemoji.parse() will provide a valid URL for image replacement if found.

Cocktail emoji: https://twitter.github.io/twemoji/72x72/1F378.png Female Sign (Doesn't work): https://twitter.github.io/twemoji/72x72/2640.png Male Sign (Doesn't work): https://twitter.github.io/twemoji/72x72/2642.png Staff of Aesculapius (Doesn't work): https://twitter.github.io/twemoji/72x72/2695.png

Those URLs seem to match same results as the maxcdn one(eg https://twemoji.maxcdn.com/72x72/1f378.png ) that gets used by twemoji script. I'm not sure why this happens, as the emoji/glyphs are actually in this repo, for example Female Sign. I guess they're excluded from the CDN they're distributed to, perhaps to selectively ignore them?

You can see here. Perhaps it will render differently for you, so here are some screenshots of output from my browsers:

Chrome: codepen_chrome

Chrome - Context Menu: codepen_chrome_context

Firefox: codepen_firefox

Firefox - Context Menu: codepen_firefox_context

Some of the disparity is because I'm messing around with my fontconfig to get more consistent/correct rendering on my system for emoji. I need to reboot for some changes to take effect I think, and Firefox or Chrome hasn't been restarted since the last change I made to my fontconfig. That said, Firefox tends to handle Variation Selectors better than Chrome afaik(at least with the browser emoji font support in "about:config"). The last emoji in that example which is missing in Firefox context menu does render in colour as expected too btw.

EDIT: Twitter CDN used by the actual twitter site does have the emoji for Female Sign and others.

My guess would be that perhaps code like this is stripping the FE0F VS16 codepoint and rendering the emoji version. The use of FE0E VS15 codepoint does not have handling for being stripped/detected. Afaik, the specific ones rendering as text, do so by default?(they need VS16 to technically render their colour/emoji variant)

None of the three you mentioned that render as text are present in this file which is used by a regexp to lookup emoji to differentiate from other text content here in code. That is the cause.

So any website/app that uses twemoji library for detecting emoji out of text content will need to apply the regexp to a different file. If you do get colour versions, it's because the code specifically looks for characters/glyphs that end with the VS16 codepoint, then strips it, but also grabs the URL from the looks of it if available/valid.

I have added a line before the parse command in my linked codepen:

twemoji.base = "https://abs.twimg.com/emoji/v2/"

This still only uses the cocktail emoji as an image, so I assume that's the case if you try to do any of your text rendered glyphs with VS16? If not, then there's some other handling going on elsewhere.

@polarathene:

This is a project for an emoji font by twitter.

No, it is not. This repository contains the Twemoji graphics (AI, SVG and 72px PNG) and several JS scripts to support substituting emoji characters by those images.

Fonts can be build based upon the graphics and this is an example of just that.

@charlotte-buff The relevant regular expression is found in the sibling project twemoji-parser. It claims to be generated by a shell script, source/emoji/scripts/generate.sh, that I could not find. (Perhaps itʼs in twitter-text now.) This seems to be the cause of the problem.

@Crissov thanks for the clarification, I've been through a lot of tabs on fonts and emoji with Linux lately(especially fontconfig related), twemoji stuff I recall various github repos, one being that you linked to. I thought for sure I was responding to this issue on a repo about the font, apologies.

Probably from coming across this issue, when the actual building of the font is built as shown here, pulling in this repo for the assets, like you described.

It claims to be generated by a shell script, source/emoji/scripts/generate.sh, that I could not find.

It's an internal script, not available as open-source.

This seems to be the cause of the problem.

That script likely still uses this file which is referenced as a source here in the library code. The dummy /twemoji/ value gets replaced at the end of the script with the generated regex you've linked to. The comment still seems to indicate it was utilizing the mentioned txt url.

twitter / twemoji

Many emoji variation sequences are not recognised #363