Incomplete localization / emoji names and keywords hardcoded in English

[baumerdev]

You can choose a locale which changes the language of the interface and the categories. But the emoji names and keywords still are in English, because they are in the data files and not in the i18n files

Data file with hard coded names and keywords: https://github.com/missive/emoji-mart/blob/main/packages/emoji-mart-data/sets/14/native.json Example for translation: https://github.com/missive/emoji-mart/blob/main/packages/emoji-mart-data/i18n/de.json

There are translated Emoji names and keywords available in https://github.com/unicode-org/cldr-json/tree/main/cldr-json/cldr-annotations-full/annotations https://github.com/unicode-org/cldr-json/tree/main/cldr-json/cldr-annotations-derived-full/annotationsDerived

So for my case I wrote a little script that combines the emoji-mart data with the data from cldr

import fs from "node:fs";

// ️Variation Selector-16
const VS16 = "\ufe0f";
// Zero Width Joiner
const ZWJ = "\u200D";
// Remove VS16 and ZWJ from emoji to compare CLDR / emoji-mart data
const strippedEmoji = (emoji) => emoji.replaceAll(VS16, "").replaceAll(ZWJ, "");

// Read emoji-mart data
let data = JSON.parse(fs.readFileSync("data.json", "utf-8"));

// Read CLDR data
let deFull = JSON.parse(fs.readFileSync("cldr/de-full.json", "utf-8"));
let deDerived = JSON.parse(fs.readFileSync("cldr/de-derived.json", "utf-8"));

// Combine data
for (let emojiId in data.emojis) {
  let emojiData = data.emojis[emojiId];
  let emoji = strippedEmoji(emojiData.skins[0].native);

  const deFullEmoji = Object.keys(deFull.annotations.annotations).find(
    (deFullEmoji) => emoji === strippedEmoji(deFullEmoji)
  );
  if (deFullEmoji) {
    emojiData.name = deFull.annotations.annotations[deFullEmoji].tts[0];
    emojiData.keywords = deFull.annotations.annotations[deFullEmoji].default;
  }

  const deDerivedEmoji = Object.keys(
    deDerived.annotationsDerived.annotations
  ).find((deDerivedEmoji) => emoji === strippedEmoji(deDerivedEmoji));
  if (deDerivedEmoji) {
    emojiData.name = deDerived.annotationsDerived.annotations[deDerivedEmoji].tts[0];
    emojiData.keywords = deDerived.annotationsDerived.annotations[deDerivedEmoji].default;
  }
}

// Write lolcalized data
fs.writeFileSync("data-de.json", JSON.stringify(data));

After that I can use the data in my Picker and have localized names shown and can use the search with localized keywords

import dataDE from "data-de.json";
// ...
<Picker data={dataDE} locale="de" />

Don't know if this is something that should be integrated somehow into emoji-mart itself. It'd mean either providing native/Google/Apple/... data for each language or to separate the localization part from these data files and moving it to the current i18n files which I guess would mean more code changes.

[ghost]

@baumerdev @EtienneLem @nolanlawson @rugk

I am trying to apply this solution. I am trying to output translated emoji names and emoji keywords.

I am not able to do it, the instructions here are not enough.

If Emoji-mart is not going to offer translated emojis by default, it should at least offer an official tutorial and offer on yours README.md. Please make a tutorial based on the current one here, so people could go and make their own implementation of translations for the languages they need.

I don't see how difficult that would be. It would actually be a prove that emoji-mart is really developed for internalization.

Or if you don't want to write a README.md, at least point me to a working solution. Take for example, french or german, is there somewhere a working example of how to display emojis in emoji-mart that are translated to french or german?

I tried so far downloading translated .json from CLDR, download from emojibase, and I am having difficulty with the conversions, the internal functioning of emoji-mart, regarding how it receives the translated data. I appreciate so much your library, but simply some users do not know english at all, it is not acceptable to display emojis only in english.

[baumerdev]

I don't know if I have gained the wrong impression, but I find the tone of the post inappropriately demanding for an open-source project. 🤔

Anyways, here is my example as a step-by-step instruction.

1. Download https://raw.githubusercontent.com/unicode-org/cldr-json/main/cldr-json/cldr-annotations-full/annotations/de/annotations.json and save it as ./cldr/de-full.json
2. Download https://raw.githubusercontent.com/unicode-org/cldr-json/main/cldr-json/cldr-annotations-derived-full/annotationsDerived/de/annotations.json and save it as ./cldr/de-derived.json
3. Download https://raw.githubusercontent.com/missive/emoji-mart/main/packages/emoji-mart-data/sets/14/native.json and save it as ./data.json
4. Create a file ./convert.mjs and paste in the content from my example code above, the one beginning with import fs from "node:fs";
5. Run node convert.mjs
6. You now have a file data-de.json that you can import, see the example above beginning with import dataDE from "data-de.json";

If you need any other language, you have to replace the de in the URLs, file names and within the convert script.

[ghost]

I really appreciate your immediate response and your kindness in indicating the path regarding how you managed to do it. Now I got it! Following instructions currently available, I was not able to reach it.

Oh I see... I didn't meant to be rude or something. I tried my best to bring the problem that I faced previously, and instead of only complaining and requiring someone to solve my specific scenario for free, I pointed out the sense of opportunity that is available for this library regarding translation of emojis. I am not requesting this open-source library to offer the translations natively, although that would be appreciated. I am not requesting anything so difficult, I offered two different alternatives that may even not be the best ones available. But at least can be considered to lead to a solution: either to write a few lines on readme.md or to simply mention an existing solution for us to observe. Some other projects do this in their documentation, so I don't see how requesting these things are more demanding solutions than what is already available. Yeah sometimes I write in a rush, it is important someone tell me to think about my tone, thank you, and I am sure that my suggestions do not exhaust the discussion. Writing a more detailed documentation is going to increase quality, I would not help anyone if I were to be quiet on this issue. For sure the owner of this project is going to appreciate receiving my feedback as an indication regarding where it can improve things, and they now are in position to consider exposing your instructions in their documentation of this library, in order to allow other people to achieve it from now on... I don't see how that is going to be much demanding. Respectfully

[AlexAndBear]

Just stumbled over this issue. Big thx to @baumerdev for the solution 👏

I'd really love, to see something like this to be implemented in the lib itself if possible.

So we always will have a synced state if a new emoji version comes out.