Add localization (internationalization)

ivan-aksamentov commented 1 year ago

Resolves: https://github.com/hodcroftlab/covariants/discussions/292 https://github.com/hodcroftlab/covariants/issues/311

This adds internationalization (i18n). Simply speaking the text in the application is now translated to multiple languages, and you can now switch between these languages.

What's new:

[x] there is a dropdown on nav bar, which allows to switch locale (language)
[x] the last selected locale is remembered in Local storage of the browser, so page refreshes and repeated visits preserve the locale
[x] ~the lang URL parameter now allows to set language in the URL, so that user navigating to this URL will see the app in this language (overriding Local storage)~ [it was braking the router; temporarily disabled]

Implementation:

Hardcoded strings

Strings hardcoded in the app code (like text of buttons or of nav bar links) are now translated using react-i18next (1, 2). Also dates and numbers are translated using other handy packages. All translatable strings need to be surrounded by t() function, which comes from useTranslationSafe() hook.

The usual string interpolation in these strings does not work (interpolation will happen before the string is extracted for translation)! So it needs to be replaced with string interpolaton of i18next format:

"Hello, ${name}!" -> t("Hello, {{name}}!", { name: "Covariants" })

i.e. interpolated variables should be additionally set using a dict, passed as the second param to the t-function. Try to pick variable names that are different from keys in the dict (i.e. don't use JS shorthand syntax), because any refactor that leads to variable rename will break the interpolation (the variable in the template string will not be renamed by the refactoring tools), and the app will be showing uninterpolated content without any warning.

Maintenance procedure for hardcoded strings:

yarn i18n:extract: Extract English strings. The tool parses the app code and takes strings surrounded by t() and puts them into web/src/i18n/resources/en/common.json.
yarn i18n:extract: This adds additional words from web/src/i18n/additional_keys.json to the web/src/i18n/resources/en/common.json. These are some of the words which appear in the UI, but are not in the code, for example because they come from data (e.g. names of countries and of continents).
yarn i18n:translate: This translates the English strings from web/src/i18n/resources/en/common.json and creates or updates web/src/i18n/resources/<locale>/common.json for each locale we support. This uses AWS Translate paid service and requires AWS credentials configured. But of course the translation can also be done manually, without using any services. For that, just add the necessary key-value pairs to the JSON file corresponding to a language.
yarn i18n:fix: Massages the resulting translations, and applies some of the obvious fixes, because AWS Translate is not perfect. For examples, it omits spaces around string interpolation.

Markdown content

Markdown content files have been moved from content/ to web/src/content/${locale}. For now there is only en/ directory for English locale. Other locales should be just copies of the en/ directory, but with content in the corresponding language.

The names of directories should be the same as here, or here, or as the letter code in the language selection dropdown in the app.

Markdown content now should only be loaded using <MdxContent filepath="${filepath}"/>, where filepath is the path relative to web/src/content/en (or any other locale than en, they are mirrors).

If localized md file is missing, the English version will be loaded as a fallback.

Maintenance procedure for markdown content:

I haven't figured how to automatically bulk-translate markdown yet. It's MDX actually, because we use React tags there. So this will likely need to be done manually. Just keep md files in web/src/content/${locale} for each locale in sync with English locale in web/src/content/en. "Just". It's a large upfront work, and it also requires expert knowledge every time something changes in English version. Typically you need many experts, per language, because rare human knows more than a handful of these.

Further work:

What languages do we need?
The app and its code needs to be inspected and the missing hardcoded strings need to be surrounded with t(), i.e. "Hello" -> "t("Hello")".
check md files included into md files: I tried to replace all imports with <MdxContent/>, but might have missed something.
check if any of the strings in the app contain extra $, { or } . This might have been left over by me when converting string interpolation from JS to i18next format.
write translations for md content
obviously, improve translation quality. AWS Translate often generates very poor quality translations.
Support plurals, cases, declinations, etc. This is a big work - each string needs to be annotated with grammar context. Probably out of our reach.

vercel[bot] commented 1 year ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
covariants	✅ Ready (Inspect)	Visit Preview	Jun 13, 2023 10:32am

emmahodcroft commented 1 year ago

List of files that contain large chunks of text that would need to be translated if translation were made possible:

covariants/content/: Faq.md Home.md NameTable.md PerAreaIntro.md PerClusterIntro.md PerCountryCasesIntro.md RegionBreakdownInfo.md SharedMutations.md VariantsPageIntro.md

Every variant page in the clusters folder (currently 48)

Each of the country intros in PerCountryIntro folder

I think this is all the pages...?

hodcroftlab / covariants