multi-lingual CWL guide

[mr-c]

• Fix inter links issue by converting the markdown to RST
• [x] configure sphinx to build a multi-lingual site https://github.com/common-workflow-language/user_guide/pull/442
• [x] build English version at en/
• [ ] add redirect static pages at the old paths pointing to the new paths
  • [ ] the key is <meta http-equiv="refresh" content="0; url=""/>

Later:

• [ ] once the above is done, add a language switcher to the header and/or sidebar
• [ ] add a "translate this page" link to the sidebar
• [ ] Add a credit for the translation platform hosting. Maybe rename the "LICENSE" page to be about the User Guide itself and include credits & contributing information there?
• [ ] bonus, can we auto-guess the language based upon the Accept-Language header, for those who have it set?

~~See https://github.com/swcarpentry/styles/issues/210 especially this list of Jekyll plugins for multilingualism: https://github.com/Anthony-Gaudino/jekyll-multiple-languages-plugin#10-other-language-plugins~~

[kinow]

An existing extension for Sphinx & internationalization: https://pypi.org/project/sphinx-intl/

[mr-c]

👍 That plugin can be used with https://docs.weblate.org/en/latest/devel/sphinx.html

[swzCuroverse]

@kinow , @mr-c what is left to do on this ticket? Is this just setting up the capability to show and create the translations appropriately or do we have this as a placeholder that we need the translations themselves.

[kinow]

@swzCuroverse the past user guide website had a drop down at the top right corner, where the user could choose a language and then get a Google Translate version of the user guide. That was not ported to the new user guide.

We can either add that back, or allow for community contributed translations. Either using GitHub and Sphinx plug-ins directly, or using an external site like Transifex and a Sphinx plug-in. Or both options could be used, and have just one page translated introducing CWL.

[swzCuroverse]

@kinow If I have someone to help with some of the translation work (if we to have the one-page translated)?

[kinow]

@kinow If I have someone to help with some of the translation work (if we to have the one-page translated)?

That would be great!

I noticed that the user guide has sporadic contributions from users during hackathons and conferences, and new users and contributors joining the project. I wasn't sure if it would be doable to try to translate the whole project using something like Transifex as other projects do.

So my idea for this issue when I was working on the user guide was to do something similar to what Apache Jena did: translate basic information about the project (same page, in different languages). I think something that would give users an initial idea of what is CWL (standards, community, project, etc.).

Maybe we could have the Quick Start page translated.... or all the pages under Introduction. Or, another alternative would be to have a separate page to introduce users, similar to the page about CWL in Japanese linked in the CWL homepage. I prefer having the Quick Start or Introduction translated, instead of a new page… and that's something I would volunteer to keep up to date in Portuguese and maybe Spanish.

As for the implementation, someone needs to spend some time comparing the existing ways to provide multi-lingual support in Sphinx. I know there are existing extensions, with permissive license, but I never used them :disappointed_relieved:

Cheers Bruno

[swzCuroverse]

@Mackenzie-OO7 thought they might want to help with front-end part. Tagging them here.

[kinow]

Excellent! @Mackenzie-OO7 feel free to look at the existing discussion and either test some of the links above, or if you have any other suggestion or idea, just let us know! Thanks!

[swzCuroverse]

" prefer having the Quick Start or Introduction translated, instead of a new page… and that's something I would volunteer to keep up to date in Portuguese and maybe Spanish." --- I agree.

[swzCuroverse]

@kinow I am going to experiment on a branch to see if I can get a simple version going. Is it OK, if I ping you for help if I get stuck :)

[kinow]

Sure thing!!!! Thanks @swzCuroverse !

[swzCuroverse]

Ok - so here is the basic plan :+1:

• [x] generate PO files
• [x] connect repo with PO files to webplate
• [x] TRANSLATION happens
• [x] Sphinx is configured with multiple languages
• [x] each language loads the appropriate files to generate the HTML files during build
• [x] have continuous localization

Not quite sure about 4 and 5, tagging @kinow . However we can start on 1 and 2.

For (1 and 2) - this is probably the best guide to start -- I would focus on this and getting the po files in weblate. https://docs.weblate.org/en/latest/devel/sphinx.html and https://docs.readthedocs.io/en/latest/guides/manage-translations-sphinx.html

For (3) there are bunch of tools that may or maynot work here -- but weblate has a section of https://docs.weblate.org/en/latest/admin/machine.html (6) We probably want to set it up so that when the repo changes things are automatically update in Weblate -- https://docs.weblate.org/en/latest/admin/continuous.html

Also just probably need to read through the doc in general :)

Once we get set up there is a "promotion" section -- https://docs.weblate.org/en/latest/devel/share.html# --- this is what we can probably share on the forum, etc to let community members know what is happening and ask if they can contribute

[swzCuroverse]

Also -- https://docs.weblate.org/en/latest/admin/continuous.html#push-changes

[mr-c]

(1) generate PO files

Here you go https://github.com/common-workflow-language/user_guide/pull/359

[mr-c]

Which language will be translated into first?

Hosted weblate allows for bootstraping new languages via one of the following services

• Apertium APy
• Baidu
• DeepL
• Google Translate
• LibreTranslate
• Microsoft Translator

We might want to pick a different engine for different languages, so please let me know

[mr-c]

I translated the https://www.commonwl.org/user_guide/LICENSE.html into pt and es using https://creativecommons.org/licenses/by/4.0/deed.pt and https://creativecommons.org/licenses/by/4.0/deed.es

But rendering that using make -e SPHINXOPTS="-D language='pt'" html shows this bug:

https://github.com/executablebooks/MyST-Parser/issues/690 "Internationalization does not work on indirect links"

@kinow Is there a particular reason we use markdown instead of rst?

[kinow]

@kinow Is there a particular reason we use markdown instead of rst?

Answered on Element, just copying the answer here:

The idea was that more contributors would be familiar with markdown than rst. We are using markdown, but our setup was based on jupyterhub's & pydata initially, where both can be used.

I'm syncing my branch and testing this now :+1:

[kinow]

Reproduced the issue after changing locale_dirs = ['locales/'] to ../locales (thanks @m-rc`). Nothing in the logs of Sphinx, no errors nor warnings. Searching some more...

To note that make -e SPHINXOPTS="-D language='pt'" html worked, but using watch instead of html showed me the English version (or maybe I had to force-reload, but I'm using html + python -m http.server --directory _builds/html 8000 to test it for now). (EDIT: got a fix for that, will send a PR later).

[kinow]

Now I know where the problem happens. TL;DR #358 is probably the fastest fix. It's due to the order things happen during the parsing & translation in myst/sphinx/docutils.

It's due to the order (1) MyST parser parses links in a document (for Docutils), (2) calls Sphinx transforms including the Locale one, which (3) replaces the document text.

At (3) the document object has a structure like <paragraph>Todo o conteúdo blabla <<reference refuri="https://">Common Creative...</reference> ....</paragraph>. But what the Locale transform uses is the text Todo o conteúdo blabla Common Creative....

At (1) the parser uses the context to build the links and place them in the document object. At (3) the document object doesn't have a structured with the links, and the Locale transform uses the text only anyway. Locale will (4) call the MyST parser again for the translated text, and at this point the MyST parser won't do anything about the links as all the text it has is just one paragraph of the Markdown file.

I think I might be able to come up with a way to monkey-patch MyST to fix it in the next days/week, and produce a pull request to MyST Parser. Alas, it could take a bit longer due to how sphinx / myst / docutils work with translation. So if there's a deadline for the translations/internship, the PR @mr-c created to convert some pages to reST might be the quickest and best solution for now :+1:

Cheers -Bruno

[kinow]

weblate & translation related videos from FOSDEM 2023:

• Weblate community BoF Meetup, feedback sharing, and discussion
• Bringing your project closer to users – translating libre with Weblate
• Translate All The Things! An Introduction to LibreTranslate
• Should there be a standard in libre localization? Ideas on how to make it easy for translators to contribute to any FOSS project they like

[mr-c]

@kinow I added some more details to the first comment

[kinow]

@kinow I added some more details to the first comment

Thanks!

[mr-c]

FYI:

• https://www.commonwl.org/user_guide/en
• https://www.commonwl.org/user_guide/ja/
• https://www.commonwl.org/user_guide/pt_BR/
• https://www.commonwl.org/user_guide/pt_PT/
• https://www.commonwl.org/user_guide/zh_Hans/