[DISCUSSION] Migrate docs to markdown.

[Andries-Smit]

Issue

Based on technical analysis #999, we could state that the quality of the multi-lingual documentation is poor with over a 1000 broken links and build warning. Despite the many people who have contributed wither their best effort and valuable time in translating the texts, but it resulted in broken mark-up, links and unreachable documentation.

“As a (potential) user I do not trust my live to a medical product that does not have quality documentation!”

The documentation is the first point of interaction for a new user. And the user will assume the there is a relation to the quality of the documentation and the quality of the product. And if he or she does not understand the product, will not start using it.

What should we do? I am sorry to raise the question. Should we continue this way?

Our question is: the effort of the translator’s and maintainers, should it be spent on average to low quality (translated) documentation, or should it focus on have 1 documentation language in excellent quality? Translations are less important these days as more machine translations tools are available and with reasonable to great quality! (What do you think?)

We should either facilitate an easy translatable documentation flow while maintaining the technical quality or drop some languages and make the view documentation languages shine. My suggestion before we give up translations; use markdown instead of restructure text:

• Most issues (build warnings) in the translation are caused by crowdin (machine suggested) translations to break the mark up.

• Links to anchors are broken, /Sample-Setup.html#required-components as the heading Required Components is translated it will always break all the link to the anchor.

  To give translated documentation another shot. -My suggestion is do 2 actions;

  • convert the restructured text to markdown and use untranslated header anchors.
  • Language independent anchors

convert to markdown

Pro Markdown

• Supported by crowdin (rst is not)
• Links are harder to break. (not space sensitive)
• No accidental translation of the rst directive
• MD is easier to understand by users (rst is less common and more technical). google trends
• md is less technical. rst is more technical a documentation language for programming languages. We don’t use any of the advanced features.

Cons Markdown

• Migrating files is an effort. Converter
• Some pending translations might get lost in the migration.
• Some rst feature are missing. work around

Are we missing some pros and cons… let us know.

Migration Actions:

• Sync/upload translations from GitHub
• Gradual migration, one file at the time (and all its translations).
• Approve pending translations.
• Merge translation branch into master
• Fix build warning and link warning in rts files
• Use Pandocs to convert to markdown (no wrap)
• Fix heading level, and page title
• Use standard formatter
• Commit and merge into master
• Upload new files with translation to crowdin.

Fix heading anchors

Page URLs are not translated. However all heading anchors are auto-generated which is awesome, however if you work with translations, anchors changed with the heading is translated.

For language independent anchors in markdown we can leverage eval_rst

```eval_rst
.. _settings-aaps:

Settings in AAPS (English) / ## Instellingen in AAPS (Dutch)

TODO check alternative [MyST](https://myst-parser.readthedocs.io/en/latest/using/syntax.html#targets-and-cross-referencing) syntax, seems simpler `(heading_id)=`

**Actions:**
- After migration to markdown
- Add the manual anchor
- Let crowdin propagate the added link into the translated files
- The old anchors will stay available. 
- Clean up translated anchors (one at the time :( )
- Consider removing the auto generated headings....

Please share your thoughts on links, quality, `md` vs `rst` and translations…

[Suchiman]

:+1: for Markdown over RST. I personally found crowdin to be not very intuitive, as it sometimes appears as if its hiding whether you're working on plain text or markup which then leads to the markup breaking. Some things are not easily (automatically) translated, for example, the menu option text's on the pumps, This is something I've contributed fixes for in the past.

While i could live with only english docs, i can only speak for myself but i've encountered several people on gitter for which that would be a hurdle.

One thing i'd like to see would be using english filenames for the translations as well, such that you can switch between the versions by simply substituting the language in the url, for example: en: androidaps.readthedocs.io/en/latest/Usage/Open-APS-features.html de: androidaps.readthedocs.io/de/latest/Usage/Open-APS-features.html That would make it unnecessary to translate the relative links and reduce the risk of them breaking as well.

[Philoul]

The reason of so many dead link is the necessity today to translate them... the other reason is 2 differents format rst/md with different rules, and sometimes the proposed translations by crowdin introduce extra spaces that could create broken links. The only rule that works in all languages is "never translate a link, never translate an anchor"... last point, there are no integrated tool to verify broken links (I ran recently linkcheck utility on web site, but you told me it's not the right way to make this verification... (french and german are I think the translations with the greatest % of translated pages and the lowest % of broken links, but I think it's because we worked a lot on it, Achim mainly for german and me for french). I think rules for link translation, even if explain in doc, are too complicated for occasional translators...

[xJoe]

I agree with links being the biggest problem at the moment and I am happy to hear your proposal for a solution which I second. Btw @Suchiman filenames are English only at the moment. The problems are caused by links to headlines (i.e.

• https://androidaps.readthedocs.io/en/latest/Installing-AndroidAPS/Building-APK.html#this-article-is-divided-into-two-parts
• https://androidaps.readthedocs.io/de/latest/Installing-AndroidAPS/Building-APK.html#dieser-artikel-ist-in-zwei-teile-geteilt

But I disagree with automatic translation for several reasons:

• Topic is partly very specific and would lead to bad automatic translation. (Can often be seen in Crowdin by the proposed automatic translations.) Additionally grammar in German is partly poor.
• I pay close attention that menu translation are the same in app and docs.
• Bad translation related to both upper topics would also have impact on people's trust as described in your quote. (Btw. I do not completely agree with this point as potential users are aware that it is DIY. A lot of people spent their sparetime for this project and as it is completely free for users they must accept some imperfection. If they can't they must go for commercial systems and accept their restrictions.)

Moving from RST (back) to MD is fine for me.

Just English docs is not an option for me:

• There are a lot of people out there who have difficulties understanding English and I do not want to keep them apart from this great improvement of their diabetes management.
• Even if some translations are sub-optimal and in many languages quite some text is untranslated I would keep the different languages. The more the better -as long as they are maintained.
• Quite some people invested time for the different language versions which I would not devalue by removing those.
• @Andries-Smit just made a big improvement in docs structure with search only in selected language and more detailed navigation. I wouldn't want to loose this.

[Suchiman]

@xJoe

Btw @Suchiman filenames are English only at the moment. The problems are caused by links to headlines

... wha... oh... yeah... you're right, have i been halluzinating? I could have sworn trying to change the language in the url didn't work in the past and usually resulted in 404, which meant i had to open the docs in the other language and then navigate the complex nested navigation menu (or was this an issue with folderpaths instead of filenames? nevermind the anchors) and the new (it's new, right? or is my memory that crap) flatter navigation is much more easy to navigate, very nice!

[xJoe]

@Suchiman, Andries-Smit's changes made a great improvement. Before those it was tricky to move from EN to another language. If you wanted to move from one non-EN language to another non-EN language this was quite easy - always asuming there were no anchor-links to headlines.

[MarieT1D]

Hello and thanks for all your work. Just if your are interested, the reason why we (at least I) were doing most in .rst and I think we might have even translated from .md to .rst when restructuring the docs 1,5 years ago, was because there were some more features and .rst is the default plaintext markup language used by Sphinx (https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html). I think before that, there were quite a lot .md files in the docs. We were trying some things with the structure and I can't remember if there were other reasons for using .rst, it feels too long ago^. But concering Crowdin, it seems like it is a very good solution to move back to .md.

So thanks a lot for your effort

[Philoul]

I prefer md on my side, I use a WYSIWYD text editor for md files (Typora) on my local fork, so it's easier to work on it

My two main difficulties for consitency of translations are 1) anchors (I often check directly with my browser, if target page is already published. Sometimes the "correct translation" of anchor doesn't work and you have to replace it by #id1, and some other times, rules for anchor translation could be complicated when you have specials characters like - ( , ' é à ... 2) consistency between AAPS translation and AndroidAPSdocs translation (I often open 2 tabs, one on Crowdin page for doc, and another Crowdin page for AAPS to verify translation used in the app. When I have a label or acronym or button's name that is in english language of AAPS app, I have to find correct translation in my AAPS app on my phone, or in Crowdin for modules I don't use). It's more difficult for Omnipod doc page, because translations could be in the String file of Main app, Omnipod or in Rileylink module...

[Philoul]

@xJoe @Andries-Smit How we go further to improve the doc concerning format (md/rst) and links (anchors translated/untranslated)?

With v3 we will have to work again on doc to update or create new pages (new page for dash, treatments updated, user entries...), maybe we can start to use new rules for links creation and translation and file format.

Concerning md or rst, do you have decided which format to use ? (diaconn page created 4 monthes ago use rst, but I understood md was easier and better for doc...)

[tanja3981]

I converted the troubleshooting section to md now, mainly because I like MD much more and was reworking it anyway.

[peterleimbach]

I tested the replacement of english implicit anchors with explicit anchors.

It worked for me in this way.

(incorrect-use-of-automation)=

Incorrect use of automation

I wrote a small perl script for this..

Open is for me the questions how Crowdin relates to this.

[Philoul]

Is it better to create explicit links identical than implicit one, or just generate an explicit link with simple rule (link-1, link-2...) ? I'm afraid of confusion for translators if some links are implicit and others explicit with same syntax... (and when english text is changed, explicit links must remain unchanged).

[peterleimbach]

At the moment I see no chance to disable the automatic links for the headings. It's possible to say only level 1 in the configuration for the myst-parser but that's not working for me. I get always generated links for the section headers.

I can write an explicit anchor in front of the same section. (We have to check in Crowdin that this link is really transparent for the translator as we believe as it is markdown.)

I would prefer at the moment if a) the english version stes the anchros and references => only a link warning free version should then be a base for Crowdin b) the translators and proffreaders don't change the references or anchors at all as they don't see them.

I would prefer the naming as it is at the moment as you have to copy the links anyway from my point of view but I am open for another schema as section-1 or something is for sure better to write but it must be consistent and not changing in the future as this will corrupt the formally correct refences.

At the moment there is from my point of view a bug in the actual version of the myst_parser which results in writing the references with .md instead of without.

[peterleimbach]

I have a first local (sphinx with myst_parser) english version with only md files and a conversion script for one language (tested with english) but a) index.rst conversion with pandoc did not work. I made it manually and I think we have to do this for the relevant languages too. b) 3 rst style notes have to be migrated manually too. c) There are a lot link errors which come from the myst_parser problem but can be change with sed d) There are other link errors too which we have to correct manually because they are broken for which reason ever.

But I think the format conversion is no big issure. The issue is the Crowdin part where I wait until Andries is back.

And I think we have to change the version for sphinx, install / use the myst_parser, test that the scripting for the badges works etc. I fear that's will consume some more time.

[peterleimbach]

I made a local multilanguage sphinx site with the concept from https://www.sphinx-doc.org/en/master/usage/advanced/intl.html.

Then I followed @Andries-Smit idea of a machine translation and used the open source translation engine argos in a local installation for translating the po files which have been generated by Sphinx in the step before https://github.com/argosopentech/argos-translate.

To get and write the information from the po files I used https://polib.readthedocs.io/en/latest/quickstart.html in a short buggy python program. (I am no programmer at all but for the prototype it was enough. ;-))

For the small prototype the concept worked well.

Next step is to use the whole content from the english version of AndroidAPSDocs a) to see that it works with this larger content too and b) that the quality of the translation is good enough from our point of view.

[peterleimbach]

The idea with creating a multilingual version with po files did unfortunately not work.

a) I was able to create the .pot file and then the .po files but in the pot files Sphinx is not able in some cases to seperate content from markup which results in unpredictable side effects. I tried with md and rst files. Just to be sure that Sphinx is not more on the rst side. Sometimes links or paths to images have been translated. b) the quality of the machine translation was from my point of view not good. Maybe this could be solved with another translation service but a) is from my point of view a showstopper. The translation engine is super easy to install if someone wants to make his own tests - really cool.

I suggest we make a discord telco to discuss the possibilites we have to solve the problem with the translations. I will contact you there.

[peterleimbach]

We will change all doc pages in the medium term from reStructuredText to Markdown format bit by bit.

1. We discussed this here and from my point of view all voted for markdown and no one opted out to do this
2. Therfore I updated the existing documentation to inform the readers
3. I will start with migrating the english version in a local repo (format change, removing broken links, adding unique subscetion header links) and give this for quality check to @Andries-Smit in a "local/private" RTD environment
4. Then I will do the same with the German version where we can see if side effects happen on Crowdin which I can change myself
5. Then I will do it for all the other languages too together with one of the respective translation teams.
6. It will take some time but after that we can concentrate on the main work - the content.
7. There might be side effects for the build badges but I think we can solve this forward.

I will close the issue now.