Use asciidoc for documentation

ciampix commented 3 years ago

This is a feature request. I am suggesting of writing all docs in a standard format, I suggest ASCIIDOC. Why am I asking this?

1) Because Markdown is not a (real) format. I mean that Markdown is a "family" of dialects of formats. Currently there is not a definition of what is Markdown standard, just a more or less common subset of definitions. When people write Markdown in Github they actually write in the "github flavor" of markdown. That is just "one" MD flavor of the many existing ...

2) Because Markdown lacks a lot of structures. MD was used exactly for this reason: simplicity at the cost of lack of features. No tables, no file include, no multilang, no intex, no toc, no sub rendering (latex, graphwiz, etc.) support, etc.

Instead Asciidoc for example is almost as complete as docbook, think of it as a simpler to use docbook alternative (see https://en.wikipedia.org/wiki/AsciiDoc). Asciidoc is a standard that is in this very moment in course of public standardization under the wing of the Eclipse foundation (See: https://www.eclipse.org/org/workinggroups/asciidoc-charter.php and https://projects.eclipse.org/proposals/asciidoc-language)

If this proposal will be accepted I can even do all the conversion work for you. Moreover, since github "understands" the asciidoc format quite well (it uses the same ruby libraries on which is based asciidoctor, the most advanced asciidoc compiler to date), after conversion one just have to save the doc files with the .adoc extension instead of .md and github will render them perfectly.

Let me know your (devs) decision

Marco Ciampa ciampix (a m p e r s a n d) posteo.net

kantord commented 3 years ago

A couple questions:

Would you even convert README.md into Asciidoc? Would that be technically possible?
I understand the issue with Markdown not being "one standard", however it has the benefit of being very widely known. Wouldn't using a not so well-known standard create a barrier for people?

kantord commented 3 years ago

Because Markdown lacks a lot of structures. MD was used exactly for this reason: simplicity at the cost of lack of features. No tables, no file include, no multilang, no intex, no toc, no sub rendering (latex, graphwiz, etc.) support, etc.

In most cases you can use tables in Markdown, as well as tables of content. You can also have latex and graph viz 🤔. They might need some extra setting up, and it's true that they are not standardized that much. Here's an example of a markdown-based documentation site. It's actually just a different theme for the documentation generator we're already using: https://squidfunk.github.io/mkdocs-material/

I think multilang is the key factor here. I don't know of a good way of creating a multilang documentation using just markdown that doesn't make you basically manually re-create all the changes in every language. How would this be done using asciidoc? Can you show me some examples of how this would be set up?

Another important factor might just be "approachability". I think it's important that the documentation site looks clean, simple and attractive. Do you have some examples of asciidoc-based documentation sites that we could imitate?

ciampix commented 3 years ago

A couple questions:

* Would you even convert README.md into Asciidoc? Would that be technically possible?

Yes ... that is always doable manually obviously, although you can use pandoc (and see what you get with the "github flavor option" ... https://pandoc.org/ ;-) that has an (very basic) asciidoc output filter (it does not "understand" asciidoc table for example...) that means that you can convert any input file you can throw to pandoc into an asciidoc document, but unfortunately pandoc lacks of an asciidoc input filter that would convert asciidoc into almost anything ... there is some ongoing work on it but it seems to be stagnant ...

I've successfully converted README.md into README.adoc (using pandoc) ... it need some manual work though on tables for example...

* I understand the issue with Markdown not being "one standard", however it has the benefit of
  being very widely known.

yes but only the md "standard" subset

Wouldn't using a not so well-known standard create a barrier for people?

if you stick to the "standard" md features, that unfortunately are very few and very basic, that's ok. See for example pandoc on "markdown" options:

man pandoc

[...]

   Markdown variants
   In addition to pandoc’s extended Markdown, the following Markdown variants are supported:

   • markdown_phpextra (PHP Markdown Extra)

   • markdown_github (deprecated GitHub-Flavored Markdown)

   • markdown_mmd (MultiMarkdown)

   • markdown_strict (Markdown.pl)

   • commonmark (CommonMark)

   • gfm (Github-Flavored Markdown)

   • commonmark_x (CommonMark with many pandoc extensions)

   To see which extensions are supported for a given format, and which are enabled by default, you can use the command

          pandoc --list-extensions=FORMAT

   where FORMAT is replaced with the name of the format.

When you start using more advanced features (like tables and such) you end up definitely using some "dialect" commands that restrict your audience on that specific "dialect". That dialect is not understood universally from all md parsers out there (in this case it is understood only by github and partially by pandoc for example) so people definitely will have trouble rendering those documents in the future. For an example on what does mean convert a markdown using something outside github, try using pandoc like this:

pandoc -i README.md -o README.html

That is not a "barrier" for now, but it will be in the future. You know, it is just the old discussion on "standard formats" going on here... ;-) https://xkcd.com/927/

ciampix commented 3 years ago

Because Markdown lacks a lot of structures. MD was used exactly for this reason: simplicity at the cost of lack of features. No tables, no file include, no multilang, no intex, no toc, no sub rendering (latex, graphwiz, etc.) support, etc.

In most cases you can use tables in Markdown, as well as tables of content.

mmm seems to me that md let you use html into md instead of handling tables "the markdown way", isn't it?

You can also have latex and graph viz thinking. They might need some extra setting up, and it's true that they are not standardized that much. Here's an example of a markdown-based documentation site. It's actually just a different theme for the documentation generator we're already using: https://squidfunk.github.io/mkdocs-material/

If you really want to use those pythonist tools I suggest Sphinx ... https://www.sphinx-doc.org/en/master/ It is another lightweight markup language (rest) but at least it is a standard (not official but "de facto") and a GREAT tool... I prefer Asciidoc though... my preferences order: asciidoc->rest->markdown ...

I think multilang is the key factor here. I don't know of a good way of creating a multilang documentation using just markdown that doesn't make you basically manually re-create all the changes in every language. How would this be done using asciidoc?

for the moment you have to use 2 tools: po4all (https://po4a.org/) and an asciidoc compiler (like asciidoc https://asciidoc.org/ or asciidoctor https://asciidoctor.org/)

You may ask yourself: "wait a minute: po4all is the solution! I can use it to manage multilanguage docs easily directly from markdown!". Well perhaps yes, I have to be honest, I did not try it... but there are some things that needs to be handled by the language converter. For example if you convert md into PDF or epub, the page, chapter, index, title descriptors must be handled by the markup language converter, and there is no "lang" option in any markdown converter that I know of... yes maybe I am a bit too meticulous... ;-)

Can you show me some examples of how this would be set up?

Sure: see KiCad docs https://docs.kicad.org/ in the upper right corner you can switch the language... The set up is actually a bit tricky, I am searching a way to make it better and simpler...

Another important factor might just be "approachability". I think it's important that the documentation site looks clean, simple and attractive. Do you have some examples of asciidoc-based documentation sites that we could imitate?

See the previous answer...

I copied the idea from the GIMP documentation. GIMP docs are docbook xml and use the xml2po tool to handle the multilanguage translation. The idea was good but I wanted to use some simpler document source format to help doc authors focus on doc contents instead of docbook arcane format. Asciidoc is as simple as Markdown but a lot more powerful and standard so I suggested it for the KiCad documentation project. Now it has become the KiCad document format.

PS: I am the translator of the GIMP and KiCad documentation in Italian.

kantord commented 3 years ago

mmm seems to me that md let you use html into md instead of handling tables "the markdown way", isn't it?

there are native table formats as well, here's how you do them in GitHub flavored markdown for example: https://github.github.com/gfm/#tables-extension-

The other "extra features" don't necessarily have to mean including html either, for example here's how you include mathjax in mkdocs material:

$$
\operatorname{ker} f=\{g\in G:f(g)=e_{H}\}{\mbox{.}}
$$

mkdocs also uses mermaid to render graphs: https://mermaid-js.github.io/mermaid/#/

and the syntax is also "native" Markdown:

``` mermaid
graph LR
  A[Start] --> B{Error?};
  B -->|Yes| C[Hmm...];
  C --> D[Debug];
  D --> B;
  B ---->|No| E[Yay!];

So it doesn't have to mean that you have to use weird tricks to support these things.

If you really want to use those pythonist tools I suggest Sphinx ... https://www.sphinx-doc.org/en/master/

I don't necessarily want to use Python tools at all! As long as the tool is easy to configure and is based on open source software I'm fine with anything. I personally found Sphinx much harder to configure than most other documentation tools, but that doesn't have to be a deal breaker.

I really like this Spinx theme by the way: https://sphinx-themes.org/sample-sites/furo/

if you convert md into PDF or epub, the page, chapter, index, title descriptors must be handled by the markup language converter

I think converting the documentation to epub or PDF is not an essential feature :thinking: btw I am thinking of publishing the courses themselves in PDF, and for that purpose I'm considering to use weasyprint: https://weasyprint.org/ I have used it in the past to create print material, and it's pretty amazing. I think it could be used with any html documentation system, even if said system doesn't natively support pdf

kantord commented 3 years ago

Well perhaps yes, I have to be honest, I did not try it...

I think it might be worth it to try it.

I think you have really good arguments and you have a lot more experience in this topic than I do. But I still think that there are perhaps more arguments to consider:

Simplicity is really important for this project. I think there are many ways we should be aiming to simplify things even more
People don't want to read documentation. Only the documentation that people will actually read will help them. Therefore it's more important to keep things simple and design a product that's easy to use. If the documentation system we use is not "batteries included" then it might up creating more need for documentation.
I want to reiterate that it's probably very important that documentation sites look clean, simple and modern. If it looks tedious, then only the most invested people will actually spend time reading the documentation

kantord commented 3 years ago

Another note is that long-term, I envision LibreLingo as more of a standard and an ecosystem of (often independent), small pieces of software and documentation than a monolith. As parts of the project mature, they can become independent of each other and be even moved to separate repos or maintained by different people. Or even new things invented by other people completely independent of me. So I think we don't want a large monolithic documentation either. Just information to keep in mind.

ciampix commented 3 years ago

mmm seems to me that md let you use html into md instead of handling tables "the markdown way", isn't it?

there are native table formats as well, here's how you do them in GitHub flavored markdown for example: https://github.github.com/gfm/#tables-extension-

I do not want to argue with you but the simple fact that a site like github was forced to create an unambiguous and not trivial, like the "reference", specification is the definitive proof that md is not a standard.

The other "extra features" don't necessarily have to mean including html either, for example here's how you include mathjax in mkdocs material:
$$
\operatorname{ker} f=\{g\in G:f(g)=e_{H}\}{\mbox{.}}
$$

also not standard

mkdocs also uses mermaid to render graphs: https://mermaid-js.github.io/mermaid/#/

and the syntax is also "native" Markdown:

what do you mean with "native"? Github? Ok is it github now an ISO/OSI chapter that can certificate a "standard"? See my point?

``` mermaid
graph LR
  A[Start] --> B{Error?};
  B -->|Yes| C[Hmm...];
  C --> D[Debug];
  D --> B;
  B ---->|No| E[Yay!];
So it doesn't have to mean that you have to use weird tricks to support these things.

Using the github "flavour" is a trick...

If you really want to use those pythonist tools I suggest Sphinx ... https://www.sphinx-doc.org/en/master/

I don't necessarily want to use Python tools at all! As long as the tool is easy to configure and is based on open source software I'm fine with anything. I personally found Sphinx much harder to configure than most other documentation tools, but that doesn't have to be a deal breaker.

I really like this Spinx theme by the way: https://sphinx-themes.org/sample-sites/furo/

if you convert md into PDF or epub, the page, chapter, index, title descriptors must be handled by the markup language converter

I think converting the documentation to epub or PDF is not an essential feature thinking btw I am thinking of publishing the courses themselves in PDF, and for that purpose I'm considering to use weasyprint: https://weasyprint.org/ I have used it in the past to create print material, and it's pretty amazing. I think it could be used with any html documentation system, even if said system doesn't natively support pdf

That is what I consider "a trick".

In general you never know what brings you the future. So it is always a good thing to start with the right foot even if one now does not see the reason, it could become evident later on...

ciampix commented 3 years ago

I even found that exists a handy markdown to asciidoc converter... I'll definitely try it out!

https://github.com/asciidoctor/kramdown-asciidoc

kantord commented 3 years ago

Nice!

kantord commented 3 years ago

So I've been looking into asciidoc things a little bit more and I can say that it indeed seems to be the case that it is easier to work with and somewhat more human friendly. It seems to have a couple more base features that make it easy to structure documentation sites.

I'm not that convinced that it's overall that much better than, say, mkdocs-material, but it seems like it's definitely not much worse either. From what I can see basically the only weakness of the asciidoc tooling is that the documentation sites don't look that modern/clean.

I personally won't have time in the near future to work on this, but I can help you set it up. Or we can try to find a 3rd person who will help. We can ask someone from the chat to volunteer for example

ciampix commented 3 years ago

Ok nice! I'm (with my very little programming experiences) at your disposal!

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ciampix commented 3 years ago

I'm leaving a comment just to keep this alive. I'm still at your disposal about Asciidoc ...

kantord commented 3 years ago

Since I have no experience with asciidoc or any of the related software, I'm not sure what to do. Do you know what would be the best first step?

Since you mentioned https://github.com/asciidoctor/kramdown-asciidoc, maybe the first step could be trying to create a branch where the documentation is converted using this tool, and see how good the result is?

kantord commented 3 years ago

Do you know about this, btw? https://antora.org/

ciampix commented 3 years ago

Since I have no experience with asciidoc or any of the related software, I'm not sure what to do. Do you know what would be the best first step?

Since you mentioned https://github.com/asciidoctor/kramdown-asciidoc, maybe the first step could be trying to create a branch where the documentation is converted using this tool, and see how good the result is?

I've trying it right now ... it seems to me that it's working great! This evening I'll try it better ... when done, if it's good enough, what do we do next?

ciampix commented 3 years ago

Ho! I tested kramdown and it worked wonderfully for all docs! What do I do now? I can convert all md files into adoc in one time. As a first step I can make a merge request if you agree...

kantord commented 3 years ago

As a first step I can make a merge request if you agree...

sure, but I'd prefer to only merge it once we also know how to render and publish it

ciampix commented 3 years ago

Well github already supports asciidoc natively like md, so that it should change nothing. I mean they would be rendered on the fly by the github site. But at the same time you will be able to render them simply with a asciidoctor filename.adoc command.

kantord commented 3 years ago

Well github already supports asciidoc natively like md, so that it should change nothing. I mean they would be rendered on the fly by the github site. But at the same time you will be able to render them simply with a asciidoctor filename.adoc command.

yes, they would be rendered correctly on the GitHub page, but they would be completely broken on the website: https://librelingo.app/docs/

ciampix commented 3 years ago

Oh! How are those pages build? And why are we discussing this here instead of on a chat like Matrix for example? ;-)

kantord commented 3 years ago

I think it makes sense to discuss it here, for posterity, no? 🤔

The documentation pages are being build using MkDocs from the beginning, but unfortunately it does not support AsciiDoc :disappointed: and couldn't find a plugin for it either. Meaning we have to install a replacement for it, that's why I was looking at antora for example https://github.com/kantord/LibreLingo/issues/1013#issuecomment-891152405

ciampix commented 3 years ago

"MkDocs is a fast, simple... static site generator" ah ok like for example HUGO https://gohugo.io/ that does support asciidoc....

ciampix commented 3 years ago

https://www.cosmos547.moe/posts/hugo-with-asciidoc/

kantord commented 3 years ago

One of the issues I see with Hugo is that

It requires the hugo binary to be installed
It requires asciidoctor to be installed
Which in turn requires Ruby to be installed

So it feels like that would create a lot more extra hurdles, basically it would make setting up a local dev environment 2-3 times harder than it is now

kantord commented 3 years ago

Antora looks promising in those terms because it looks like it's node-based. Not that I have anything against go or ruby per se, but it feels a lot more convenient: https://docs.antora.org/antora/2.3/install-and-run-quickstart/

If I understand correctly (after quickly skimming through the guide) Antora could be installed through yarn so it would work out of the box for anyone who successfully set up their dev environment already. Pretty much zero extra hurdles 🤔

ciampix commented 3 years ago

Yes you're right. Lets give it a go then!

kantord commented 3 years ago

@ciampix I think the way to go is

you create the PR with the converted files
based on this PR (without merging it) we/I can try to run antora and see if it works
then decide how to actually replace mkdocs with this (I can maybe push code to your branch)
if everything works out and we like the result, merge and close this issue

ciampix commented 3 years ago

Yes you're right. Lets give it a go then! I made the PR https://github.com/kantord/LibreLingo/pull/1517 README.md was not possibile to be converted automatically. I'll try to do it manually later on...

kantord commented 3 years ago

Thanks @ciampix. See my comment at your MR.

README.md was not possibile to be converted automatically. I'll try to do it manually later on...

let's leave this one for now. I think it might be impossible to do it, because the allcontributors bot might not be compatible with asciidoc 🤔 in either case it's not the "real" docs, mostly serves as a starting point of the repository page

ciampix commented 3 years ago

Thank you! I am keeping my branch on par with main branch but still I do not know how to add mermaid support to Antora that seems to be possible somehow ...

ciampix commented 3 years ago

See asciidoctor-kroki supports them: https://github.com/Mogztter/asciidoctor-kroki#supported-diagram-types and it could be integrated into Antora: https://awesomeopensource.com/project/Mogztter/asciidoctor-kroki#antora-integration but since I do not know how to do it properly I'm stuck. Here what I did:

cd apps/docs & added

asciidoc:
  extensions:
    - asciidoctor-kroki

to antora-playbook.yml then did:

yarn add asciidoctor-kroki

I'm curious: what am I missing? (Please be patient and consider that I know NOTHING about JS...)

kantord commented 3 years ago

Thank you! I am keeping my branch on par with main branch but still I do not know how to add mermaid support to Antora that seems to be possible somehow ...

are there any viable alternatives to mermaid that might be supported easily by antora? I'm not hard set on mermaid, as long as we have something nice for the diagrams

kantord commented 3 years ago

See asciidoctor-kroki supports them: https://github.com/Mogztter/asciidoctor-kroki#supported-diagram-types and it could be integrated into Antora: https://awesomeopensource.com/project/Mogztter/asciidoctor-kroki#antora-integration but since I do not know how to do it properly I'm stuck. Here what I did:

cd apps/docs & added
asciidoc:
  extensions:
    - asciidoctor-kroki
to antora-playbook.yml then did:
yarn add asciidoc-kroki
I'm curious: what am I missing? (Please be patient and consider that I know NOTHING about JS...)

are you getting any errors? are you sure that the syntax used for the graph is correct? I think in the kroki examples, the syntax looks a bit different

ciampix commented 3 years ago

It was trivial. Just delete the comma in the [,mermaid] header. As I wrote elsewhere unfortunately there is a bug in the asciidoctor-kroki library that prevents the functioning of the image links (that still work if you load the single image in another browser window). Anyway it works. :-)

kantord / LibreLingo

Use asciidoc for documentation #1013