Possible conversion of the nixpkgs manual from docbook to mdbook #156309

Open Sciencentistguy opened 2 years ago

mdBook is a manual generator that generates paged manuals. It is what is used by the nix manual, and a large number of books in the Rust community.

As opposed to the nixpkgs manual, which is one monolithic page, mdBook generates paged documentation which is much easier to read.

Is converting the nixpkgs documentation to mdBook desirable at this time? I certainly think that it would make the documentation more approachable and easier for beginners and advanced users alike to find what they're looking for.

I would be happy to have a look at doing this myself if people think it'd be like to get merged.

Prior art: https://github.com/NixOS/nixpkgs/pull/105036

Any consensus on documentation generation tool for nixpkgs manual?

yes please

see also

one monolithic page

this is not usable on "slow" hardware. it makes sense for print, but not for screen

workarounds

challenges

links
- https://github.com/NixOS/nixpkgs/issues/127813
syntax highlighting
- client-side with highlight.js https://github.com/NixOS/rfcs/pull/72#discussion_r461388486
- ideally done on the server (SSR) to produce plain html without javascript (less work for clients, pages load faster)
(more challenges?)

possible solutions

mdbook in Rust, also used for the Nix manual https://github.com/NixOS/rfcs/pull/72#issuecomment-663003944* markdown compiler in C https://github.com/NixOS/nixpkgs/issues/127813#issuecomment-866383957

Some CSS that makes the monolithic file's injected table of contents more accessible is all that would be needed. I personally find it easier to search and navigate as is--especially from a TUI browser.

However this option is so far from perfect...

As mdBook adheres to CommonMark there is so many features missing for robust documentation:

admonitions (support in Nix's docs currently through one of the CommonMark RFCs, but not mdBook)
attribution for blockquotes
figures for diagrams
easy ability to add attributes to elements (such as <img src width height>) which leads to writing inline HTML which loses portability to non-HTML targets (such as manpages)
no ability to offset headings when doing imports (#include custom to mdBook) leading to <h1> followed by <h1> or needing to care about the heading level in the separate document negating it's usefulness as a separate document
document metadata requires a nonstandard sidecar TOML file (author, license, copyright, title, keywords, etc.) instead of it being embedded into the document

Most of these issue are or could be fixed up by the tool and pulling in custom preprocessors, but now the documentation is not be nonportable and tied to a specific tool instead of just following a specification that provides the ability to do rich documents--resulting in writers needing to know about the quirks of Nix's documentation. Documents can't even be rendered properly on this Git forge:

See the {#id} syntax currently in any random document
See the CommonMark option for a {.note} admonition (example).

mdBook's docs here have issues:

overuse of bold assuming because they don't have access to the admonition RFC (one of which breaks the HTML semantics for a <blockquote> which is reserved for quoting text)
syntax highlighting requires JavaScript (supplied highlight.js doesn't even support Nix)
using <ol> instead of sections numbers or CSS counter

I don't want to see documentation quality go down and littered with tool-specific hacks due to limitations and fragmentation of conflicting Markdown specification. If anything, I'd like to see documentation move away from Markdown to something that could support richer experience for readers like AsciiDoc, reStructuredText, et.al. that are portable and can render fine an any code forge or through any other tool trying to consume the document.

portable and can render fine an any code forge

markdown and html

attrsets.md (github) (gitlab) (gitea) (srht)

formats like asciidoc, org-mode, docbook, ... are too esoteric and they all have their quirks and limitations on the github blob API, for example, absolute paths are broken in asciidoc and org-mode

Your examples show {} tags for I guess Pandoc to interpret which breaks the GitHub rendering specifically. Because of this, admonitions aren't using the CommonMark RFC syntax and instead are using blocks like :::{.note} vs. :::note.

I disagree that those syntaxes are esoteric though--they provide very useful features for writing articles and documentation that Markdown alone can't provide. If they are a better tool for the job, then having a to learn one of these non-hard languages with a spec that supports all needed features is better and the users will have learned, say reStructuredText, instead of yet another Nixpkgs-flavored Markdown.

I also just realized that by default, mdbook is reliant on highlight.js for syntax highlighting. This has some issues:

users with JavaScript disable for security reasons aren't supported
hundreds of thousands of users are going to need to waste CPU+battery, bandwidth, and rendering time to parse and do the syntax highlighting--the same parsing the previous user did--which is wasteful on big scale.

If this happens, I would at least expect syntax highlighting to be done at build time.

I disagree that those syntaxes are esoteric though--they provide very useful features for writing articles and documentation that Markdown alone can't provide. If they are a better tool for the job, then having a to learn one of these non-hard languages with a spec that supports all needed features is better and the users will have learned, say reStructuredText, instead of yet another Nixpkgs-flavored Markdown.

This has already been settled by RFC 0072 so there is really no point in re-opening the format conversation. Markdown with custom extensions was chosen as it allows people to achieve 90% using a familiar syntax. All other languages are at least an order of magnitude less often used and thus present a barrier to entry for contributing to documentation.

The comparative rarity of those languages also means that user will need to learn them not once but multiple times. Users typically do not write documentation very often so their brain will undoubtedly end up garbage collecting the jumble of random punctuation. Here at least DocBook had the advantage of a trivial syntax with somewhat memorable tags, that could be auto-completed by editor when not sure.

But again, that part of discussion is over and we should now focus on choosing the toolchain.

I also just realized that by default, mdbook is reliant on highlight.js for syntax highlighting. This has some issues:
1. users with JavaScript disable for security reasons aren't supported

I would argue that is pretty insignificant, as far issues with it go. Slightly more annoyingly, the search functionality is reliant on JavaScript as well. But even that can be worked around.

For large manuals the lack of proper handling of references is pretty noticeable.

And, of course, since we agreed on extending CommonMark, the poor extensibility support is annoying as well.

2. hundreds of thousands of users are going to need to waste CPU+battery, bandwidth, and rendering time to parse and do the syntax highlighting--the same parsing the previous user did--which is wasteful on big scale.

That will probably by negligible compared to the use of resources by Nix itself. And do not forget the opportunity cost of spending developer time on something that is in many eyes “good enough”. The harsh reality is that build-time syntax highlighting will probably only be implemented if someone who cares does that.

See https://github.com/NixOS/nixpkgs/pull/108063 for a multi-page manual

NixOS / nixpkgs

Possible conversion of the nixpkgs manual from docbook to mdbook #156309