drop .md pages in our own documentation

[12rambau]

My request is actually the reverse of https://github.com/pydata/pydata-sphinx-theme/issues/639

The conclusion of https://github.com/pydata/pydata-sphinx-theme/issues/639 was "come as you are". I was liking it at the time but I'm not liking it anymore. My rational is that we are a Sphinx based theme, our goal is to extend as much as we can what Sphinx does and make it better, not promoting new tool (even when they are great as Jupyterbooks or myst). Why ? because then not only we are making ourselves bound to their releases processes but more importantly we are forcing users to not only learn sphinx and rst but also learn all this extra-lib that they may not know/need.

myst in itself is parsing .md files to .rst before sending them to the Sphinx builder. It's great and it will work without any changes from our side so why not stick to the .rst way ?

I know that many of you are big users of jupyterBooks so you are more used to myst than I am and you feel it's easier. I still think that there are more people like me than like you as this theme can be used for things that are 100% unrelated to Jupyter. Even though I understand that for simple text documentation it's just fine and I agree that .rst can be a bit convoluted sometimes but whenever you try to do more complicated things it becomes:

• messy: How can one consider these tabs in .md easier to read than an indented .rst directive https://github.com/pydata/pydata-sphinx-theme/blob/main/docs/user_guide/theme-elements.md ?
• difficult: it's forcing people to understand .rst (where the directives are actually documented) and the .rst parsing performed by myst as myst didn't redocumented docutils entirely as this one: https://docutils.sourceforge.io/docs/ref/rst/directives.html#csv-table
• non reusable: This custom directive can only be used in a .md file and not in a pure sphinx based one: https://github.com/pydata/pydata-sphinx-theme/blob/main/docs/_extension/gallery_directive.py, I think it's a mistake.

Note that I would happily do as in sphinx-design where they decided to promote myst by showing for every commands the 2 writting options: https://sphinx-design.readthedocs.io/en/latest/grids.html but my suggestion here is to harmonize the writting of our docs to allow new users to participate without enforcing a new markup language.

[drammock]

not promoting new tool [...] because then not only we are making ourselves bound to their releases processes but more importantly we are forcing users to not only learn sphinx and rst but also learn all this extra-lib that they may not know/need.

The extra dependency / release cycle is indeed a big pain point for me. The latter point I think is not quite right --- it's not users who need to learn MyST in order to use PST. Rather it's contributors who need to learn it in order to contribute. Still a problem though.

I'll skip the points about MyST being "messy" and "difficult" (I don't disagree, but I can see how someone might). But the "non-reusable" point just bit me last week; I strongly agree that a Sphinx theme that supports rST should not have a custom directive that only works for MyST source files. If we're going to support both rST and MyST, any custom stuff we add to Sphinx should ideally support both rST and MyST, but if it only supports one it should be rST.

Just to explicitly separate the issues:

1. do we change gallery_directive.py to work with rST? (it's in our public API, so we need to either deprecate or maintain backward compatibility)
2. do we keep allowing our own docs source files to be in MyST? (if not, we can eliminate some build dependencies and thus simplify maintainer's job)
3. do we keep providing built-in support for MyST source files in users' docs? (if not, we need to do a deprecation cycle and provide some change management / migration guidance for what extra steps they will need afterward)

My vote is:

• on (1) we should either make the directive configurable (output="myst" or "rst" perhaps) or else provide two directives (GalleryGridMyst and GalleryGrid)
• on (2) I am in favor of all our docs being in rST format
• on (3) I'm not sure but I lean toward simplifying what we provide, mostly because we are very short on maintainer time so any simplifications that save us time seem good to me.

[dbitouze]

My 2 cents as a user.

Would PyData Theme be missing the MyST equivalent syntax of RST one, I probably wouldn't give it a try: too much a pain to translate from one to another (even with the help on Pandoc).

The site I'm working on (a French LaTeX FAQ) will use only (MyST) Markdown: RST is IMHO a pain, isn't well know (especially by the future contributors of the FAQ) and most of the generic text editors:

• have a (good) Markdown mode,
• don't have a (good) RST mode.

[AA-Turner]

it's in our public API, so we need to either deprecate or maintain backward compatibility

The current gallery_directive only seems to by in p-d-t's own documentation, so I think you'd be free to change to RST right away.

It seems possible to implement the directive using nodes, or to auto-detect the containing file's type and adjust, but I agree with you that simplification to reST is probably the best approach.

A

[12rambau]

@dbitouze we will remain compatible with myst anyway as it's made to parse .md to .rst. It's just that it will be up to the users to decide. Note that .rst is perfectly well supported in all the text editor I use (visual studio, jupyterlab, github itself). The main difference being that Sphinx add rich directives on top of the syntax.

I checked your documentation and you are not using any of the advanced directive structure of myst so I guess that's why you don't see my point on complexity. myst is simple until you start doing complex stuff.

[12rambau]

@AA-Turner I would prefer to go down the node approach, I'll check the sphinx-design documentation and API to see how they implemented grids.

[AA-Turner]

At a glance, you'd probably need to instantiate a sphinx_design.grids.GridDirective and then call .run() on that instance.

[dbitouze]

@dbitouze we will remain compatible with myst anyway as it's made to parse .md to .rst. It's just that it will be up to the users to decide.

Do you mean the user will decide to see only .rst or only .md codes in the documentation?

Note that .rst is perfectly well supported in all the text editor I use (visual studio, jupyterlab, github itself).

I'm using Emacs :wink:, which has a (not very powerful) RST mode. And, again, I guess future contributors of our LaTeX FAQ will probably be much more accustomed to Markdown than to RST (because of GitHub, GitLab, StackExchange, StackOverflow, etc. which use Markdown for the posts, issues reports, etc.)

The main difference being that Sphinx add rich directives on top of the syntax.

Sorry, I don't what you mean.

I checked your documentation and you are not using any of the advanced directive structure of myst so I guess that's why you don't see my point on complexity. myst is simple until you start doing complex stuff.

Which advanced directive structure of myst do you mean?

[12rambau]

@dbitouze I think you missed the point of this thread. The documentation you are writing is using the myst extension so you can continue to use it the exact way you were previously and if any changes are made related to this issue they will have 0 influence on how you write you .md files. Here the conversation is about the theme own documentation that is currently build with a mix of both file type.

Sorry, I don't what you mean.

With myst you can integrate any directives supported in your doc (Sphinx or any other extension) with the "``" syntax. making it IMO as complicated as .rst without the possibility to use indentation. If you still don't see what I'm talking about then you are the perfect target for themyst` extention so continue to use it !

[dbitouze]

Here the conversation is about the theme own documentation that is currently build with a mix of both file type.

OK, sorry for the noise, then. But, just to be sure, will the theme documentation still expose both .rst and .md syntaxes, e.g. on this page?

Sorry, I don't what you mean.

With myst you can integrate any directives supported in your doc (Sphinx or any other extension) with the "```" syntax. making it IMO as complicated as .rst without the possibility to use indentation.

At least the following does work:

Foo
: bar:
  - baz

  ```
  Test
  ```

but...

If you still don't see what I'm talking about then you are the perfect target for the myst extention so continue to use it !

... maybe, indeed, I completely missed the point of this thread. In such a case, sorry again for the noise.

[drammock]

The current gallery_directive only seems to by in p-d-t's own documentation, so I think you'd be free to change to RST right away.

ah, you're correct. That simplifies things. So if we switch all our doc source files to rST then we can (must!) also convert that directive at the same time.

[trallard]

Wearing a maintainer hat: I am ok with either all of our PST (Pydata Sohunx Theme) docs being on rst or md based - whatever makes the team's life easier is good for me. If this means we drop internal (i.e. not user-facing) dependencies, and this makes the maintenance of the theme easier, I am +1.

As a PST user and reinforcing the points made above by other users, it is important that we keep providing support for end-users to write in whichever markup language suits them best, md or rst. While I have been using rst for many years, I do have to recognise that the md is more intuitive (this is my own perception because despite having used rst for many years, I need to keep revisiting the cheatsheet all the time). Newer contributors seem to be more familiar with md, so the contribution barrier from this POV is lower.

Note that I would happily do as in sphinx-design where they decided to promote myst by showing for every commands the 2 writting options: sphinx-design.readthedocs.io/en/latest/grids.html but my suggestion here is to harmonize the writing of our docs to allow new users to participate without enforcing a new markup language.

I thought this was a point we had agreed on before. All our examples show both syntax examples. There is #1366 but I am unsure if that added both syntax possibilities for all the examples/components, so we might need to revisit this.

Now onto this point:

But the "non-reusable" point just bit me last week; I strongly agree that a Sphinx theme that supports rST should not have a custom directive that only works for MyST source files. If we're going to support both rST and MyST, any custom stuff we add to Sphinx should ideally support both rST and MyST, but if it only supports one it should be rST.

If we indeed have any custom directives that are myst only, then it makes sense that whatever we have in PST supports rst too. In hindsight, the custom directive was the gallery_directive in which case it seems that the action path would be to migrate our own docs to rst and then adjust this as we work on this. Sounds accurate?

[drammock]

I think you nailed it @trallard. I do think that having all of PST doc files in .rst will make maintainers lives easier (on balance), and I agree that we must continue to support .md source files for users.

the custom directive was the gallery_directive in which case it seems that the action path would be to migrate our own docs to rst and then adjust this as we work on this

Agreed, this seems like the right approach (given that the custom directive was internal to our docs, not something users could have been using in their own sites --- we are free to change it at will).

[choldgraf]

For what it's worth, I definitely don't like writing rst, but if others are aligned on the need to remove MyST from the pst docs, I won't object.