Brainstorming on MyST docstrings via sphinx.ext.autodoc

[consideRatio]

When converting rST projects to use MyST, directives from sphinx.ext.autodoc's such as automodule can cause issues.

The sphinx extension sphinx.ext.autodoc provides directives (automodule, autoclass, ...) that emits rST like .. py:module:: kubespawner.utils and provides the directives with raw docstrings from the referenced Python code.

Background

Here is an example of how you would use sphinx.ext.autodoc from a MyST document correctly.

# Utilities

```{eval-rst}
.. automodule:: kubespawner.utils

The automodule directive will emit rST directives and raw docstrings that also ends up parsed as rST.

## Two issues

Here is an example using the `automodule` directly as a MyST directive, and it leads to two issues:

````markdown
# Utilities

```{automodule} kubespawner.utils

![image](https://user-images.githubusercontent.com/3837114/205635415-b2168643-d50d-437c-bba4-f6e519d201c6.png)

### Issue 1 - sphinx.ext.autodoc emits rST formatted directives

sphinx.ext.autodoc will emit rST formatted directives even though invoked from a MyST document like:

````markdown
# Utilities

```{automodule} kubespawner.utils

### Issue 2 - sphinx.ext.autodoc has made us write docstrings with rST

A project who has used sphinx.ext.autodoc has needed to write the docstrings in rST rather than MyST historically due to always being parsed with rST.

If we are to support MyST docstrings, I think of one initial key goal, and a stretch goal:

- __Key goal__: to support a project wide default declaring docstrings to be either rST or MyST (100% rST docstrings or 100% MyST docstrings).
- __Stretch goal__: to support exceptions from the default in some way, using both rST and MyST docstrings.

I see the key goal as a very valuable goal resolving my its own, and the stretch goal as valuable but also likely more complicated to implement and use sustainably.

## Ideas to address this

I'm not sure if and how to address these issues. I've opened this issue to brainstorm if we can find reasonable resolutions to these issues - especially if we can find a way to support 100% MyST based docstrings via sphinx.ext.autodoc. A solution that is too complicated and hard to maintain isn't worth developing in my mind.

### Support directly in sphinx.ext.autodoc?

Maybe sphinx.ext.autodoc could add support for this directly?

- sphix.ext.autodoc should somehow decide if it should emit rST or MyST directives, either by configuration or automatic detection.
- sphinx.ext.autodoc could optionally also wrap the raw docstring in a directive like eval-rst based on configuraition

### Support by wrapping or patching sphinx.ext.autodoc in another sphinx extension?

Maybe a sphinx extension could be created (myst-autodoc), which would patch or influence sphinx.ext.autodoc somehow. Maybe by wrapping/redefining the provided directives and running rst2myst on the old definition, or maybe patching some functions in sphinx.ext.autodoc?

### Your ideas?

I opened this issue hoping to overview the situation and find actionable paths towards resolving this in a good way that is sutainable to maintain and use.

[welcome[bot]]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:

[chrisjsewell]

Heya, I'd say this is a duplicate of https://github.com/executablebooks/MyST-Parser/issues/228

[chrisjsewell]

Basically its made non-trivial by sphinx hard coding rST into the autodoc logic: https://github.com/sphinx-doc/sphinx/issues/8018

[chrisjsewell]

Basically we either need to either "motivate" the sphinx maintainers to change (or allow a change) to their internal autodoc code. Or find the the "least invasive" (most maintainable) way to override it

[chrisjsewell]

Definitely like to see it happen though

[consideRatio]

@chrisjsewell thanks for helping link things together!

Applying rst2myst on myst or a mix of myst/rst isn't good

I considered if rst2myst could be applied to MyST without making changes if its actually ending up parsing myst content. If that would be the case, it could be called on anything autodoc emits. I don't think that is possible though, and would at best lead to only few inconsistencies.

Currently, running rst2myst on myst content would lead to something like this.

-1. Make sure you have successfully completed {ref}`contributing/setup`.
+1. Make sure you have successfully completed \{ref}\`contributing/setup\`.

So whatever resolution we come up with, we must avoid ending up with a mix of myst and rST I think. Well, unless the rST is really MYST, wrapped in an eval-rst directive, which makes it really 100% MyST still.

Due to that, I'm thinking you are correct to say that whats needed is a removal of the rST assumptions used in sphinx.ext.autodoc.

[consideRatio]

Current understanding of the challenge

sphinx.ext.autodoc doesn't do parsing itself, but constructs rST text involving raw docstrings. This will create a mix of rST and MyST if MyST docstrings are used. Whenever a mix of rST and MyST is created, we can't reasonably disentangle things, so the crux is to avoid that from happening.

To avoid creating a mix of MyST and rST, we can:

• Transform MyST docstrings into rST before sphinx.ext.autodoc embeds it into its rST output
  • Is there a myst2rst converter we can use?
  • If so, we can use the autodoc-process-docstring sphinx event for which an event handler can be registered to convert the docstring from myst2rst so it can be parsed by rst along with the rest of the rst emitted by sphinx.ext.autodoc.
• Add support for sphinx.ext.autodoc to construct MyST + raw docstrings
• (?) Add support for sphinx.ext.autodoc to not construct MyST or rST, but parser generic docutils output or similar.

[chrisjsewell]

@consideRatio want to give this a go https://sphinx-autodoc2.readthedocs.io/en/latest/quickstart.html#using-markdown-myst-docstrings 😄