Python documentation tool support: ReStructured Text language/Sphinx/Etc

[chrisjrn]

Pants is currently of limited utility for users who want to publish Python code documentation using standard Python tools, such as Sphinx, and ReadTheDocs.

This ticket is a placeholder for scoping out support for:

• ReST language support
• Building docs with Sphinx under Pants
• Publishing to ReadTheDocs.org/.com
• Migrating the Pants documentation to ReadTheDocs, by Sphinx or some other tool

[kaos]

Just throwing https://www.mkdocs.org/ in as a means to help frame the design to be flexible enough to take different tooling into account, as appropriate.

[cognifloyd]

Duplicate of #18288, but this one has more details. So, I'm closing that one.

[cognifloyd]

I've also heard of MyST which wraps Sphinx to use Markdown instead of ReST while still benefitting from all the other features of sphinx (directives, links between doc pages, intersphinx links, ...).

[cognifloyd]

Other tools I need to add to pants at some point:

• rstcheck: https://github.com/rstcheck/rstcheck
• readme_renderer: https://github.com/pypa/readme_renderer
• rst-lint: https://github.com/twolfson/restructuredtext-lint

(The StackStorm project needs these, but I haven't had time to work on plugins yet.)

[cognifloyd]

Sphinx support was added and then removed due to flaky tests:

• https://github.com/pantsbuild/pants/pull/15512
• https://github.com/pantsbuild/pants/pull/15758

I think the approach in that PR was almost right. One central sphinx_project target is a good UX for users, and should be simple enough for tailor. The implementation there, however, was a little over-simplistic as all of the rst source files were owned by the sphinx_project target, which would have made linting with other tools more difficult.

Instead, I think the sphinx_project should be a target generator that:

• generates rst_source targets for rst sources in sub directories that are not already owned by another target
• infers a dependency on any manually created rst_sources/rst_source targets
• also generates a python_source target for conf.py
• possibly also generate resource (or file?) targets for images referenced by the sources, or at least the rst_source should get an inferred dep on the resource that owns those images. Same for css, js, or other files.

The key insight is that target generators can generate more than one type of target.

[cburroughs]

I'm not sure if this is implicit in "Python code documentation", but it would be spiffy if this eventually included ironing out any rough edges to generating pretty pages from python docstrings, such as with https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html or https://mkdocstrings.github.io/.