Generate and publish API reference pages

[ScottTodd]

We have enough user-facing APIs to add a dedicated Reference section on our user website.

Python bindings

Most of our Python classes and functions have docstrings, but the files are scattered throughout our codebase and split across several module namespaces and pip packages.

We could use mkdocstrings with our new mkdocs site:

• https://mkdocstrings.github.io/
• https://chrieke.medium.com/documenting-a-python-package-with-code-reference-via-mkdocs-material-b4a45197f95b

https://wiki.python.org/moin/DocumentationTools has more options. Doxygen could be nice, as it supports all the languages we use.

TFLite bindings

TFLite / TFLite Java bindings can probably just link to TFLite's docs for API references, with differences highlighted on our pages.

C API

The API headers are well documented (example: iree/base/api.h) but the documentation is split across many files (example: iree/vm/api.h just exports various headers from iree/vm/). Aggregating into reference pages like https://mlir.llvm.org/docs/LangRef/ or https://mlir.llvm.org/doxygen/namespacemlir.html could be helpful.

Java bindings

Our Java bindings are experimental and incomplete at the moment. If we develop them into a larger, supported API, we could use Javadoc or Doxygen.

MLIR Dialects

We generate documentation for our MLIR dialects and publish them today at https://google.github.io/iree/Dialects. These are not user-facing (for most, but not all, definitions of "IREE user"), but it's useful to discuss them here too. Tablegen doc generation uses https://github.com/google/iree/blob/main/build_tools/cmake/iree_tablegen_doc.cmake to generate markdown files

[ScottTodd]

Python API docs have been started: https://github.com/google/iree/pull/7183

[Sameeranjoshi]

MLIR Dialects

https://google.github.io/iree/Dialects - 404 - Not found

[ScottTodd]

This issue was written before we migrated from our old website to our new website. The page was at that location. It doesn't exist now.

[ScottTodd]

Here's an archive of what we had published for the MLIR dialects: https://github.com/openxla/iree/tree/99c6ce409783eaffb37dd46f3f4a6ef761556f99/docs/Dialects

[ScottTodd]

I just discussed publishing MLIR dialect reference pages with @jpienaar .

https://github.com/openxla/iree/pull/13991 gives reference pages a clear place to go

Preview:

Ideas:

• Set table of contents depth to omit "Attributes", "Operands", "Results" from the TOC: https://stackoverflow.com/questions/63828084/mkdocs-how-can-i-specify-toc-depth-for-a-single-document
• Add mermaid.js diagrams to docs and render them? https://squidfunk.github.io/mkdocs-material/reference/diagrams/
• Check that the pages are easy enough to view on mobile and desktop (particularly around the TOC and scrolling)
• Trim the fully qualified namespaces from the C++ symbol names, e.g. hal.buffer.assert (::mlir::iree_compiler::IREE::HAL::BufferAssertOp) -> hal.buffer.assert (BufferAssertOp)
• Don't publish the SimpleIO sample dialect
• Add MLIR syntax highlighting (contribute to Pygments or use highlightjs-mlir?)
• Add some context to the reference pages? Which dialects are (more) implementation details and which are (more) hackable/user-input dialects/ops

[ScottTodd]

MLIR dialect reference pages are published now at https://openxla.github.io/iree/reference/mlir-dialects/

---

The Python bindings have some documentation support at https://github.com/openxla/iree/tree/main/docs/api_docs/python -> https://iree-python-api.readthedocs.io/en/latest/

That could either be updated / actively maintained as-is, or we could fold it into the main website at e.g. https://openxla.github.io/iree/reference/python/. I think it's worth testing mkdocstrings or some other integration into the main website. We might be able to keep the same styling as the rest of the website if API docs are easy enough to navigate. Read the Docs doesn't seem to be providing that much extra support (like symbol search) on top of basic pagination and page styling.

[ScottTodd]

I gave mkdocstrings a try, progress at https://github.com/ScottTodd/iree/tree/docs-python-mkdocstrings

Impressions:

• It's a bit awkward compared to Sphinx, though we could get it close with customization options, a few edits to our docstrings, and perhaps some some CSS
  • The section style options don't seem to handle line breaks in the source... a multiline comment should not spill into a second row in a table, for example
  • We have some Sphinx-style code blocks (.. code-block:: python, that aren't getting picked up even with the sphinx docstring style
• Integration alongside the rest of our documentation on the same website, with the same theme, etc. is quite nice
• Configuring a page for each of our Python modules could be a bit tedious, though there are ways to automate that, and we already have a directory of .rst files that has likely drifted from the source for readthedocs

[ScottTodd]

Ah, missed a few dialect pages: https://github.com/openxla/iree/tree/main/llvm-external-projects/iree-dialects/include/iree-dialects/Dialect

• Input (iree_input)
• LinalgExt (iree_linalg_ext)
• LinalgTransform (iree_linalg_transform)

Steps to include:

• Build using CMake, if not covered by iree-doc target: https://github.com/openxla/iree/blob/ccf886b786f70d9bda4e2f7a01fe6825cffc5ab8/docs/website/generate_extra_files.sh#L27-L32
• Add --strip-prefix flag to CMakeLists.txt files: https://github.com/openxla/iree/blob/ccf886b786f70d9bda4e2f7a01fe6825cffc5ab8/build_tools/cmake/iree_tablegen_doc.cmake#L58-L60
• Copy into website directory: https://github.com/openxla/iree/blob/ccf886b786f70d9bda4e2f7a01fe6825cffc5ab8/docs/website/generate_extra_files.sh#L38-L42
• Maybe add context or some other note about why the dialects are in llvm-external-projects/iree-dialects (using the postprocess_dialect_docs.py script? Or add more information to the dialect summary/description strings?)
• Add to website nav: https://github.com/openxla/iree/blob/ccf886b786f70d9bda4e2f7a01fe6825cffc5ab8/docs/website/mkdocs.yml#L141-L144 https://github.com/openxla/iree/blob/ccf886b786f70d9bda4e2f7a01fe6825cffc5ab8/docs/website/docs/reference/mlir-dialects/index.md?plain=1#L8-L10

[ScottTodd]

Adding on to the previous comment, files like https://github.com/openxla/iree/blob/main/llvm-external-projects/iree-dialects/include/iree-dialects/Dialect/LinalgTransform/CMakeLists.txt will need _add_doc functions, with the logic in add_mlir_doc (looks like our LLVM_EXTERNAL_PROJECTS code doesn't use MLIR/LLVM CMake helper functions, at least for tablegen?)

[ScottTodd]

I started adding the llvm-external-project/iree-dialect/ dialects/ops to the website on a branch: https://github.com/ScottTodd/iree/tree/docs-iree-dialects.

StructuredTransformOpsExt.td is a bit weird to generate a standalone page for (extensions to an existing dialect, missing some high level / summary information) so I'm not sure if it should be included:

We could add some context like how the ops there extend https://mlir.llvm.org/docs/Dialects/Transform/ (either in the .td file somehow or by inserting some text into the .md file that is used by the static site generator)

fyi @ftynse @nicolasvasilache @matthias-springer (How useful would you find a page like https://openxla.github.io/iree/reference/mlir-dialects/Flow/ for the IREE transform extensions? Any suggestions for how to add context?)

[nicolasvasilache]

I need to double check what remains in there but the idea would be to completely remove it.

[ftynse]

It's useful to have IMO. I'm constantly checking the main transform dialect page for syntax and pre/post-conditions of the transforms I'm using. For context, it should be sufficient to link to https://mlir.llvm.org/docs/Dialects/Transform/#overview at the top of the page. There will be an overview and a description of the extension mechanism. Alternatively, split out the "documentation" part of https://github.com/llvm/llvm-project/blob/main/mlir/docs/Dialects/Transform.md into a separate .md file that will be included from Transform.md and into IREE doc somehow.

[ScottTodd]

An idea for C / Python API docs (and possibly MLIR...?): we can publish versions of the docs, with a version matching each stable (or major) release. Having a separate repo like https://github.com/pytorch/pytorch.github.io/tree/site/docs would give us a good place to store generated files without bloating the main repo (and we could also pull in projects like shark-turbine into the generation if that would make sense).