Publish notebooks in hosted documentation

[rland93]

I'm aware that there is a way to automatically run and publish jupyter notebooks in Sphinx documentation. I think it's a plugin for Sphinx that makes this possible.

It would be a nice addition to the documentation to have the existing notebooks there, and further, would reduce friction when writing tutorial documentation, since the tutorial can be created and tested as a notebook, and then published in one step instead of copy-pasting code.

[moorepants]

I use https://github.com/jupyter/jupyter-sphinx/ in many of my projects. You write in RestructuredText as usual but use the .. jupyter-execute:: directive instead of the standard code ones. This is my current project that uses it: https://moorepants.github.io/learn-multibody-dynamics/ if you'd like to see an example output.

[rland93]

Can't seem to get the GH actions for building/hosting documentation working with sphinx docs. Something something pandoc.

reading sources... [ 35%] autoapi/pendsim/utils/index
| reading sources... [ 41%] autoapi/pendsim/viz/index
| reading sources... [ 47%] customctl
| 
| Notebook error:
| PandocMissing in customctl.nblink:
| Pandoc wasn't found.
| Please check that pandoc is installed:
| https://pandoc.org/installing.html
| make: *** [Makefile:22: html] Error 2
| [sphinx-action] Build failed with 0 warnings
| Traceback (most recent call last):
|   File "/entrypoint.py", line 22, in <module>
|     action.build_all_docs(github_env, [os.environ.get("INPUT_DOCS-FOLDER")])
|   File "/sphinx_action/action.py", line 167, in build_all_docs
|     raise RuntimeError("Build failed")
| RuntimeError: Build failed

[moorepants]

Maybe install pandoc?

[rland93]

Fixed!

Good lord, what a tricky issue. Writing this down just to memorialize this process.

So, I wanted to host the notebooks and outputs in the documentation. But, I didn't want to rewrite the notebooks to be embedded in the sphinx documentation (which, @moorepants, turned out to be the most sensible thing).

Well, alright, there's plugins to convert them to rst -- then, they can get converted into html and github will host them. To do this, there's a thing called nbsphinx. nbsphinx works fine, if the notebooks are under the same root folder as the documentation. But, this isn't so. Our problem is that we have docs in pendsim/docs/ and notebooks in pendsim/notebooks. They're not in the same folder.

So, we actually can use a thing called nbsphinx-link to basically place a "link" to the notebooks in docs, so that nbsphinx thinks they're in there when the documentation is built.

That was yesterday.

Then, of course, this introduces a bunch of dependencies into the sphinx pipeline: you have to actually (?) run the notebooks when generating the documentation, or at least parse them. And then, you need to convert the markdown and output cells in the notebooks to rst. This was the origin of the missing pandoc dependency above. Another interesting thing that happens: when the notebooks are executed from the repo/notebooks folder, they're one level up from the repo; so if you want to import pendsim from the repo, you need to move one level up in the first cell of the notebook. BUT: if actually nbsphinx is running the notebooks, and they're in a different directory that's two levels up, you have a problem, because the interpreter is actually looking for pendsim in repo/docs. Now, you can jump to two levels up in the notebooks to fix this issue, but that will break them if you run them from the directory they're supposed to be running from, notebooks. Ask me how I know. The structure that works looks like this:

repo/
├─ notebooks/
│  ├─ experiment1.ipynb
├─ docs/
│  ├─ experiment1.nblink

Then, if you tell the interpreter to go one level up in experiment1.ipynb, that'll work whether you're in notebooks/ or docs/.

The fix is to uh... don't try to do it like this. Turned out to be a github actions learning experience for me, and one of those things where you waste 4.5 hours to save 45 minutes.

(?) or maybe just parse outputs? I'm still not sure.