Proposition: text-based source files via Jupytext

[rsokl]

Source control for notebooks is awful. Editing notebooks is awful (@davidmascharka can maybe attest to these more than anyone on the planet). Meta data in notebooks is awful.

I am proposing a workflow that consists of writing / version-controlling .md or .py files as the source documents of PLYMI. And then using jupytext and nbsphinx to convert these to notebooks and then html.

Background

Why do we use notebooks in PLYMI? The answer was that it was by far the easiest solution for me when I first started working on this project. nbsphinx provides an incredibly easy means for jumping from a familiar notebook format to nice html. That being said, only a couple of PLYMI's sections actually use nbsphinx as intended: example 1, example 2. The rest of these sections could be written in pure .rst... almost.

The reason why we need nbsphinx for all of this is because of these guys:

These custom "admonition boxes", with nice headers and code-blocks, cannot be produced in reSt files. The author of nbsphinx rolled his own special support for these boxes.

I experimented with PLYMI without these boxes, and it turns out that they are super important to the experience that we give the reader. Delimiting takeaways, reading comprehension questions, etc. is critical. So until common markdown provides support for these, we need nbsphinx and the whole stack that it builds off of.

Proposed Solution

I would like to use jupytext to mediate conversions between notebooks and text files. From their readme:

Have you always wished Jupyter notebooks were plain text documents?

Yes.

Wished you could edit them in your favorite IDE?

Yes...!

And get clear and meaningfull diffs when doing version control?

YES!!!

Then... Jupytext may well be the tool you're looking for!

OKAY!!

jupytext supports a workflow in which a notebook, say notebook.ipynb, can be "paired" with a python script, notebook.py (or other type of file, like markdown). You can now edit the script and run jupytext --sync notebook.py to propagate changes to the notebook. You can also edit the notebook, and changes will propagate to the text file automatically. Ultimately we would only share the text files, and generate the notebook locally as needed.

Thus a full work cycle of editing PLYMI source and building the site anew would look like:

1. Pull PLYMI
2. Edit text files
3. Version control the text files
4. Use jupytext to build the notebooks from text
5. Use sphinx/nbsphinx to build html from notebooks
6. Version control the html

For @davidmascharka an edit workflow would simply be steps 1-3.

I am going to experiment with this and verify that we can indeed generate all of PLYMI as-is from text files as the true source files.

From there, I will mess with the smaller details / config setting for jupytext to see if I can prescribe a simple, portable workflow that can be stood up from scratch easily. And one that gets us from text to html quickly. Ultimately it is important that this methodology does not require specialized knowledge of these tools.

I would love feedback thoughts on this. Let me know if anything about jupytext and its capabilities jump out as being particularly useful for PLYMI or if you have any concerns.

Edit: it looks like nbsphinx can leverage jupytext directly if we modify the sphinx conf.py file. This means that sphinx itself would do the text -> notebook conversion.

[rsokl]

Leaving this open for a bit in case people want to catch up with the significant change. All (but two) notebooks were replaced with markdown in #81!

@davidmascharka the text files are so... beautiful 😭

[rsokl]

Current issues with jupytext-markdown:

• code blocks in jupytext-markdown cannot have consecutive newline characters
• they do not retain the different flavors of raw-cells

Neither of these are hard-blocking issues. I am working with the jupytext folks to help remedy this.

[rsokl]

These issues were fixed in jupytext 1.1.0