include notebooks into RTD documentation

[gplssm]

The Sphinx extension nbsphinx integrates notebooks in Sphinx docs. The normal workflow includes execution of a notebook. Because this takes too long for all scenarios of WindNODE_ABW, it is not an option. Execution can be bypassed and notebooks can rendered from executed notebooks. As the size and number of notebooks and changes made to these would grow the size of the repo, @nesnoj and I decided not to commit executed notebooks to this repo. Instead, we are thinking of creating a separate repo to only store executed notebooks. Git tags could help to manage versions.

Btw, Binder is not really an option because of computational limitations and the need to store the results data somewhere publicly accessible.

Tasks

• [x] Check how notebooks from external sources can be included, i.e. nbsphinx-link We do it differently
• [x] Check if (even anonymous) gists are a good alternative to a github repo
• [x] May it is an option to convert notebooks to .rst and render from there via sphinx include NOT

[gplssm]

Updated tasks after some research, prototyping and re-thinking of it

• [x] Write function to parse directory "notebooks" and include "..toc::" direct on the top and write everything to create single_scenario_results.rst
• [x] Remove unused content from .ipynb template and save it in a separate issue for @nailend
• [x] Adjust heading levels in .ipynb template such that they are aligned with TOC
• [x] Remove static notebook top heading and replace it dynamically with scenario name extracted from file name using {{ env.doc2path(env.docname, base=None) }}. Read in prolog and epilog of nbsphinx
• [x] Add function to conf.py that downloads executed notebooks from gist
• [ ] ~Add executed-notebook-to-publish-on-gist script/function for uploading notebooks created with papermill~
• [x] Change: download .ipynb only if docs/notebooks is empty or does not exist
• [x] Exchange Zenodo token by read-only one for zenodo.org (instead of sandbox.zenodo.org)

[gplssm]

In general, the above ticked tasks are done. Some things done differently

• Using Zenodo instead of gist.github.com because GitHub limits max. file size to 100 mb (we have ~140 mb NBs)
• Top heading in each notebook isn't replaced with jinja templating
• no automatic upload script so far for executed .ipynb NBs, simple drag+drop on Zenodo is ok

In general, the NBs showing results for each scenario can be integrated into the documentation (see on the example of two NBs). But it might not work with all NBs. I we could run into problems because of resource limits of RTD, which limits each build to 15 min. First build failed because of other reason and the second try ran into the resource limits

I see two options to get around this

1. Somehow reduce file size of ipynb dramatically (down to ~20...30 %)
2. Use .html exports of NBs and include them statically (at least 140 mb -> 50 mb)

[gplssm]

I found 5 figures causing 5/6th of the file size

• 3.3 Autarky municipalities: 13.6 MB
• 4.3 Timeseries: 12 MB
• 7.2 Line Loading Distribution: 46 MB
• 8.3.1 Activation: 22 MB
• 8.3.3 DSM Timeseries: 26 MB

Executed notebooks (.ipynb) require 19 to 22 Mb disk space after removing these figures. Still, the docs build process fails on RTD.

[nesnoj]

* 7.2 Line Loading Distribution: 46 MB

Does the data or the plot itself consume that much space? I'd assume that the full data is stored somewhere in the plot JS? We could use a different renderer to make the largest plots static, e.g. svg, example (source):

import plotly.graph_objects as go
fig = go.Figure(
    data=[go.Bar(y=[2, 1, 3])],
    layout_title_text="A Figure Displayed with the 'svg' Renderer"
)
fig.show(renderer="svg")

Still, the docs build process fails on RTD.

Still because of the build time?