Create automatic links to Jupyter Notebooks in Binder for running code snippets and tutorials

[jsheunis]

The idea

The idea is to create links to free Binder environments in the cloud such that users following the handbook can run example code in Jupyter notebooks on Binder, i.e. without having to install DataLad or other dependencies locally. This issue is here for discussion / sharing idea / tracking progress.

Starting point

See the playground repo (datalad-tutorial-binder) which already has a Binder environment set up for the OHBM tutorial.

Next steps

Next steps would include seeing how we can streamline the conversion of rst files to Jupyter notebooks (currently using sphinxcontrib-jupyter) without having to edit the notebooks after conversion.

Also, all of this should be incorporated into the handbook building process without breaking stuff.

Things to be aware of

• The asyncio error with DataLad import into Jupyter notebooks/labs
• Issue with cloning and subsequently getting a dataset using https described here

Next steps

I'll be working on this on and off, if anyone wants to join in, please do!

[welcome[bot]]

(Image: CC-BY license, The Turing Way Community, & Scriberia. Zenodo. http://doi.org/10.5281/zenodo.3332808) Hi there, and welcome to the DataLad Handbook! :orange_book: :wave: Thank you for filing an issue. We're excited to have your input and welcome your idea! :blush: If you haven't done so already, please make sure you check out our Code of Conduct.

[jsheunis]

Update

Have been looking into ways to generate notebooks (from rst files) that are ready for execution, i.e. they don't require post-conversion editing. Looks like sphinxcontrib-jupyter has limited capability to interpret and format different scripting languages for code blocks.

For example line 13-16 of the OHBM.rst:

In the command line, this always start with the general ``datalad`` command::

   datalad

will convert correctly to a code cell in the resulting notebook, but the line magic ! will have to be prepended to the datalad command if it needs to be run in a Python kernel.

I've tried other options like:

In the command line, this always start with the general ``datalad`` command

.. code-block:: <bash> <shell> <sh> <console> <python>

    datalad

however this format only works when python is specified for the syntax highlighting, and then it converts in the same way as described above for the original lines.

Will have to look into some other options/workarounds.

[jsheunis]

Update:

I looked into this again during the handbook hackathon. It looks like the following tools will be useful in this context:

• MyST: https://myst-parser.readthedocs.io/en/stable/index.html
• MyST-NB: https://myst-nb.readthedocs.io/en/latest/index.html
• rst-to-myst: https://rst-to-myst.readthedocs.io/en/stable/index.html
• jupytext: https://jupytext.readthedocs.io/

Basically: rst-to-myst can be used to convert rST docs into MyST-type Markdown docs, which can be read and executed as notebooks by Jupyter interfaces (-notebook, -lab, Binder, etc). Alternatively, MyST-type Markdown docs can also be converted into notebooks by jypytext.

I've played around with this locally and it seems better than anything else I've tried before, although the challenge will come in with converting all (custom) directives correctly into MyST. E.g. the {code cell} tag is currently not correctly added to the MyST code cells in markdown (upon first try, no further customization) and hence the cells don't display as executable code. The kernel loads correctly as the bas kernel, though.

In general, MyST (which has a sphinx plugin) could be worth considering for (aspects of) the handbook in future.

Tagging @adswa and @mih in case of interest.

[adswa]

I have missed this update when you posted it 4 months ago - thanks @jsheunis, this indeed sounds useful! I'll keep it on my radar, and maybe pick a slot to place it into the handbook RFD.