Improving Tutorials

[steremma]

Documentation in general

In general, a well written and maintained documentation can be divided into 4 concrete elements as explained in this talk:

1. Reference
2. Discussions
3. Tutorials
4. How to Guides

1 in our case is achieved with the docstrings and the Sphinx building the html content from them. We already have a strong base reference and multiple people are working on increasing coverage (myself included).

Regarding 2, we already have gitter, mailing lists and issue specific discussions in the GitHub issues and Pull Requests pages.

We have some overlap between 3 and 4 since we use jupyter notebooks for both but it is not very clear which notebook is a tutorial or a guide. For example I would say the sklearn_api notebooks are tutorials because they only show the basics (holding the user by the hand), while the model specific notebooks like word2vec IMDB/Wikipedia are more like guides because they solve a very specific problem. Perhaps we need to split them into two categories (folders) named tutorials and guides.

The problem

Its extremely important that the tutorials run always (new releases do not break the notebooks) and everywhere (the tutorial will run on every OS, python version or distribution etc.). This is very hard to guarantee at the moment.

Solution

We need to test the notebooks. By testing I mean make sure all cells run and not raise any error. (Can we also test for exact outputs?) Most google results show bad solutions but these two seem promising (although a bit hacky):

• https://gist.github.com/lheagy/f216db7220713329eb3fc1c2cd3c7826
• https://blog.thedataincubator.com/2016/06/testing-jupyter-notebooks/

Advantages

• Probably less work than the alternative
• No additional dependencies

Disadvantages

• Seems a bit hacky to me (@menshikh-iv thoughts?)
• These tests will make sure that no error was raised but will say nothing about the correctness of results. We could put assertions in the notebooks to make sure they raise, but this will make the notebooks look ugly.
• The notebooks are not automatically converted to rsts trivially, we therefore end up dealing with 2 "sources of truth" : the official doc page and the notebooks themselves.

Alternative

As we discussed with @menshikh-iv, maybe we should migrate from notebooks to Sphinx gallery. Using this approach our tutorials are Python scripts.

Advantages

• The tutorials are python files rather than .ipynb files.
• We can generate the documentation rsts directly. This ensures that there is a single source of truth and that new notebooks can be trivially added to the online documentation.

Disadvantages

• Requires new dependencies that have to do with plotting because this is mostly a plotting framework.
• Will take more time to implement since I need to get familiar with it.
  • It is not entirely obvious to me how we will be testing it.

Extra thoughts

Could we also add visualizations in tutorials? This is easy using both alternatives but I am not sure if we can come up with meaningful graphs.

[menshikh-iv]

Thanks for posting @steremma :+1: Guys @yurkai @anotherbugmaster @CLearERR let's discuss current point!

[menshikh-iv]

Some thoughts:

About 2: we almost ignore stack-overflow, this doesn't look like a good idea (because this is very popular QA service and significantly most popular than our mailing lists)

About 3, 4 (general view): we didn't split it, we have many notebooks that contain:

• tutorials
• howto guides
• benchmarks
• whatever

It's all in a heap and it's very difficult to find what you need. We have no any kind of "index".

About "testing" that proposed by @steremma - I'm already trying these solutions, this doesn't work, because

1. Most of our notebooks need several hours for the run
2. In notebooks, users import any random stuff (that even not our "test dependency")
3. Impossible to check code-style correctly in this way (thanks "ipython magics" =/)

Also, notebooks produce many problems like

• hard to change (if you change 1-line, you need to rerun all notebook)
• different ipython/python versions -> metadata that always changed (useless diff that always happens)
• hard to write py2+3 compatible notebook (and the check is very slow and annoying too, many manual actions)
• impossible to resolve the merge conflict

For this reason, @anotherbugmaster and I propose https://github.com/sphinx-gallery/sphinx-gallery approach for "tutorials" and "how to" guides (instead of notebooks). But for the case, when our notebook demonstrates large (in the meaning of size of data/running time/consuming memory) end2end example - we can create a new repo (for notebooks only) and move it to this repository.

Also, "for free" we receive nice features:

• sphinx-based indexing
• possibility to make links directly from documentation (and don't think about that this link will be broken after some time)
• single "source of truth" (right now, we have some intro tutorials as .rst and .ipynb that completely desync, because contributors modify only .ipynb)
• binder integration (now this will make sense)

About Disadvantages of this approach that was mention by @steremma:

Requires new dependencies that have to do with plotting because this is mostly a plotting framework.

This is not a problem, because of current dependency for documentation only (not "core" dependency).

Will take more time to implement since I need to get familiar with it.

This isn't hard, really, also @anotherbugmaster make several examples, how to use this in #1809

It is not entirely obvious to me how we will be testing it.

It's very simple: when you build documentation - you run all of these "gallery" examples (this is one CLI option).

[steremma]

I agree with the points raised, especially the difficulties with keeping track of notebooks as I have experienced the same issues in my workplace. The linked PR will be a useful reference for working with sphinx gallery. I will start looking into it as soon as I complete my previously assigned tasks (probably in 1 week from now)