Documentation structure overhaul

[PeterDSteinberg]

• [ ] See how Holoviews uses notebooks that are tested automatically to fill out code and output examples in the documentation. Coordinate with @jbednar on the details of how that works. We have too many code snippets in Elm's docs to maintain currently. Aim to remove code snippets expressed in the ./docs folder (somehow get it automatically from notebook cells + add CI testing to those notebooks). May involve making new notebooks.
  • [ ] The outputs of cells should be tracked so you can do assertions
  • [ ] Get the documentation snippets tested somehow on each push / Travis test
• [ ] We also need doctests in the source code - not enough there now (delay this work ~1.5 months)
• [ ] We also have plan the messaging around earthio + xarray_filters as experimental repos and how to get help there or find help ...? Should help for all these be in one place (the current readthedocs)? Phased implementation towards 2 to 3 separate docs sites that link to each other?

Order of operations:

• [ ] Copy the scaffolding of the notebook testing / docs system of Holoviews - customize as necessary (can start this immediately)
• [ ] Content edits - regarding the interactions of repos (the docs are now all elm-based with little description about the point of Earthio / xarray_filters and not much help on those either). Approx date for this is: Sept 15, 2017
• [ ] Content edits - regarding refactored usage: Oct 1, 2017 or earlier

[PeterDSteinberg]

cc @gbrener Ideas?

[jbednar]

Copy the scaffolding of the notebook testing / docs system of Holoviews - customize as necessary (can start this immediately)

Yes and no! Our scaffolding is amazing and also hellish. We have agreed on a plan to make it portable between projects, but we don't have time to start that plan yet until we can deliver on one of our client projects that has a milestone approaching. I agree that it's urgent to get this sort of stuff in place for elm (as it is for datashader), but is there any way this could be delayed by 1 month? If so we'll be able to clean it up, separate it into a useful system, and then start using it widely.

[gpfreitas]

@PeterDSteinberg I'll wait for your answer to @jbednar's question.

In the meantime, can someone explain at a high level the pros/cons of the way Bokeh does documentation (traditional Sphinx, from what I can see) vs the way that holoviews does?

[PeterDSteinberg]

Delaying for a month sounds good @jbednar @gpfreitas . I'll label this issue as deferred.

[jbednar]

In the meantime, can someone explain at a high level the pros/cons of the way Bokeh does documentation (traditional Sphinx, from what I can see) vs the way that holoviews does?

The HoloViews website is actually based on the Bokeh docs infrastructure, but it extends it to build the site nearly entirely from notebooks rather than from .rst, to automate various additional aspects of the gallery when building from notebooks, to add automatic cross-linking between notebooks (so that the notebooks link to .ipynb files but when deployed as a web site the links go to the deployed .htmls), and to allow automatic testing of notebook outputs.

The result is that writing and maintaining the website becomes nearly entirely a matter of writing notebooks and putting them into a certain directory structure, which is good because those same notebooks are then available to the users to re-run and compare their results, and then use them as starting points. Everything then stays up to date automatically as long as you maintain the notebooks, in principle (once we finish up some key sticky points).

We may be able to move the "month away" time forward; we have yet another project that needs a website, and so we are meeting on Monday to see if there's any way we can get this functionality published as a viable separate tool earlier.

[PeterDSteinberg]

@gpfreitas is working on documentation in xarray_filters

[PeterDSteinberg]

See notes on issue #185 regarding documentation needs related to Elm PR #192's changes for evolutionary search.

[PeterDSteinberg]

Docstrings are absent or insufficient in these places (and likely others):

• elm.pipeline.steps - See @gpfreitas 's work in xarray_filters for wrapping the docstring of an underlying wrapped function (xarray_filters.datasets) and do the same kind of thing for elm.pipeline.steps wrapping the docs of the sklearn estimator that is the ._cls attribute of an elm.pipeline.steps class like elm.pipeline.steps.linear_model.LinearRegression and including a note that the wrapping of that sklearn class is intended for use with xarray_filters.MLDataset / xarray.Dataset.

[PeterDSteinberg]

cc @hsparra

[PeterDSteinberg]

See #206

[PeterDSteinberg]

See https://github.com/ContinuumIO/elm/issues/189

[gbrener]

@PeterDSteinberg are we still considering adopting the Holoviews style of sharing notebooks between examples, CI testing, and documentation? I looked into the Holoviews approach a bit, and found it to align with @jbednar's earlier comments - it seems a bit "rough around the edges", and not in a generally-applicable state yet. I did find a tool called "nbsphinx" which might get us closer to this goal, if you still think it's something worth pursuing.

[jbednar]

The CI testing stuff should improve very soon now; Chris Ball has just set up an nbval-based approach that looks like it should greatly simplify the amount of code and special configuration we have to maintain. (I.e., now that nbval is available, we can let them take the bulk of the hit for doing such testing, with our own special-purpose stuff being just a little extra effort). Thus having dual-purpose example/testing notebooks is a reasonable short-term goal.

That leaves sharing notebooks between examples and docs, i.e. triple-purpose notebooks. That stuff is definitely still rough in the holoviews approach, which won't be fixed in October. For datashader what I am doing is to write the notebooks assuming that it will eventually work smoothly, and that way the documentation will exist, it just won't easily be published to a web site. Then once the web publishing bit becomes more generalizable, it should transition relatively painlessly to use the new support. Unless someone on the elm project wants to help us make that happen sooner? :-)

[PeterDSteinberg]

Phased implementation of new conda pkg for doing this docs auto-gen and auto-test of notebooks:

• First - conda pkg just for the documentation auto-gen from notebooks (@gbrener working on this)
  • Install nbsphinx and make easy to use import nbsphinx - see nbsphinx in conf.py of holoviews/doc
  • Readme/ boilerplate on how to do the auto-gen package. What is the package called. Docs-builder?
• Second - testing of the docs / notebooks automatically

[PeterDSteinberg]

Plan

• This is also discussed in https://github.com/bokeh/datashader/issues/437
• @gbrener @TomAugspurger and I will adapt that generalization of the notebooks testing/auto-docs around Nov 14 that way we can focus on content / other changes rather than the testing/docs harness