Test suite: run all notebooks in one or more dirs rather than explicitly named notebooks

DiODeProject / MuMoT

Multiscale Modelling Tool - mathematical modelling without the maths

https://mumot.readthedocs.io/

GNU General Public License v3.0

21 stars 4 forks source link

Test suite: run all notebooks in one or more dirs rather than explicitly named notebooks #325

Open willfurnass opened 5 years ago

willfurnass commented 5 years ago

Currently explictly list the notebooks to be run in tox.ini. A cleaner approach would be to have

one or more directories within which all notebooks should be run
one or more distinct directories in which notebooks aren't automatically run

willfurnass commented 5 years ago

@jarmarshall @joefresna Which notebooks should/shouldn't be run by tox?

jarmarshall commented 5 years ago

I would say run everything (including in subfolders) in:

[ ] docs
[ ] TestNotebooks
[ ] DemoNotebooks

willfurnass commented 4 years ago

Getting this sorted has proved more difficult than first anticipated - our CI jobs already take ~50 mins to run (Issue #366) and attempts to run additional notebooks so as to address this issue keep resulting in cell execution timeouts (see PR #363). Bump this Issue to 1.2.0?

jarmarshall commented 4 years ago

I'm happy with that - stick it on the second release milestone for now I think...

willfurnass commented 4 years ago

@jarmarshall How about we migrate all Notebooks other than the user manual and test notebooks to other repos to cut down CI job time? These auxillary repos could have their own CI and could depend on stable versions of MuMoT.

jarmarshall commented 4 years ago

I think this could be a sensible suggestion since it could also solve another problem, of how to publish new notebooks which are linked from readthedocs, without needing to tag a new release or have readthedocs point to the master branch to get updates?

willfurnass commented 4 years ago

Notebooks in separate repos could have their own mybinder links in their READMEs and have requirements.txt files referencing stable versions of MuMoT.

We'd still need to curate mybinder links in the README and Sphinx docs for MuMoT itself for Notebooks bundled with the MuMoT package.

Disadvantages of unbundling the less general notebooks:

Not automatically tested against new MuMoT releases as pinned to specific releases
How to make unbundled notebooks discoverable if they are spread between various repos? A central index of some sort could be useful.

@jarmarshall If you want to go ahead with this have a think about which notebooks you'd want to unbundle and how they could be distributed between repos.

jarmarshall commented 4 years ago

OK, this is getting overcomplicated IMO - maybe we should revert to keeping the notebooks as is... but if those to be run as tests need to be explicitly included, we could just exclude those that take a long time / are redundant...

willfurnass commented 4 years ago

Hmm, with that approach, Notebooks not run during MuMoT CI jobs won't ever be tested against newer versions of MuMoT. If some Notebooks are moved to separate repos then at least the CI for those repos ensures they're been tested against a known version of MuMoT, even if that version isn't the latest.