Test slow notebooks

[MichaelClerx]

Similar to functional testing, we could replace the notebook testing cronjob on travis with tests on a local machine (but scheduled by azure). For example, we could fire off a test of a single notebook every 30 minutes (say at xx:15 and xx:45) This would let us get rid of the notebooks/slow notebooks division, and test even slow notebooks

[MichaelClerx]

@fcooper8472 do you think this would be easy to do with azure?

[ben18785]

@fcooper8472 @MichaelClerx Plan is to test all of the example notebooks once a month.

[MichaelClerx]

@fcooper8472 do you remember how far we got with this?

[fcooper8472]

Yes, we got no far I think!

Is @ben18785 's plan of doing a single run once a month of everything still viable?

A nice option (better IMO than doing it locally and maintaining the infrastructure) would be to see if GitHub actions could suffice. They offer 6 hours of runtime for individual runs, which I think ought to be plenty?

I could easily set this up as a trial. Firing off on a timer can be done with cron syntax so could easily be just a big thing run monthly (or weekly?).

Just to initially guess at feasibility, how long do the longest few of the long books take to run?

[MichaelClerx]

I think 6 hours should be enough! I'm still hesitant about github actions though. MS have done nothing in the last 30 years to make me trust them :-|

[fcooper8472]

I'm still hesitant about github actions though. MS have done nothing in the last 30 years to make me trust them :-|

Better get pints off GitHub then...

The previous option was to use Azure, which is also Microsoft.

We can't justify a boycott of GitHub actions without also getting rid of everything else MS related. For the record, though, I think Microsoft's commitment to the open source community over the last few years has been exceptional.

If the worry is that they might (one day) make people pay once they are hooked in:

1. It would be a PR nightmare for them
2. So could any of the other services we use that are currently free for open source

[fcooper8472]

These should all now be running daily at 4am and we can re-evaluate the effectiveness of GutHub Actions for the slow books after the first run.

[fcooper8472]

See conversation here about dependencies including autograd.

@MichaelClerx I'm moving the discussion to this issue as it may well be more relevant.

Could we just add a

        'books': [
            'autograd',
        ],

so the average user never gets non-core dependencies but we can still show off cool stuff in examples? The pints examples directory seems like the right home for automatic-differentiation-using-autograd as it seems like something a pints user might legitimately want to do?

[fcooper8472]

Hmm, there are 5 failing books:

examples/toy/automatic-differentiation-using-autograd.ipynb
examples/toy/distribution-german-credit-hierarchical.ipynb
examples/toy/model-lotka-volterra.ipynb
examples/toy/distribution-rosenbrock.ipynb
examples/toy/distribution-german-credit.ipynb

[MichaelClerx]

Thanks! There's lots of things users could want to do with pints that involve external dependencies. In fact, any serious application will. So I suggest linking to this examples on the examples README page, just like we link to examples within the PINTS repo. But let's keep the PINTS repo for developing PINTS and have all cool stuff using pints in other repos?

[fcooper8472]

Error on examples/toy/model-lotka-volterra.ipynb:

ValueError: Initial point for MCMC must have finite logpdf.

Not sure how to pin this one down... @MichaelClerx @ben18785

[MichaelClerx]

Thanks @fcooper8472 ! Ben made some improvements to the way toy ODE models work, which included changes to Lotka-Volterra, and then while reviewing the PR for this I made some further changes but failed to re-run the notebook. Fixing it now!

[MichaelClerx]

Should work now

[fcooper8472]

I think all the books should pass at this point.

@MichaelClerx I've made a temporary fix for the one book that requires autograd: it will "fail gracefully" if autograd isn't installed, with a sane message to the user suggesting they install it if they want the functionality.

I think this is substantially better than the current situation, and could be reviewed if there was an obvious new place for it to move in due course.

[MichaelClerx]

@fcooper8472 I think we can close this now?

[fcooper8472]

Yep, it's been running and stable. Typically takes between 45 and 65 mins per run.

[MichaelClerx]

Thanks!