Consitently reaching timeout of certain notebook during birdhouse-deploy CI E2E tests

[fmigneault]

When running this notebook: https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb

within the CI tests of https://github.com/bird-house/birdhouse-deploy using this command: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/d466af7d7d1a9488852926efa30f030fdb5418f5/runtest#L70-L72

The last cell code:

print("computing with dask client ...")
compute(jobs)
print("finished")

Always end up in

nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 240 seconds.

Is there a way to reduce the computation time of that cell? When testing it on other setup/machine, is the execution time releatively close to ~240s? Would it be sufficient to only increase slightly the timeout in PAVICS-E2E or is the duration way over that?

[fmigneault]

@tlvu @Zeitsperre @tlogan2000 ?

[tlogan2000]

@fmigneault The last section of notebook 3 is not usually run during JENKINS testing as it is an example for users of how to run a large calculation on an an ensemble of climate simulations ... The cell should be flagged with # NBVAL_SKIP. Output data from this example is needed for notebook 4 and 5 however, so a precalulated output is provided here : https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/output.zip

[tlvu]

@fmigneault jupyter nbconvert step is just to generate a .ipynb with the output of the run.

It's basically a 2nd run.

pytest is the first run and it is the one we are interested in.

I would suggest in your CI to disable the 2nd run with SAVE_RESULTING_NOTEBOOK=false, see https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/d466af7d7d1a9488852926efa30f030fdb5418f5/runtest#L57C9-L57C32

Having 2 runs will double the time. Since your CI can be triggered very frequently you should focus on the speed.

We run nightly so we can have both runs.

Edit:

Failure in jupyter nbconvert step will not fail the build. If you have build failure, look at the pytest step.

[fmigneault]

Let's try with SAVE_RESULTING_NOTEBOOK=false. Hopefully the output logs of pytest nbval directly in Jenkins CI will be sufficient to debug the notebooks.

[fmigneault]

reference: https://gitlab.com/crim.ca/clients/daccs/daccs-e2e-jenkins/-/merge_requests/24

[fmigneault]

SAVE_RESULTING_NOTEBOOK=false is now enabled by default (still overridable manually) CI execution time ~40-48min => ~21min.