Jupyter docker: new full build with latest of almost everything except xclim and ravenpy to smooth transition

[tlvu]

Overview

This new full build has latest of almost everything except xclim and ravenpy as intermediate step to smooth transition to pandas 2.2 freq strings changes.

Changes

• New: save conda env export, DockerHub build logs and Jenkins test result in the repo to track changes much more easily between releases

• Jenkins: add SAVE_RESULTING_NOTEBOOK_TIMEOUT for slow notebooks or slow machine

• Jupyter env changes:

  • add conda-pack so we can export the conda env outside of the docker image if need to run locally without docker
  • upgrade from Python 3.9 to 3.11
  • Relevant changes (alphabetical order):

• • birdy=0.8.4=pyh1a96a4e_0
• • birdhouse-birdy==0.8.7

major upgrade from v2 to v3

• • bokeh=2.4.3=pyhd8ed1ab_3
• • bokeh=3.4.1=pyhd8ed1ab_0
• • cartopy=0.21.1=py39h6e7ad6e_0
• • cartopy=0.23.0=py311h320fe9a_0
• • cf_xarray=0.8.0=pyhd8ed1ab_0
• • cf_xarray=0.9.0=pyhd8ed1ab_0
• • cfgrib=0.9.10.4=pyhd8ed1ab_0
• • cfgrib=0.9.11.0=pyhd8ed1ab_0
• • cftime=1.6.2=py39h2ae25f5_1
• • cftime=1.6.3=py311h1f0f07a_0
• • climpred=2.3.0=pyhd8ed1ab_0
• • climpred=2.4.0=pyhd8ed1ab_0
• • clisops=0.9.6=pyh1a96a4e_0
• • clisops=0.13.0=pyhca7485f_0
• • dask=2023.5.1=pyhd8ed1ab_0
• • dask=2024.5.0=pyhd8ed1ab_0
• • geopandas=0.13.0=pyhd8ed1ab_0
• • geopandas=0.14.4=pyhd8ed1ab_0
• • hvplot=0.8.3=pyhd8ed1ab_0
• • hvplot=0.9.2=pyhd8ed1ab_0
• • numpy=1.23.5=py39h3d75532_0
• • numpy=1.24.4=py311h64a7726_0
• • numba=0.57.0=py39hb75a051_1
• • numba=0.59.1=py311h96b013e_0

major upgrade from v1 to v2

• • pandas=1.3.5=py39hde0f152_0
• • pandas=2.1.4=py311h320fe9a_0

major upgrade to v1

• • panel=0.14.4=pyhd8ed1ab_0
• • panel=1.4.2=pyhd8ed1ab_0

major upgrade from v1 to v2

• • pydantic=1.10.8=py39hd1e30aa_0
• • pydantic=2.7.1=pyhd8ed1ab_0

Python 3.9 to 3.11

• • python=3.9.16=h2782a2a_0_cpython
• • python=3.11.6=hab00c5b_0_cpython
• • raven-hydro=0.2.1=py39h8e2dbb5_1
• • raven-hydro=0.2.4=py311h64a4d7b_0
• • ravenpy=0.12.1=py39hf3d152e_0
• • ravenpy==0.13.1
• • rioxarray=0.14.1=pyhd8ed1ab_0
• • rioxarray=0.15.5=pyhd8ed1ab_0
• • roocs-utils=0.6.4=pyh1a96a4e_0
• • roocs-utils=0.6.8=pyhd8ed1ab_0
• • scipy=1.9.1=py39h8ba3f38_0
• • scipy=1.13.0=py311h517d4fd_1
• • xarray=2023.1.0=pyhd8ed1ab_0
• • xarray=2023.8.0=pyhd8ed1ab_0
• • xclim=0.43.0=py39hf3d152e_1
• • xclim=0.47.0=py311h38be061_0
• • xesmf=0.7.1=pyhd8ed1ab_0
• • xesmf=0.8.5=pyhd8ed1ab_0
• • xskillscore=0.0.24=pyhd8ed1ab_0
• • xskillscore=0.0.26=pyhd8ed1ab_0
• • xscen=0.8.2=pyhd8ed1ab_0
• • figanos==0.3.0
• • xncml==0.2
• • xncml==0.4.0

## Test

- Deployed as "beta" image in production for bokeh visualization performance regression testing.
- Manual test notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance and it looks fine.
- Jenkins build:
  - Default notebooks, all passed: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/54792e6510adfcd1bb21e1bd31fdfa36c5c634e0/docker/saved_buildout/jenkins-buildlogs-default.txt
  - Raven notebooks, only known `HydroShare_integration.ipynb` failing: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/jenkins-buildlogs-raven.txt

## Related Issue / Discussion

- Matching notebook fixes:
  - Pavics-sdi: PR https://github.com/Ouranosinc/pavics-sdi/pull/321
  - Finch: PR url: None
  - PAVICS-landing: PR https://github.com/Ouranosinc/PAVICS-landing/pull/78
  - RavenPy: PR https://github.com/CSHS-CWRA/RavenPy/pull/356
  - Resolves https://github.com/Ouranosinc/PAVICS-landing/issues/65
  - Resolves https://github.com/Ouranosinc/PAVICS-landing/issues/66

- Deployment to PAVICS: https://github.com/bird-house/birdhouse-deploy/pull/453

- Jenkins-config changes for new notebooks: PR url: None

- Other issues found while working on this one
  - https://github.com/computationalmodelling/nbval/issues/204
  - https://github.com/jupyterlab-contrib/jupyter-archive/issues/132
  - https://github.com/CSHS-CWRA/RavenPy/issues/357
  - https://github.com/CSHS-CWRA/RavenPy/issues/361
  - https://github.com/CSHS-CWRA/RavenPy/issues/362

- Previous release: PR https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/pull/134

## Additional Information

Full diff conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/compare/81deb99a4111b9c0945e26e06809be2d90ea59b0...931cfc924a147d07b59e88badff9f170e852a03b#diff-e8f2a6a53085ae29bb7cedc701c1d345a330651ae971555e85a5c005e94f4cd9

Full new conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/conda-env-export.yml

DockerHub build log
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/docker-buildlogs.txt

[tlvu]

@tlogan2000 The new Jupyter env is live. You can send announcement to our users to restart their server to get the new version.

The previous "stable" version is now the new "previous" version. I've also added a "last-py38-stable-version" to preserve our last Python 3.8 env in case our user need an even older version to reproduce their work.

[Zeitsperre]

We need to cull a few of those entries. beta, alpha, gamma, and last are a lot of development options for images.

[tlvu]

We need to cull a few of those entries. beta, alpha, gamma, and last are a lot of development options for images.

Well beta, alpha, gamma are placeholders when we have many test candidates with slight variations to test for bug fixes. As we have just experienced, it's pretty labor intensive to release a fully tested/certified Jupyter env. So all those placeholders does help during the testing phase.

Previously we deploy those test versions on our staging host but since the staging host do not have all the data, we could not fully test those environments. And the only 100% sure way to not have any surprise is to test directly on the production host ! I've seen case where the JupyterHub is unable to launch the JupyterLab server ! Having the test env on the production host eliminate this issue.

All the app specific envs are for our apps to reproduce the exact env of their runtime. Not all of them, as you can see, are on the "Stable" image. Since their have their own version, they are not force to update at the same time we release a new Stable version. Just ensuring all the notebooks passed is way too labor intensive, I do not want to add all the apps to the certification list !

The last one are an ancien remark from David. We will eventually have last-py39, last-py311 ...

So that's the reason for all those multiple entries.

[tlvu]

We need to cull a few of those entries. beta, alpha, gamma, and last are a lot of development options for images.

Well beta, alpha, gamma are placeholders when we have many test candidates with slight variations to test for bug fixes. As we have just experienced, it's pretty labor intensive to release a fully tested/certified Jupyter env. So all those placeholders does help during the testing phase.

Previously we deploy those test versions on our staging host but since the staging host do not have all the data, we could not fully test those environments. And the only 100% sure way to not have any surprise is to test directly on the production host ! I've seen case where the JupyterHub is unable to launch the JupyterLab server ! Having the test env on the production host eliminate this issue.

All the app specific envs are for our apps to reproduce the exact env of their runtime. Not all of them, as you can see, are on the "Stable" image. Since their have their own version, they are not force to update at the same time we release a new Stable version. Just ensuring all the notebooks passed is way too labor intensive, I do not want to add all the apps to the certification list !

The last one are an ancien remark from David. We will eventually have last-py39, last-py311 ...

So that's the reason for all those multiple entries.

Oh forgot to add that we are trying to release more often so we are continuous trying to test newer versions. Those beta, alpha, gamma will continuously be useful.