bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform
https://birdhouse-deploy.readthedocs.io/en/latest/
Apache License 2.0
4 stars 7 forks source link

Jupyterhub: periodically check whether the logged-in user still have permission to access #402

Closed mishaschwartz closed 12 months ago

mishaschwartz commented 1 year ago

Overview

By setting the JUPYTERHUB_CRYPT_KEY environment variable in the env.local file, jupyterhub will store user's authentication information (session cookie) in the database. This allows jupyterhub to periodically check whether the user still has permission to access jupyterhub (the session cookie is not expired and the permission have not changed).

The minimum duration between checks can be set with the JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE variable which is an integer (in seconds).

Note that users who are already logged in to jupyterhub will need to log out and log in for these changes to take effect.

To forcibly log out all users currently logged in to jupyterhub you can run the following command to force the recreation of the cookie secret:

  docker exec jupyterhub rm /persist/jupyterhub_cookie_secret && docker restart jupyterhub

First discussed here: https://github.com/bird-house/birdhouse-deploy/pull/358#issuecomment-1737374614

Changes

Non-breaking changes

Breaking changes

Related Issue / Discussion

Related to #334

Additional Information

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2226/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : periodically-check-jupyterub-access
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-90.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1407/

NOTEBOOK TEST RESULTS
    
[2023-11-09T20:45:32.388Z] ============================= test session starts ==============================
[2023-11-09T20:45:32.388Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-09T20:45:32.388Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-11-09T20:45:32.388Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-09T20:45:32.388Z] collected 255 items
[2023-11-09T20:45:32.388Z] 
[2023-11-09T20:45:43.365Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-09T20:45:51.880Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-09T20:46:00.085Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-09T20:46:10.535Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-09T20:46:17.101Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-09T20:53:35.236Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-09T20:53:40.525Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-09T20:55:17.042Z] ...............                                                          [ 30%]
[2023-11-09T20:55:24.469Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-09T20:55:31.310Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-09T20:55:48.946Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-09T20:55:50.335Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-09T20:55:55.898Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-09T20:56:00.664Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-09T20:58:47.230Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-09T21:00:05.762Z] .............                                                            [ 53%]
[2023-11-09T21:00:10.158Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-09T21:00:12.903Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-09T21:00:31.375Z] .................                                                        [ 64%]
[2023-11-09T21:00:40.758Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-09T21:00:42.670Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-09T21:00:54.812Z] ........F                                                                [ 70%]
[2023-11-09T21:01:04.308Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-09T21:01:13.906Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-09T21:01:15.298Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-09T21:01:18.106Z] ......                                                                   [ 80%]
[2023-11-09T21:01:24.694Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-09T21:01:40.190Z] .............                                                            [ 85%]
[2023-11-09T21:01:50.446Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-09T21:02:25.273Z] ....s.                                                                   [ 88%]
[2023-11-09T21:02:33.387Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-09T21:02:47.520Z] ...                                                                      [ 90%]
[2023-11-09T21:03:02.434Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-09T21:03:24.627Z] ......                                                                   [ 92%]
[2023-11-09T21:03:25.970Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-09T21:06:05.546Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-09T21:06:05.546Z] 
[2023-11-09T21:06:05.546Z] =================================== FAILURES ===================================
    
  
mishaschwartz commented 1 year ago

@tlvu

Do they have to re-login or this is completely transparent to them? If they have to re-login, please note that in the CHANGES.md and I will have to notify our users before rolling this change out.

Yes, it is transparent to them. This feature will only start applying once the user has logged out and logged back in again.

If a user is already logged in, they will not have any authentication information added to the database yet, so the check that was added here (https://github.com/Ouranosinc/jupyterhub/pull/23) will allow them to remain logged in.

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2305/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : periodically-check-jupyterub-access
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1435/

NOTEBOOK TEST RESULTS
    
[2023-11-27T15:28:47.086Z] ============================= test session starts ==============================
[2023-11-27T15:28:47.087Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-27T15:28:47.087Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-27T15:28:47.087Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-27T15:28:47.087Z] collected 255 items
[2023-11-27T15:28:47.087Z] 
[2023-11-27T15:28:58.624Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-27T15:29:06.372Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-27T15:29:14.947Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-27T15:29:25.987Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-27T15:29:34.837Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-27T15:37:14.955Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-27T15:37:14.955Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-27T15:37:23.700Z] ...............                                                          [ 30%]
[2023-11-27T15:37:33.444Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-27T15:37:40.202Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-27T15:37:56.248Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-27T15:37:57.629Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-27T15:38:03.004Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-27T15:38:07.886Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-27T15:41:51.536Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-27T15:43:04.562Z] .............                                                            [ 53%]
[2023-11-27T15:43:07.464Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-27T15:43:09.691Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-27T15:43:26.586Z] .................                                                        [ 64%]
[2023-11-27T15:43:34.942Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-27T15:43:36.333Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-27T15:43:54.960Z] ........F                                                                [ 70%]
[2023-11-27T15:44:04.507Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-27T15:44:13.497Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-27T15:44:15.412Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-27T15:44:17.848Z] ......                                                                   [ 80%]
[2023-11-27T15:44:24.452Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-27T15:44:39.146Z] .............                                                            [ 85%]
[2023-11-27T15:44:49.147Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-27T15:45:29.156Z] ....s.                                                                   [ 88%]
[2023-11-27T15:45:37.303Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-27T15:45:50.100Z] ...                                                                      [ 90%]
[2023-11-27T15:46:05.021Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-27T15:46:28.384Z] ......                                                                   [ 92%]
[2023-11-27T15:46:30.433Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-27T15:49:04.597Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-27T15:49:04.597Z] 
[2023-11-27T15:49:04.597Z] =================================== FAILURES ===================================
    
  
tlvu commented 1 year ago

If a user is already logged in, they will not have any authentication information added to the database yet, so the check that was added here (Ouranosinc/jupyterhub#23) will allow them to remain logged in.

@mishaschwartz

Very nice to ensure transparent upgrade.

Please add a note to the CHANGES.md that each existing Jupyter user have to logout and re-login for this new change to take effect.

This is to ensure node admin is not falsely thinking the security enhancement is effective simply with the new deployment.

For us, we probably will kill all user sessions somewhere during Xmas holidays so everyone is forced to re-login so we are sure this will take effect.

One way I found to force everyone to re-login again is to ./pavics-compose.sh stop jupyterhub, then rm /var/lib/docker/volumes/jupyterhub_data_persistence/_data/jupyterhub.sqlite, then ./pavics-compose.sh start jupyterhub. If you have another way to force everyone to re-login, please let me know. This way of going behind JupyterHub back is not very elegant.

Please also document in CHANGES.md one way to force everyone to re-login.

mishaschwartz commented 1 year ago

@tlvu

Ok I've updated the CHANGES.md file with the relevant information. Please have a look to see if that makes sense

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2313/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : periodically-check-jupyterub-access
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1442/

NOTEBOOK TEST RESULTS
    
[2023-11-29T16:50:50.507Z] ============================= test session starts ==============================
[2023-11-29T16:50:50.507Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-29T16:50:50.508Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-29T16:50:50.508Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-29T16:50:50.508Z] collected 255 items
[2023-11-29T16:50:50.508Z] 
[2023-11-29T16:51:01.203Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-29T16:51:09.205Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-29T16:51:17.789Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-29T16:51:27.870Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-29T16:51:36.653Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-29T17:29:58.006Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-29T17:29:58.006Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-29T17:30:02.545Z] ...............                                                          [ 30%]
[2023-11-29T17:30:12.212Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-29T17:31:14.869Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-29T17:31:42.893Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-29T17:31:44.533Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-29T17:31:50.028Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-29T17:31:54.795Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-29T17:36:11.200Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-29T17:38:00.071Z] .............                                                            [ 53%]
[2023-11-29T17:38:04.820Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-29T17:38:07.401Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-29T17:39:13.843Z] .................                                                        [ 64%]
[2023-11-29T17:39:22.195Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-29T17:39:24.102Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-29T17:39:25.408Z] .FFFFFFFF                                                                [ 70%]
[2023-11-29T17:39:36.007Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-29T17:39:46.033Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-29T17:39:49.342Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-29T17:40:03.495Z] ......                                                                   [ 80%]
[2023-11-29T17:42:25.030Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-29T17:42:30.300Z] .............                                                            [ 85%]
[2023-11-29T17:42:42.551Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-29T17:43:23.916Z] ....s.                                                                   [ 88%]
[2023-11-29T17:43:32.055Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-29T17:43:47.600Z] ...                                                                      [ 90%]
[2023-11-29T17:44:02.491Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-29T17:44:26.394Z] ......                                                                   [ 92%]
[2023-11-29T17:44:29.418Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-29T17:47:09.044Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-29T17:47:09.044Z] 
[2023-11-29T17:47:09.044Z] =================================== FAILURES ===================================
    
  
tlvu commented 12 months ago

@tlvu

Ok I've updated the CHANGES.md file with the relevant information. Please have a look to see if that makes sense

Good me for me, thanks.

crim-jenkins-bot commented 12 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2319/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : periodically-check-jupyterub-access
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1448/

NOTEBOOK TEST RESULTS
    
[2023-11-30T15:41:14.338Z] ============================= test session starts ==============================
[2023-11-30T15:41:14.338Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-30T15:41:14.338Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-30T15:41:14.338Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-30T15:41:14.338Z] collected 255 items
[2023-11-30T15:41:14.338Z] 
[2023-11-30T15:41:24.767Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-30T15:41:32.823Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-30T15:41:41.845Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-30T15:41:51.428Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-30T15:42:00.449Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-30T15:54:51.071Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-30T15:54:51.071Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb F [ 24%]
[2023-11-30T15:54:54.935Z] FFFFFFFFFFFFFFF                                                          [ 30%]
[2023-11-30T15:55:02.455Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb F.FFF    [ 32%]
[2023-11-30T15:55:10.729Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-30T15:55:27.107Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-30T15:55:28.495Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-30T15:55:34.028Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-30T15:55:38.533Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-30T16:00:20.527Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-30T16:01:31.875Z] .............                                                            [ 53%]
[2023-11-30T16:01:35.970Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-30T16:01:38.197Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-30T16:01:55.976Z] .................                                                        [ 64%]
[2023-11-30T16:02:03.886Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-30T16:02:05.804Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-30T16:02:28.977Z] .........                                                                [ 70%]
[2023-11-30T16:02:38.497Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-30T16:02:48.181Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-30T16:02:50.099Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-30T16:02:53.174Z] ......                                                                   [ 80%]
[2023-11-30T16:03:01.312Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-30T16:03:16.364Z] .............                                                            [ 85%]
[2023-11-30T16:03:26.361Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-30T16:04:15.370Z] ....s.                                                                   [ 88%]
[2023-11-30T16:04:23.508Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-30T16:04:39.005Z] ...                                                                      [ 90%]
[2023-11-30T16:04:51.220Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-30T16:05:15.712Z] ......                                                                   [ 92%]
[2023-11-30T16:05:17.328Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-30T16:07:51.480Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-30T16:07:51.480Z] 
[2023-11-30T16:07:51.480Z] =================================== FAILURES ===================================