search

bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform
https://birdhouse-deploy.readthedocs.io/en/latest/
Apache License 2.0
4 stars 6 forks source link

update `Magpie==4.1.1` and `Twitcher==0.10.0` #466

Closed fmigneault closed 2 months ago

fmigneault commented 4 months ago

Overview

Security updates.

Changes

Non-breaking changes

Breaking changes

Related Issue / Discussion

PRs that are an agglomeration of vulnerability fixes flagged by Snyk.

Additional Information

CI Operations

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

fmigneault commented 4 months ago

Manual trigger: http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2742/

crim-jenkins-bot commented 4 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2742/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-magpie-twitcher
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1656/

NOTEBOOK TEST RESULTS 
    
[2024-07-24T18:18:05.137Z] ============================= test session starts ==============================
[2024-07-24T18:18:05.137Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-07-24T18:18:05.137Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-07-24T18:18:05.138Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-07-24T18:18:05.138Z] collected 308 items
[2024-07-24T18:18:05.138Z] 
[2024-07-24T18:18:15.097Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-07-24T18:18:30.554Z] notebooks-auth/test_cowbird_jupyter.ipynb ......FFF.                     [  9%]
[2024-07-24T18:18:38.018Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-07-24T18:19:25.344Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-07-24T18:19:43.672Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-07-24T18:19:52.103Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-07-24T18:19:57.686Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 21%]
[2024-07-24T18:27:39.077Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-07-24T18:27:39.078Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-07-24T18:27:39.858Z] ...............                                                          [ 30%]
[2024-07-24T18:27:47.094Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-07-24T18:27:57.824Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-07-24T18:28:12.965Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-07-24T18:28:18.614Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-07-24T18:28:23.142Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-07-24T18:31:46.792Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-07-24T18:32:43.838Z] .............                                                            [ 49%]
[2024-07-24T18:32:48.293Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-07-24T18:32:49.811Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-07-24T18:33:05.845Z] .................                                                        [ 58%]
[2024-07-24T18:33:13.548Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-07-24T18:33:14.489Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-07-24T18:33:32.015Z] .........                                                                [ 63%]
[2024-07-24T18:33:41.485Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-07-24T18:33:50.467Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-07-24T18:33:51.857Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-07-24T18:33:54.925Z] .....                                                                    [ 71%]
[2024-07-24T18:34:13.282Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-07-24T18:34:30.026Z] ............                                                             [ 75%]
[2024-07-24T18:34:48.141Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-07-24T18:35:11.823Z] .....s.                                                                  [ 77%]
[2024-07-24T18:35:18.426Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-07-24T18:35:35.883Z] ..                                                                       [ 78%]
[2024-07-24T18:35:42.499Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-07-24T18:36:41.689Z] .........                                                                [ 82%]
[2024-07-24T18:36:51.735Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-07-24T18:41:42.433Z] ....                                                                     [ 83%]
[2024-07-24T18:41:44.351Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 84%]
[2024-07-24T18:41:49.659Z] ....                                                                     [ 85%]
[2024-07-24T18:41:53.868Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-07-24T18:41:59.199Z] .....                                                                    [ 87%]
[2024-07-24T18:42:03.407Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-07-24T18:42:28.222Z] .............                                                            [ 91%]
[2024-07-24T18:42:32.436Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-07-24T18:42:33.508Z] ......                                                                   [ 94%]
[2024-07-24T18:42:35.806Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-07-24T18:45:09.962Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-07-24T18:45:09.962Z] 
[2024-07-24T18:45:09.963Z] =================================== FAILURES ===================================
tlvu commented 4 months ago

FYI le notebook cowbird n'a pas passé.

fmigneault commented 4 months ago

@tlvu Thanks. Will need to investigate some more... The error seems to be caused by a JSON parsing issue on the Jupyter API endpoint. Not sure what could have caused it since all other notebooks doing Magpie logins worked... Maybe I will have time Friday, but might be delayed until the end of August otherwise.

fmigneault commented 2 months ago

run tests

fmigneault commented 2 months ago

run tests

crim-jenkins-bot commented 2 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2795/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-magpie-twitcher
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1682/

NOTEBOOK TEST RESULTS 
    
[2024-09-10T17:26:17.814Z] ============================= test session starts ==============================
[2024-09-10T17:26:17.814Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-09-10T17:26:17.814Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-09-10T17:26:17.814Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-09-10T17:26:17.814Z] collected 308 items
[2024-09-10T17:26:17.814Z] 
[2024-09-10T17:26:28.528Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-09-10T17:27:34.815Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  9%]
[2024-09-10T17:27:40.134Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-09-10T17:28:27.255Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-09-10T17:28:35.946Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-09-10T17:28:45.344Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-09-10T17:28:54.285Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 21%]
[2024-09-10T17:36:21.054Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-09-10T17:36:22.447Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-09-10T17:36:30.473Z] ...............                                                          [ 30%]
[2024-09-10T17:36:38.314Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-09-10T17:36:51.142Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-09-10T17:37:06.340Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-09-10T17:37:12.261Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-09-10T17:37:16.781Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-09-10T17:40:27.401Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-09-10T17:41:37.085Z] .............                                                            [ 49%]
[2024-09-10T17:41:38.474Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-09-10T17:41:40.311Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-09-10T17:41:56.400Z] .................                                                        [ 58%]
[2024-09-10T17:42:04.452Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-09-10T17:42:05.398Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-09-10T17:42:23.736Z] .........                                                                [ 63%]
[2024-09-10T17:42:32.824Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-09-10T17:42:41.391Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-09-10T17:42:42.772Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-09-10T17:42:45.846Z] .....                                                                    [ 71%]
[2024-09-10T17:43:07.824Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-09-10T17:43:22.764Z] ............                                                             [ 75%]
[2024-09-10T17:43:37.684Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-09-10T17:44:00.321Z] .....s.                                                                  [ 77%]
[2024-09-10T17:44:08.471Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-09-10T17:44:24.775Z] ..                                                                       [ 78%]
[2024-09-10T17:44:31.381Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-09-10T17:45:32.119Z] .........                                                                [ 82%]
[2024-09-10T17:45:40.259Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-09-10T17:50:41.745Z] ....                                                                     [ 83%]
[2024-09-10T17:50:41.745Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 84%]
[2024-09-10T17:50:41.745Z] ....                                                                     [ 85%]
[2024-09-10T17:50:41.745Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-09-10T17:50:45.327Z] .....                                                                    [ 87%]
[2024-09-10T17:50:49.533Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-09-10T17:51:16.797Z] .............                                                            [ 91%]
[2024-09-10T17:51:21.001Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-09-10T17:51:23.002Z] ......                                                                   [ 94%]
[2024-09-10T17:51:25.739Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-09-10T17:54:11.262Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-09-10T17:54:11.262Z] 
[2024-09-10T17:54:11.262Z] =================================== FAILURES ===================================
fmigneault commented 2 months ago

@tlvu Requesting review again since it's been a while. I want to make sure it doesn't cause a surprise on your side. The triggers are working again though! 🎉 Seems like the Cowbird tests are working now. Might have been just a sporadic issue?

Looks like the WMS is still failing. This one has very sporadic failures. Each of the failing tests below are caused by this WMS notebook (whether using these new versions of Magpie/Twitcher or the actual ones), but we can see it also passes ~½ times.

{AF96DEFE-03E6-4E91-90A9-DD933A55C87F}

tlvu commented 2 months ago

Requesting review again since it's been a while. I want to make sure it doesn't cause a surprise on your side.

Is there any DB upgrade? Otherwise it should be fine.

fmigneault commented 2 months ago

@tlvu

Is there any DB upgrade

There shouldn't be any changes.

The 3.x⇾4.x change was only to highlight the deprecation of older Python version (<3.8), which were not used anyway by the docker already using Python 3.11.