bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform

https://birdhouse-deploy.readthedocs.io/en/latest/

Apache License 2.0

4 stars 6 forks source link

weaver: bump version to 5.6.0 #463

Closed fmigneault closed 2 months ago

fmigneault commented 2 months ago

Overview

See changes in https://pavics-weaver.readthedocs.io/en/latest/changes.html#changes-5-6-1 Running here: https://hirondelle.crim.ca/weaver/versions Corresponding changes evaluated during OGC initiatives with cross-platform provider interoperability tests.

Changes

Non-breaking changes

Weaver: bump version to 5.6.1.
- See full changes details in Weaver changes
- In summary:
- multiple control setting options to customize some behaviors
- improved OGC API - Processes standard conformance
- improved support of Common Workflow Language (CWL) features (secrets, sub-workflow, auth-propagation, etc.)
Weaver: WPS retry logic on post-compose step.
- Apply --network birdhouse_default to the Docker curl image to allow HTTP requests to properly resolve against the running services (WPS bird providers, Weave and Magpie). In some cases, this network would not be automatically resolved.
- Fix the index used during HTTP request retry to avoid going one step over the intended retry attempts.

Breaking changes

Related Issue / Discussion

Closes https://github.com/bird-house/birdhouse-deploy/pull/439 Merge the PR since it has issues running the CI somehow. Replace by this one all together.

Additional Information

CI Operations

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

crim-jenkins-bot commented 2 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2691/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-weaver
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1629/

NOTEBOOK TEST RESULTS

    
[2024-06-12T00:00:38.249Z] ============================= test session starts ==============================
[2024-06-12T00:00:38.249Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-12T00:00:38.249Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-12T00:00:38.249Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-12T00:00:38.249Z] collected 308 items
[2024-06-12T00:00:38.249Z] 
[2024-06-12T00:00:50.754Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-06-12T00:01:11.626Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  9%]
[2024-06-12T00:01:18.665Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-06-12T00:02:06.478Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-06-12T00:02:17.921Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-06-12T00:02:30.706Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-06-12T00:02:35.790Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 21%]
[2024-06-12T00:13:22.765Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-06-12T00:13:25.311Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-06-12T00:13:33.066Z] ...............                                                          [ 30%]
[2024-06-12T00:13:41.038Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-06-12T00:13:48.318Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-06-12T00:14:03.846Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-06-12T00:14:11.100Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-06-12T00:14:15.778Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-06-12T00:18:14.477Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-06-12T00:19:39.406Z] .............                                                            [ 49%]
[2024-06-12T00:19:40.792Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-06-12T00:19:43.020Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-06-12T00:20:08.416Z] .................                                                        [ 58%]
[2024-06-12T00:20:16.471Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-06-12T00:20:17.850Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-06-12T00:20:35.920Z] ........F                                                                [ 63%]
[2024-06-12T00:20:45.399Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-06-12T00:20:55.123Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-06-12T00:20:57.031Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-06-12T00:21:03.716Z] .....                                                                    [ 71%]
[2024-06-12T00:21:42.449Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-06-12T00:22:50.716Z] ............                                                             [ 75%]
[2024-06-12T00:24:57.202Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-06-12T00:25:10.672Z] .....s.                                                                  [ 77%]
[2024-06-12T00:25:17.260Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-06-12T00:25:35.478Z] ..                                                                       [ 78%]
[2024-06-12T00:25:43.628Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-06-12T00:26:48.784Z] .........                                                                [ 82%]
[2024-06-12T00:26:58.798Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-06-12T00:31:58.727Z] ....                                                                     [ 83%]
[2024-06-12T00:31:58.728Z] PAVICS-landing-master/content/notebooks/hydrology/01_Intro.ipynb .....   [ 85%]
[2024-06-12T00:32:02.964Z] PAVICS-landing-master/content/notebooks/hydrology/02_Calibration.ipynb . [ 85%]
[2024-06-12T00:32:08.160Z] .....                                                                    [ 87%]
[2024-06-12T00:32:12.406Z] PAVICS-landing-master/content/notebooks/hydrology/03_Watershed_properties.ipynb . [ 87%]
[2024-06-12T00:32:26.948Z] .............                                                            [ 91%]
[2024-06-12T00:32:31.137Z] PAVICS-landing-master/content/notebooks/hydrology/04_Time_series_analysis.ipynb . [ 92%]
[2024-06-12T00:32:32.874Z] ......                                                                   [ 94%]
[2024-06-12T00:32:35.175Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-06-12T00:35:09.315Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-12T00:35:09.315Z] 
[2024-06-12T00:35:09.315Z] =================================== FAILURES ===================================

fmigneault commented 2 months ago

@huard ~Do you know if anything changed in the reference file that could cause hummingbird to fail the ncdump?~

~Running it on hirondelle, which still use the older birdhouse-deploy=1.42.2 tag, and hummingbird=0.5_dev, I get the same output file with contents Error: generating ncdump failed. Output:. I'm starting to think the cause could be from the input file itself.~

~https://hirondelle.crim.ca/twitcher/ows/proxy/hummingbird?service=WPS&request=Execute&identifier=ncdump&version=1.0.0&dataInputs=dataset_opendap=/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc~

~Also tested on pavics, same result.~

~https://pavics.ouranos.ca/twitcher/ows/proxy/hummingbird?service=WPS&request=Execute&identifier=ncdump&version=1.0.0&dataInputs=dataset_opendap=/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc~

Scratch all that. Input URL was invalid. Works for both https://hirondelle.crim.ca/twitcher/ows/proxy/hummingbird?service=WPS&request=Execute&identifier=ncdump&version=1.0.0&dataInputs=dataset_opendap=https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc

and https://pavics.ouranos.ca/twitcher/ows/proxy/hummingbird?service=WPS&request=Execute&identifier=ncdump&version=1.0.0&dataInputs=dataset_opendap=https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc

tlvu commented 2 months ago

@fmigneault we do have a notebooks/hummingbird.ipynb so maybe you could use the same input file as that notebook in your weaver notebook? That hummingbird notebook runs nightly so we are sure the test input file and the hummingbird service both work.

fmigneault commented 2 months ago

@tlvu I don't think the issue is on the Weaver side, since it only collects the text file from hummingbird, and if the hummingbird process failed completely, it would not reach the point of reading the file.

I can run the WPS request on hummingbird on this CI test server: https://host-140-118.rdext.crim.ca/twitcher/ows/proxy/hummingbird?service=WPS&request=Execute&identifier=ncdump&version=1.0.0&dataInputs=dataset_opendap=https://host-140-118.rdext.crim.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc generating: https://host-140-118.rdext.crim.ca/wpsoutputs/hummingbird/1db03ea2-29d2-11ef-9a15-0242ac1e0211/nc_dump_mII_kU.txt

I can also look at the resulting file when the execution is performed by the notebook with Weaver: https://host-140-118.rdext.crim.ca/wpsoutputs/weaver/public/9f597496-358a-4c16-87e0-b4567d880254/output/nc_dump_MZ84Fh.txt (which does contain the error message, but where is it coming from if the output was generated?)

I am really confused about the cause of that error...

fmigneault commented 2 months ago

Update: found the potential cause.

Somewhere along the Weaver->CWL->WPS chain, either OWSLib or PyWPS inject an empty file for the dataset input, on top of the provided dataset_opendap URL. This causes hummingbird ncdump to actually receive 2 "inputs", one being an invalid NetCDF and the expected OpenDAP reference. This explains why both the error message and the expected NetCDF text dump are displayed simultaneously.

~Digging deeper for a fix to avoid this empty input injection...~ Relevant fix: https://github.com/crim-ca/weaver/pull/666

crim-jenkins-bot commented 2 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2696/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-weaver
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1633/

NOTEBOOK TEST RESULTS

    
[2024-06-14T14:33:35.126Z] ============================= test session starts ==============================
[2024-06-14T14:33:35.126Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-14T14:33:35.126Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2024-06-14T14:33:35.126Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-14T14:33:35.126Z] collected 308 items
[2024-06-14T14:33:35.126Z] 
[2024-06-14T14:33:42.735Z] notebooks-auth/geoserver.ipynb .....FFFF.....FFF.                        [  5%]
[2024-06-14T14:34:04.988Z] notebooks-auth/test_cowbird_jupyter.ipynb ........F.                     [  9%]
[2024-06-14T14:34:12.022Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-06-14T14:35:26.918Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-06-14T14:35:35.840Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-06-14T14:35:45.214Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-06-14T14:36:01.191Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 21%]
[2024-06-14T14:50:05.515Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-06-14T14:50:07.427Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-06-14T14:50:16.027Z] ...............                                                          [ 30%]
[2024-06-14T14:50:24.985Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-06-14T14:50:31.579Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-06-14T14:50:47.489Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-06-14T14:50:52.580Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-06-14T14:50:57.148Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-06-14T14:54:55.132Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-06-14T14:56:13.357Z] .............                                                            [ 49%]
[2024-06-14T14:56:16.225Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-06-14T14:56:18.137Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-06-14T14:56:38.920Z] .................                                                        [ 58%]
[2024-06-14T14:56:46.096Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-06-14T14:56:47.478Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-06-14T14:56:57.783Z] .........                                                                [ 63%]
[2024-06-14T14:57:09.111Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-06-14T14:57:19.521Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-06-14T14:57:20.471Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-06-14T14:57:24.224Z] .....                                                                    [ 71%]
[2024-06-14T14:57:46.179Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-06-14T14:58:05.291Z] ............                                                             [ 75%]
[2024-06-14T14:58:23.392Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-06-14T14:58:54.722Z] .....s.                                                                  [ 77%]
[2024-06-14T14:59:01.296Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-06-14T14:59:18.767Z] ..                                                                       [ 78%]
[2024-06-14T14:59:26.911Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-06-14T15:00:30.377Z] .........                                                                [ 82%]
[2024-06-14T15:00:40.411Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-06-14T15:08:25.465Z] ....                                                                     [ 83%]
[2024-06-14T15:08:26.424Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 84%]
[2024-06-14T15:08:32.588Z] ....                                                                     [ 85%]
[2024-06-14T15:08:37.885Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-06-14T15:08:43.364Z] .....                                                                    [ 87%]
[2024-06-14T15:08:47.567Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-06-14T15:09:03.402Z] .............                                                            [ 91%]
[2024-06-14T15:09:08.690Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-06-14T15:09:09.756Z] ......                                                                   [ 94%]
[2024-06-14T15:09:12.263Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-06-14T15:12:15.994Z] notebooks/stress-tests.ipynb ...F..                                      [100%]
[2024-06-14T15:12:15.994Z] 
[2024-06-14T15:12:15.994Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 2 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2697/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-weaver
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1632/

NOTEBOOK TEST RESULTS

    
[2024-06-14T14:33:30.739Z] ============================= test session starts ==============================
[2024-06-14T14:33:30.739Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-14T14:33:30.739Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-14T14:33:30.739Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-14T14:33:30.739Z] collected 308 items
[2024-06-14T14:33:30.739Z] 
[2024-06-14T14:33:41.007Z] notebooks-auth/geoserver.ipynb ...............FF.                        [  5%]
[2024-06-14T14:34:01.010Z] notebooks-auth/test_cowbird_jupyter.ipynb ........F.                     [  9%]
[2024-06-14T14:34:08.410Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-06-14T14:35:26.838Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-06-14T14:35:36.306Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-06-14T14:35:46.988Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-06-14T14:35:51.878Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 21%]
[2024-06-14T14:49:39.847Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-06-14T14:49:39.847Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-06-14T14:49:43.404Z] ...............                                                          [ 30%]
[2024-06-14T14:49:51.769Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-06-14T14:49:59.167Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-06-14T14:50:16.636Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-06-14T14:50:37.879Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-06-14T14:50:42.557Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-06-14T14:54:55.455Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-06-14T14:56:11.836Z] .............                                                            [ 49%]
[2024-06-14T14:56:15.916Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-06-14T14:56:17.799Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-06-14T14:56:37.183Z] .................                                                        [ 58%]
[2024-06-14T14:56:44.939Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-06-14T14:56:45.886Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-06-14T14:57:03.174Z] .........                                                                [ 63%]
[2024-06-14T14:57:12.446Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-06-14T14:57:24.610Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-06-14T14:57:25.992Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-06-14T14:57:29.376Z] .....                                                                    [ 71%]
[2024-06-14T14:57:51.342Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-06-14T14:58:10.639Z] ............                                                             [ 75%]
[2024-06-14T14:58:28.774Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-06-14T14:59:00.293Z] .....s.                                                                  [ 77%]
[2024-06-14T14:59:08.419Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-06-14T14:59:25.222Z] ..                                                                       [ 78%]
[2024-06-14T14:59:33.349Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-06-14T15:00:41.465Z] .........                                                                [ 82%]
[2024-06-14T15:00:51.482Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-06-14T15:08:51.530Z] ....                                                                     [ 83%]
[2024-06-14T15:08:51.530Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 84%]
[2024-06-14T15:08:54.919Z] ....                                                                     [ 85%]
[2024-06-14T15:08:59.141Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-06-14T15:09:04.879Z] .....                                                                    [ 87%]
[2024-06-14T15:09:09.087Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-06-14T15:09:14.236Z] .............                                                            [ 91%]
[2024-06-14T15:09:18.444Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-06-14T15:09:20.187Z] ......                                                                   [ 94%]
[2024-06-14T15:09:22.880Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-06-14T15:12:27.043Z] notebooks/stress-tests.ipynb ...F..                                      [100%]
[2024-06-14T15:12:27.043Z] 
[2024-06-14T15:12:27.043Z] =================================== FAILURES ===================================

fmigneault commented 2 months ago

OK, so Weaver's tests are working again now, but I'm getting new errors from GeoServer tests and the stress-test.

@mishaschwartz @tlvu Any chance those errors are introduced by https://github.com/bird-house/birdhouse-deploy/pull/463/commits/f252ee8ad5af827a1163f85ed9fa30ffb76a0bee including latest birdhouse-deploy changes as well? Both GeoServer and THREDDS (used in stress test) are somewhat irrelevant for Weaver.

fmigneault commented 2 months ago

Running build: http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/ to test (http://daccs-jenkins.crim.ca/job/PAVICS-e2e-workflow-tests/job/master/1634/) if master can run on its own without this PR's change.

mishaschwartz commented 2 months ago

Any chance those errors are introduced by https://github.com/bird-house/birdhouse-deploy/commit/f252ee8ad5af827a1163f85ed9fa30ffb76a0bee

It looks like the tests on the master branch (http://daccs-jenkins.crim.ca/job/PAVICS-e2e-workflow-tests/job/master/1634/) passed so I don't think so.

But I don't think that the new errors have anything to do with this PR either since the code here hasn't changed

@fmigneault can you explain what changes you made to resolve the error here (https://github.com/bird-house/birdhouse-deploy/pull/463#issuecomment-2161847658) so that we can help figure out what changed?

tlvu commented 2 months ago

OK, so Weaver's tests are working again now, but I'm getting new errors from GeoServer tests and the stress-test.

@mishaschwartz @tlvu Any chance those errors are introduced by f252ee8 including latest birdhouse-deploy changes as well? Both GeoServer and THREDDS (used in stress test) are somewhat irrelevant for Weaver.

@fmigneault pretty sure 100% not related to that autodeploy change since CRIM Jenkins do not autodeploy anything.

That said, for stress-test I also see random errors on our side, on our staging machines where we track master. Much less frequent on our production system were we track our own fork with the caching in Magpie/Twitcher disabled.

fmigneault commented 2 months ago

run tests

crim-jenkins-bot commented 2 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2703/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : update-weaver
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1639/

NOTEBOOK TEST RESULTS

    
[2024-06-18T22:02:57.669Z] ============================= test session starts ==============================
[2024-06-18T22:02:57.669Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-18T22:02:57.669Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-18T22:02:57.669Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-18T22:02:57.669Z] collected 308 items
[2024-06-18T22:02:57.669Z] 
[2024-06-18T22:03:08.653Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-06-18T22:04:40.168Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  9%]
[2024-06-18T22:04:40.169Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-06-18T22:05:26.674Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-06-18T22:05:36.049Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 16%]
[2024-06-18T22:05:44.837Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 18%]
[2024-06-18T22:05:55.212Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 21%]
[2024-06-18T22:13:14.624Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 25%]
[2024-06-18T22:13:16.540Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 25%]
[2024-06-18T22:13:24.316Z] ...............                                                          [ 30%]
[2024-06-18T22:13:32.165Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2024-06-18T22:13:38.895Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2024-06-18T22:13:54.237Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 36%]
[2024-06-18T22:13:59.181Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2024-06-18T22:14:03.701Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 39%]
[2024-06-18T22:17:56.016Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2024-06-18T22:19:05.138Z] .............                                                            [ 49%]
[2024-06-18T22:19:07.848Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 50%]
[2024-06-18T22:19:09.641Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2024-06-18T22:19:25.697Z] .................                                                        [ 58%]
[2024-06-18T22:19:33.589Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 60%]
[2024-06-18T22:19:34.973Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2024-06-18T22:19:51.993Z] .........                                                                [ 63%]
[2024-06-18T22:20:01.764Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2024-06-18T22:20:11.258Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2024-06-18T22:20:13.185Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 69%]
[2024-06-18T22:20:16.508Z] .....                                                                    [ 71%]
[2024-06-18T22:20:38.486Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 71%]
[2024-06-18T22:20:55.396Z] ............                                                             [ 75%]
[2024-06-18T22:21:13.520Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-06-18T22:21:38.594Z] .....s.                                                                  [ 77%]
[2024-06-18T22:21:45.181Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 78%]
[2024-06-18T22:21:59.975Z] ..                                                                       [ 78%]
[2024-06-18T22:22:09.995Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 79%]
[2024-06-18T22:23:10.507Z] .........                                                                [ 82%]
[2024-06-18T22:23:20.523Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-06-18T22:28:19.166Z] ....                                                                     [ 83%]
[2024-06-18T22:28:19.166Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 84%]
[2024-06-18T22:28:19.166Z] ....                                                                     [ 85%]
[2024-06-18T22:28:22.467Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-06-18T22:28:28.336Z] .....                                                                    [ 87%]
[2024-06-18T22:28:31.659Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-06-18T22:28:46.826Z] .............                                                            [ 91%]
[2024-06-18T22:28:52.128Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-06-18T22:28:53.180Z] ......                                                                   [ 94%]
[2024-06-18T22:28:55.867Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-06-18T22:31:35.068Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-18T22:31:35.068Z] 
[2024-06-18T22:31:35.068Z] =============================== warnings summary ===============================

fmigneault commented 2 months ago

@mishaschwartz @tlvu ⬆️ Retested and everything worked. Seems like the previous errors were sporadic issues...