fix weaver data-source resolution for workflow steps

[fmigneault]

Overview

Weaver: Adjust invalid data_sources.yml definitions.

Changes

Non-breaking changes

• Weaver: Adjust invalid data_sources.yml definitions.

  • Add the missing data_sources.yml volume mount for weaver-worker.
  • When weaver-worker runs a Workflow, the nested step process locations need to be resolved according to the current "localhost" instance. However, the Web API running in weaver service is not visible from the worker. Since the configuration is shared between weaver and weaver-worker, use the public endpoint of weaver to make process URL resolution consistent, and also provide more useful references in job logs when resolution fails.

Breaking changes

• n/a

Related Issue / Discussion

• identified while working on https://github.com/crim-ca/weaver/pull/661

Additional Information

• https://pavics-weaver.readthedocs.io/en/latest/processes.html#data-source

CI

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2683/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : fix-weaver-data-source
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1625/

NOTEBOOK TEST RESULTS

    
[2024-06-05T06:59:09.686Z] ============================= test session starts ==============================
[2024-06-05T06:59:09.686Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-05T06:59:09.686Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-05T06:59:09.686Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-05T06:59:09.686Z] collected 276 items
[2024-06-05T06:59:09.686Z] 
[2024-06-05T06:59:19.157Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-06-05T06:59:45.416Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-06-05T06:59:52.459Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-06-05T07:00:47.606Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-06-05T07:00:56.918Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-06-05T07:01:05.065Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-06-05T07:01:19.369Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 23%]
[2024-06-05T07:08:26.871Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-06-05T07:08:26.871Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-06-05T07:08:33.789Z] ...............                                                          [ 34%]
[2024-06-05T07:08:41.481Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-06-05T07:08:50.336Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-06-05T07:09:04.625Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-06-05T07:09:14.067Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-06-05T07:09:18.154Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 44%]
[2024-06-05T07:13:10.575Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-06-05T07:14:16.118Z] .............                                                            [ 54%]
[2024-06-05T07:14:18.291Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-06-05T07:14:20.066Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-06-05T07:14:35.775Z] .................                                                        [ 65%]
[2024-06-05T07:14:42.791Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2024-06-05T07:14:44.176Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2024-06-05T07:15:02.544Z] .........                                                                [ 71%]
[2024-06-05T07:15:12.063Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2024-06-05T07:15:21.318Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2024-06-05T07:15:22.262Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-06-05T07:15:25.817Z] .....                                                                    [ 79%]
[2024-06-05T07:15:40.717Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2024-06-05T07:15:55.534Z] ............                                                             [ 84%]
[2024-06-05T07:16:10.451Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-06-05T07:16:31.025Z] .....s.                                                                  [ 86%]
[2024-06-05T07:16:37.592Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-06-05T07:16:52.362Z] ..                                                                       [ 88%]
[2024-06-05T07:17:00.490Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-06-05T07:17:59.638Z] F........                                                                [ 91%]
[2024-06-05T07:18:07.789Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-06-05T07:23:07.781Z] F...                                                                     [ 93%]
[2024-06-05T07:23:07.781Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-06-05T07:25:30.966Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-05T07:25:30.966Z] 
[2024-06-05T07:25:30.966Z] =================================== FAILURES ===================================