Rename variables, constants and files from PAVICS to Birdhouse

[mishaschwartz]

Overview

For historical reasons the name PAVICS was used in variable names, constants and filenames in this repo to refer to the software stack in general. This was because, for a long time, the PAVICS deployment of this stack was the only one that was being used in production. However, now that multiple deployments of this software exist in production (that are not named PAVICS), we remove unnecessary references to PAVICS in order to reduce confusion for maintainers and developers who may not be aware of the historical reasons for the PAVICS name.

This update makes the following changes:

• The string PAVICS in environment variables, constant values, and file names have been changed to BIRDHOUSE (case has been preserved where possible).
  • For example:
    • PAVICS_FQDN -> BIRDHOUSE_FQDN
    • pavics_compose.sh -> birdhouse_compose.sh
    • THREDDS_DATASET_LOCATION_ON_CONTAINER='/pavics-ncml' -> THREDDS_DATASET_LOCATION_ON_CONTAINER='/birdhouse-ncml'
• Comment strings and documentation that refers to the software stack as PAVICS have been changed to use Birdhouse.
• Recreated the pavics-compose.sh script that runs birdhouse-compose.sh in backwards compatible mode.
  • Backwards compatible mode means that variables in env.local that contain the string PAVICS will be used to set the equivalent variable that contains BIRDHOUSE. For example, the PAVICS_FQDN variable set in the env.local file will be used to set the value of BIRDHOUSE_FQDN.
• Create a new entrypoint in bin/birdhouse that can be used to invoke pavics-compose.sh or birdhouse-compose.sh from one convenient location. This script also includes some useful options and provides a generic entrypoint to the stack that can be extended in the future.
• Removed unused variables:
  • CMIP5_THREDDS_ROOT

Migration Guide

• Update env.local file to replace all variables that contain PAVICS with BIRDHOUSE. Variable names have also been updated to ensure that they start with the prefix BIRDHOUSE_.

  • see env.local.example to see new variable names
  • see the BACKWARDS_COMPATIBLE_VARIABLES variable (defined in default.env) for a full list of changed environment variable names.

• Update any external scripts that access the old variable names directly to use the updated variable names.

• Update any external scripts that access any of the following files to use the new file name:

  old file name	new file name
  pavics-compose.sh	birdhouse-compose.sh
  PAVICS-deploy.logrotate	birdhouse-deploy.logrotate
  configure-pavics.sh	configure-birdhouse.sh
  trigger-pavicscrawler	trigger-birdhousecrawler

• The following default values have changed. If your deployment was using the old default value, update your env.local file to explicitly set the old default values.

  old variable name	new variable name	old default value	new default value
  POSTGRES_PAVICS_USERNAME	BIRDHOUSE_POSTGRES_USERNAME	postgres-pavics	postgres-birdhouse
  THREDDS_DATASET_LOCATION_ON_CONTAINER	(no change)	/pavics-ncml	/birdhouse-datasets
  THREDDS_SERVICE_DATA_LOCATION_ON_CONTAINER	(no change)	/pavics-data	/birdhouse-service-data
  THREDDS_DATASET_LOCATION_ON_HOST	(no change)	'${DATA_PERSIST_ROOT}/ncml'	'${BIRDHOUSE_DATA_PERSIST_ROOT}/thredds-datasets'
  THREDDS_SERVICE_DATA_LOCATION_ON_HOST	(no change)	'${DATA_PERSIST_ROOT}/datasets'	'${BIRDHOUSE_DATA_PERSIST_ROOT}/thredds-service-data'
  (hardcoded)	BIRDHOUSE_POSTGRES_DB	pavics	birdhouse
  PAVICS_LOG_DIR	BIRDHOUSE_LOG_DIR	/var/log/PAVICS	/var/log/birdhouse
  (hardcoded)	GRAFANA_DEFAULT_PROVIDER_FOLDER	Local-PAVICS	Local-Birdhouse
  (hardcoded)	GRAFANA_DEFAULT_PROVIDER_FOLDER_UUID	local-pavics	local-birdhouse
  (hardcoded)	GRAFANA_PROMETHEUS_DATASOURCE_UUID	local_pavics_prometheus	local_birdhouse_prometheus

• Update any jupyter notebooks that make use of the PAVICS_HOST_URL environment variable to use the new BIRDHOUSE_HOST_URL instead.

• Set the BIRDHOUSE_POSTGRES_DB variable to pavics in the env.local file. This value was previously hardcoded to the string pavics so to maintain backwards compatibility with any existing databases this should be kept the same. If you do want to update to the new database name, you will need to rename the existing database. For example, the following will update the existing database named pavics to birdhouse (assuming the old default values for the postgres username):

  docker exec -it postgres psql -U postgres-pavics -d postgres -c 'ALTER DATABASE pavics RENAME TO birdhouse'

  You can then update the env.local file to the new variable name and restart the stack

• Set the BIRDHOUSE_POSTGRES_USER variable to postgres-pavics in the env.local file if you would like to preserve the old default value. If you would like to change the value of BIRDHOUSE_POSTGRES_USER then also update the name for any running postgres instances. For example, the following will update the user named postgres-pavics to postgres-birdhouse:

  docker exec -it postgres psql -U postgres-pavics -d postgres -c 'CREATE USER "tmpsuperuser" WITH SUPERUSER'
  docker exec -it postgres psql -U tmpsuperuser -d postgres -c 'ALTER ROLE "postgres-pavics" RENAME TO "postgres-birdhouse"'
  docker exec -it postgres psql -U tmpsuperuser -d postgres -c 'ALTER ROLE "postgres-birdhouse" WITH PASSWORD '\''postgres-qwerty'\'
  docker exec -it postgres psql -U postgres-birdhouse -d postgres -c 'DROP ROLE "tmpsuperuser"'

  Note that the postgres-qwerty value is meant just for illustration, you should replace this with the value of the BIRDHOUSE_POSTGRES_PASSWORD variable. Note that you'll need to do the same for the stac-db service as well (assuming that you weren't previously overriding the STAC_POSTGRES_USER with a custom value).

  Changes

Non-breaking changes

• comment changes
• most changes are backwards compatible (but not all)

Breaking changes

• variable, file, constant name changes

Related Issue / Discussion

• Resolves #414

Additional Information

Links to other issues or sources.

• [x] Options/flags that modify how the compose script behaves should all use BIRDHOUSE_ as prefix
• [x] manually test upgrading an instance using the migration guide (above)
• [x] manually test autodeploy (note: we need to automate this somehow)
• [ ] update associated projects:
  • [x] https://github.com/DACCS-Climate/marble_client_python
  • [x] https://github.com/bird-house/pavics-jupyter-base
  • [ ] https://github.com/ouranosinc/pavics-datacatalog (deprecated, low priority)

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

[mishaschwartz]

To be consistent with the organization name (https://github.com/bird-house), I think the comment/doc mentions should be formatted either as bird-house or Bird-House instead of Birdhouse.

Yes, but it is consistent with the repo name: birdhouse-deploy. I think we're currently being consistent with the right thing. Even http://bird-house.github.io/ starts with "Welcome to Birdhouse" (not "Bird-house")

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2524/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2525/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov

[fmigneault]

run tests

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2531/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2533/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1553/

NOTEBOOK TEST RESULTS

    
[2024-03-13T14:52:10.274Z] ============================= test session starts ==============================
[2024-03-13T14:52:10.274Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-03-13T14:52:10.274Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-03-13T14:52:10.274Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-03-13T14:52:10.274Z] collected 264 items
[2024-03-13T14:52:10.274Z] 
[2024-03-13T14:52:20.959Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-03-13T14:52:55.716Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-03-13T14:53:00.706Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-03-13T14:53:10.501Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 17%]
[2024-03-13T14:53:20.214Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 19%]
[2024-03-13T14:53:31.317Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 22%]
[2024-03-13T15:01:13.459Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 27%]
[2024-03-13T15:01:13.459Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 27%]
[2024-03-13T15:01:16.473Z] ...............                                                          [ 33%]
[2024-03-13T15:01:26.206Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-03-13T15:01:32.942Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 37%]
[2024-03-13T15:01:49.946Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 39%]
[2024-03-13T15:01:55.232Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-03-13T15:01:59.921Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 44%]
[2024-03-13T15:05:20.601Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-03-13T15:06:37.512Z] .............                                                            [ 54%]
[2024-03-13T15:06:39.421Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-03-13T15:06:41.646Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-03-13T15:06:57.697Z] .................                                                        [ 65%]
[2024-03-13T15:07:06.045Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 68%]
[2024-03-13T15:07:07.430Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 68%]
[2024-03-13T15:07:25.525Z] .........                                                                [ 71%]
[2024-03-13T15:07:35.217Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-03-13T15:07:44.892Z] finch-master/docs/source/notebooks/finch-usage.ipynb ...F..              [ 78%]
[2024-03-13T15:07:46.278Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2024-03-13T15:07:49.660Z] ......                                                                   [ 81%]
[2024-03-13T15:07:56.249Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2024-03-13T15:08:12.713Z] .............                                                            [ 86%]
[2024-03-13T15:08:22.708Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2024-03-13T15:08:59.393Z] ....s.                                                                   [ 89%]
[2024-03-13T15:09:07.525Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-03-13T15:09:21.665Z] ...                                                                      [ 90%]
[2024-03-13T15:09:33.893Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2024-03-13T15:09:55.516Z] ......                                                                   [ 93%]
[2024-03-13T15:09:58.250Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-03-13T15:12:38.381Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-03-13T15:12:38.381Z] 
[2024-03-13T15:12:38.382Z] =================================== FAILURES ===================================
    
  

[fmigneault]

@mishaschwartz The THREDDS failing notebooks in https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-1994632197 is related to failing Magpie auth. This auth happens to use PAVICS_HOST variable, so I'm not sure if this could be caused by the rename/backward-compat handling of this PR.

11:12:38  # Open session
11:12:38  with requests.Session() as session:
11:12:38      session.auth = MagpieAuth(f"https://{pavics_host}/magpie", AUTH_USR, AUTH_PWD)
11:12:38      # Open a PyDAP data store and pass it to xarray
11:12:38      store = xr.backends.PydapDataStore.open(SECURED_URL, session=session)
11:12:38      ds = xr.open_dataset(
11:12:38          store, decode_cf=False
11:12:38      )  # Attributes are problematic with this file.
11:12:38  ds
11:12:38  
11:12:38  Traceback:
11:12:38  
11:12:38  ---------------------------------------------------------------------------
11:12:38  HTTPError                                 Traceback (most recent call last)
11:12:38  File /opt/conda/envs/birdy/lib/python3.9/site-packages/requests_magpie.py:69, in MagpieAuth.__call__(self, request)
11:12:38       68 try:
11:12:38  ---> 69     response.raise_for_status()
11:12:38       70 except RequestException as e:
11:12:38  
11:12:38  File /opt/conda/envs/birdy/lib/python3.9/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
11:12:38     1020 if http_error_msg:
11:12:38  -> 1021     raise HTTPError(http_error_msg, response=self)
11:12:38  
11:12:38  HTTPError: 401 Client Error: Unauthorized for url: https://host-140-154.rdext.crim.ca/magpie/signin
11:12:38

[mishaschwartz]

The THREDDS failing notebooks in https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-1994632197 is related to failing Magpie auth.

Thanks, I'll check it out

EDIT

I don't think that this is the problem. The url from the logs looks right:

11:12:38  HTTPError: 401 Client Error: Unauthorized for url: https://host-140-154.rdext.crim.ca/magpie/signin

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2534/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2536/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2537/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1555/

NOTEBOOK TEST RESULTS

    
[2024-03-14T14:43:58.767Z] ============================= test session starts ==============================
[2024-03-14T14:43:58.767Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-03-14T14:43:58.767Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-03-14T14:43:58.768Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-03-14T14:43:58.768Z] collected 264 items
[2024-03-14T14:43:58.768Z] 
[2024-03-14T14:44:10.000Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-03-14T14:46:22.139Z] notebooks-auth/test_cowbird_jupyter.ipynb ......FFF.                     [ 10%]
[2024-03-14T14:46:26.665Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-03-14T14:46:35.316Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 17%]
[2024-03-14T14:46:45.297Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 19%]
[2024-03-14T14:46:56.591Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 22%]
[2024-03-14T14:54:44.636Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 27%]
[2024-03-14T14:54:44.636Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 27%]
[2024-03-14T14:54:51.485Z] ...............                                                          [ 33%]
[2024-03-14T14:55:00.888Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-03-14T14:55:09.057Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 37%]
[2024-03-14T14:55:26.664Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 39%]
[2024-03-14T14:55:32.489Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-03-14T14:55:37.176Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 44%]
[2024-03-14T14:58:49.729Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-03-14T15:00:02.790Z] .............                                                            [ 54%]
[2024-03-14T15:00:04.431Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-03-14T15:00:06.958Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-03-14T15:00:27.020Z] .................                                                        [ 65%]
[2024-03-14T15:00:34.768Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 68%]
[2024-03-14T15:00:36.165Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 68%]
[2024-03-14T15:00:37.527Z] .FFFFFFFF                                                                [ 71%]
[2024-03-14T15:00:49.147Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-03-14T15:00:57.978Z] finch-master/docs/source/notebooks/finch-usage.ipynb ...F..              [ 78%]
[2024-03-14T15:00:59.361Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2024-03-14T15:01:02.763Z] ......                                                                   [ 81%]
[2024-03-14T15:01:10.905Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2024-03-14T15:01:26.828Z] .............                                                            [ 86%]
[2024-03-14T15:01:36.831Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2024-03-14T15:02:19.789Z] ....s.                                                                   [ 89%]
[2024-03-14T15:02:27.927Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-03-14T15:02:43.452Z] ...                                                                      [ 90%]
[2024-03-14T15:02:55.770Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2024-03-14T15:03:21.050Z] ......                                                                   [ 93%]
[2024-03-14T15:03:23.908Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-03-14T15:05:58.095Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-03-14T15:05:58.095Z] 
[2024-03-14T15:05:58.095Z] =================================== FAILURES ===================================
    
  

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2538/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2540/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[mishaschwartz]

@fmigneault It looks like tests are failing because the PAVICS_IMAGE and PAVICS_VERSION variables are not set in the env.local file that is being used to run tests on Jenkins. This means that the DOCKER_NOTEBOOK_IMAGES variable is not set properly (doesn't have enough values and so an error is raised when it is parsed later on).

Do you mind checking it out?

See for example from the console output when setting up the stack on jenkins:

+ export PAVICS_IMAGE=
+ export EO_IMAGE=pavics/crim-jupyter-eo:0.3.0
+ export NLP_IMAGE=pavics/crim-jupyter-nlp:0.4.0
+ echo
+ cut -d : -f2
+ export PAVICS_VERSION=
+ echo pavics/crim-jupyter-eo:0.3.0
+ cut -d : -f2
+ export EO_VERSION=0.3.0
+ echo pavics/crim-jupyter-nlp:0.4.0
+ cut -d : -f2
+ export NLP_VERSION=0.4.0
+ export DOCKER_NOTEBOOK_IMAGES=                                pavics/crim-jupyter-eo:0.3.0                                pavics/crim-jupyter-nlp:0.4.0
+ export JUPYTERHUB_IMAGE_SELECTION_NAMES=pavics-workflow-test:                                          eo-crim:0.3.0                                          nlp-crim:0.4.0

Note that DOCKER_NOTEBOOK_IMAGES only has two entries.

And then from the build logs:

jupyterhub                   |     Traceback (most recent call last):
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/app.py", line 3380, in launch_instance_async
jupyterhub                   |         await self.start()
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/app.py", line 3107, in start
jupyterhub                   |         self.subapp.start()
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/app.py", line 235, in start
jupyterhub                   |         hub.load_config_file(hub.config_file)
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 113, in inner
jupyterhub                   |         return method(app, *args, **kwargs)
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 950, in load_config_file
jupyterhub                   |         for (config, fname) in self._load_config_files(
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 909, in _load_config_files
jupyterhub                   |         config = loader.load_config()
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/traitlets/config/loader.py", line 626, in load_config
jupyterhub                   |         self._read_file_as_dict()
jupyterhub                   |       File "/usr/local/lib/python3.10/dist-packages/traitlets/config/loader.py", line 659, in _read_file_as_dict
jupyterhub                   |         exec(compile(f.read(), conf_filename, "exec"), namespace, namespace)  # noqa
jupyterhub                   |       File "/srv/jupyterhub/jupyterhub_config.py", line 180, in <module>
jupyterhub                   |         c.DockerSpawner.image_whitelist = {os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[0]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[0], os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[1]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[1], os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[2]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[2], 'jupyter/scipy-notebook': 'jupyter/scipy-notebook', 'jupyter/r-notebook': 'jupyter/r-notebook', 'jupyter/tensorflow-notebook': 'jupyter/tensorflow-notebook', 'jupyter/datascience-notebook': 'jupyter/datascience-notebook', 'jupyter/pyspark-notebook': 'jupyter/pyspark-notebook', 'jupyter/all-spark-notebook': 'jupyter/all-spark-notebook', }    # noqa
jupyterhub                   |     IndexError: list index out of range

UPDATE

Though it wasn't set properly here either and the tests still ran: https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-1994632197

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2541/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2545/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2547/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[fmigneault]

@mishaschwartz

It looks like tests are failing because the PAVICS_IMAGE and PAVICS_VERSION variables are not set in the env.local file that is being used to run tests on Jenkins. https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-2000304395

I'm worried about was the script is doing if that is the case. PAVICS_IMAGE and PAVICS_VERSION are not variables defined in birdhouse-deploy. The birdhouse-compose script should not temper with those. The definitions are as follows for the CI:

export PAVICS_IMAGE="$(echo ${DOCKER_NOTEBOOK_IMAGES} | grep pavics/workflow-tests | xargs)"
export PAVICS_VERSION=$(echo ${PAVICS_IMAGE} | cut -d ':' -f2)

Therefore, these variables should not be impacted by the current PR.

I can dig deeper to see what is happening.

[fmigneault]

@mishaschwartz Found the problem. Since JUPYTERHUB_ was added as prefix to DOCKER_NOTEBOOK_IMAGES, the variables that depend on it cannot be resolved anymore. Backward compatible names should be handled for this.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2548/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1558/

NOTEBOOK TEST RESULTS

    
[2024-03-19T22:37:27.182Z] ============================= test session starts ==============================
[2024-03-19T22:37:27.182Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-03-19T22:37:27.182Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-03-19T22:37:27.182Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-03-19T22:37:27.182Z] collected 270 items
[2024-03-19T22:37:27.182Z] 
[2024-03-19T22:37:38.184Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-03-19T22:37:52.381Z] notebooks-auth/test_cowbird_jupyter.ipynb ......FFF.                     [ 10%]
[2024-03-19T22:38:01.141Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-03-19T22:38:50.335Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ..FF..          [ 16%]
[2024-03-19T22:38:59.699Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-03-19T22:39:09.409Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-03-19T22:39:22.119Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-03-19T22:47:03.850Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-03-19T22:47:03.850Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-03-19T22:47:04.949Z] ...............                                                          [ 34%]
[2024-03-19T22:47:13.992Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-03-19T22:47:21.297Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-03-19T22:47:37.533Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-03-19T22:47:42.559Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2024-03-19T22:47:47.065Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 45%]
[2024-03-19T22:51:14.280Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-03-19T22:52:28.180Z] .............                                                            [ 55%]
[2024-03-19T22:52:32.655Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-03-19T22:52:35.000Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-03-19T22:52:52.114Z] .................                                                        [ 66%]
[2024-03-19T22:52:59.279Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 68%]
[2024-03-19T22:53:00.668Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-03-19T22:53:02.032Z] .FFFFFFFF                                                                [ 72%]
[2024-03-19T22:53:13.155Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-03-19T22:53:22.485Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 78%]
[2024-03-19T22:53:23.878Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-03-19T22:53:27.260Z] ......                                                                   [ 81%]
[2024-03-19T22:53:35.403Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2024-03-19T22:53:52.056Z] .............                                                            [ 86%]
[2024-03-19T22:54:02.083Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-03-19T22:54:48.992Z] ....s.                                                                   [ 89%]
[2024-03-19T22:54:57.201Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-03-19T22:55:09.725Z] ...                                                                      [ 90%]
[2024-03-19T22:55:24.649Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-03-19T22:55:47.977Z] ......                                                                   [ 93%]
[2024-03-19T22:55:49.330Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-03-19T22:58:29.217Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-03-19T22:58:29.217Z] 
[2024-03-19T22:58:29.217Z] =================================== FAILURES ===================================
    
  

[mishaschwartz]

@fmigneault

Since JUPYTERHUB_ was added as prefix to DOCKER_NOTEBOOK_IMAGES, the variables that depend on it cannot be resolved anymore. Backward compatible names should be handled for this.

These are backwards compatible in the current script. How is the initial DOCKER_NOTEBOOK_IMAGES being set? In other words, which repo contains the env.local file that is being used to run the instance for the jenkins tests?

Wait... do the CI tests just use a copy of env.local.example?

Also did you make some change to the jenkins configuration so that https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-2008305656 ran properly or is there further action required?

[fmigneault]

@mishaschwartz

DOCKER_NOTEBOOK_IMAGES is set using the value from https://github.com/bird-house/birdhouse-deploy/blob/8218166d5c8c7163293a656930ff85762eff4b60/birdhouse/components/jupyterhub/default.env#L11 to always use the latest test image, but is then filtered by the following to avoid any extra images that could be added later on

export PAVICS_IMAGE="$(echo ${DOCKER_NOTEBOOK_IMAGES} | grep pavics/workflow-tests | xargs)"
export PAVICS_VERSION=$(echo ${PAVICS_IMAGE} | cut -d ':' -f2)

The obtained value for PAVICS_IMAGE is then used to override DOCKER_NOTEBOOK_IMAGES with additional jupyter images from https://github.com/crim-ca/pavics-jupyter-images/.

The CI is not a copy of env.local.example. It enables most of the components to evaluate them.

I did not change anything on the CI. I will go check the logs. Not sure why this one worked suddenly.

[mishaschwartz]

@fmigneault

Where is this code from?

export PAVICS_IMAGE="$(echo ${DOCKER_NOTEBOOK_IMAGES} | grep pavics/workflow-tests | xargs)"
export PAVICS_VERSION=$(echo ${PAVICS_IMAGE} | cut -d ':' -f2)

It would be best to migrate that code to use the new versions or support both as a fallback:

export PAVICS_IMAGE="$(echo ${JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES} | grep pavics/workflow-tests | xargs)"

OR

export PAVICS_IMAGE="$(echo ${JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES:-${DOCKER_NOTEBOOK_IMAGES}} | grep pavics/workflow-tests | xargs)"

[fmigneault]

The code is only in the env.local of the CI instance. I can update it to patch the CI using ${JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES:-${DOCKER_NOTEBOOK_IMAGES}}, but I am concerned that anyone that was using DOCKER_NOTEBOOK_IMAGES before will have their instance suddenly breaking.

[mishaschwartz]

@fmigneault

If they're using it in a setting defined by JUPYTERHUB_CONFIG_OVERRIDE or JUPYTERHUB_ENABLE_MULTI_NOTEBOOKS then it's handled by this component:

https://github.com/bird-house/birdhouse-deploy/pull/428/files#diff-f6b072b3a5d3b39f2917942d75cd999a28407ad287df52a92a4d22487ff91b22

If they're doing what you're doing in the CI env.local file then the env.local file will need to be migrated. You could also simply add:

export DOCKER_NOTEBOOK_IMAGES="${JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES}"

above that line.

The only other option is to have the new variable names set the old variable names in backwards compatible mode. To solve this specific issue though it would have to do so before env.local is read.

[fmigneault]

The only other option is to have the new variable names set the old variable names in backwards compatible mode. To solve this specific issue though it would have to do so before env.local is read.

Doesn't the strategy to convert PAVICS_ variables into BIRDHOUSE_ variables handle the case where variables are overridden by env.local using DELAYED_EVAL? Couldn't that be applied as well?

[fmigneault]

Looking at the output logs of https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-2008305656, I see that https://github.com/bird-house/birdhouse-deploy/blob/pavics-to-birdhouse/birdhouse/scripts/create-magpie-users fails with an issue about MAGPIE_SERVER_URL="https://$BIRDHOUSE_FQDN/magpie" resolution.

The command looks like this:

+ docker run --rm --name create_magpie_users -v /tmp/create-magpie-authtest-user.yml:/tmp/create-magpie-authtest-user.yml:ro -v /tmp/create_magpie_users:/tmp/create_magpie_users:rw pavics/magpie:3.38.0 magpie_cli batch_update_users https:///magpie [...]

[mishaschwartz]

@fmigneault

Doesn't the strategy to convert PAVICS variables into BIRDHOUSE variables handle the case where variables are overridden by env.local using DELAYED_EVAL? Couldn't that be applied as well?

Yes, but in your case the old variable name (DOCKER_NOTEBOOK_IMAGES) needs to be present when env.local is parsed. This is before the variable conversion occurs.

I think I have a simple enough solution to this though ... stay tuned for a new commit shortly.

[mishaschwartz]

@fmigneault

I see that https://github.com/bird-house/birdhouse-deploy/blob/pavics-to-birdhouse/birdhouse/scripts/create-magpie-users fails with an issue about MAGPIE_SERVER_URL="https://$BIRDHOUSE_FQDN/magpie" resolution.

So the issue here is that the PAVICS_FQDN variable will only be used to override the BIRDHOUSE_FQDN variable if BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED is set to True.

Since this script runs independently of pavics-compose.sh, that variable is not set when this script is run.

I think you'll have to set BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED=True in the jenkins config for this to work for now until you can update the env.local file in jenkins.

[fmigneault]

Should BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED be set to True by default? I think that would ease the transition until the instance maintainer decides to disable it explicitly after porting their variables.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2552/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[tlvu]

Sorry I have not looked at this PR last week. Will look at it next week.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2557/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[mishaschwartz]

@fmigneault

The DOCKER_NOTEBOOK_IMAGES variable is still not being set properly in the most recent tests (https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-2018009140) for the reasons you described here (https://github.com/bird-house/birdhouse-deploy/pull/428#issuecomment-2009669287)

However, this should have been resolved because now the old DOCKER_NOTEBOOK_IMAGES variable was set properly in 7915189 and backwards compatible mode is now the default as of fd83d30.

Maybe the issue is with how jenkins is reading from the env.local file? I see in the logs that at one point it is sourcing the env.local file directly instead of reading it with read-configs.includ.sh:

[Pipeline] echo
[INFO] Loading daccs-configs 'test' environment definitions.
[Pipeline] sh
+ . /home/jenkins/agent/workspace/...../birdhouse-deploy/env.local

If the env.local file is sourced directly, how is it using the default value DOCKER_NOTEBOOK_IMAGES? Are the default values added at a previous step?

If the environment variables are not loaded through read-configs.include.sh then the fix that I implemented in 7915189 will not apply.

Just as a side note, this might be a lot easier for me to debug if I could get access to the DACCS-iac-birdhouse repo please.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2558/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[fmigneault]

@mishaschwartz

The creation of the instances is tricky due to how floating IPs are assigned to OpenStack instances from the available pool. The source local.env that you see is only on the Jenkins side for loading definitions such as TEST_PAVICS_SDI_WEAVER such that everything can be defined once in the same file reused by the instance. DOCKER_NOTEBOOK_IMAGES is ignored on that side. The instance itself that gets spawned on OpenStack uses a unix service that calls pavics-compose.sh with all the underlying sourcing strategies it normally defines with read-configs.include.sh. I believe that explicit call to pavics-compose.sh might be the cause since the file does not exist anymore. I will have to try adding a symlink or something to see if it resolves.

Sadly, I cannot give access directly to this repo because it includes keys added by our TI team for security.

[mishaschwartz]

I believe that explicit call to pavics-compose.sh might be the cause since the file does not exist anymore. I will have to try adding a symlink or something to see if it resolves.

That file still exists. It was left for backwards compatibility. So it's probably not that

I guess I'll leave debugging the testing issue to you then since you have the better visibility of what's going on :shrug:

[fmigneault]

I believe that explicit call to pavics-compose.sh might be the cause since the file does not exist anymore. I will have to try adding a symlink or something to see if it resolves.

That file still exists. It was left for backwards compatibility. So it's probably not that

You're right! Got mixed up from a previous revision of this PR. Guess I'll have to spawn an instance to debug what is happening... Sorry for the delay that causes.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2562/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://host-140-118.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[fmigneault]

@mishaschwartz

Here is the output from a manually started test instance, which uses this call to start the stack:

> pavics-compose.sh up -d > /home/ubuntu/last-pavics-compose-start.log 2>&1

> cat last-pavics-compose-start.log 
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
INFO:     Resolved docker-compose directory: [/home/ubuntu/birdhouse-deploy/birdhouse]
WARNING:  This file [/home/ubuntu/birdhouse-deploy/birdhouse/birdhouse-compose.sh] is being executed through a non-supported interface for the Birdhouse software. This file may be moved or updated without warning.
WARNING:  The BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED variable is being set to 'True' by default. To avoid this behaviour set this variable or execute this file through a supported interface.
INFO:     Using local environment file at: /home/ubuntu/birdhouse-deploy/birdhouse/env.local
WARNING:  Deprecated variable [EXTRA_CONF_DIRS] is overriding [BIRDHOUSE_EXTRA_CONF_DIRS]. Check env.local file.
WARNING:  Variable [POSTGRES_PAVICS_USERNAME] employs a deprecated default value recommended for override. Check env.local file.
WARNING:  './config/proxy' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/magpie' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/twitcher' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/canarie-api' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/geoserver' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/finch' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/raven' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/hummingbird' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/thredds' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/portainer' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/jupyterhub' in BIRDHOUSE_EXTRA_CONF_DIRS does not exist
WARNING:  './config/cowbird' in a dependency of ./optional-components/test-cowbird-jupyter-access does not exist
INFO:     Using local environment file at: /home/ubuntu/birdhouse-deploy/birdhouse/env.local
WARNING:  Deprecated variable [PAVICS_FQDN] is overriding [BIRDHOUSE_FQDN]. Check env.local file.
WARNING:  Deprecated variable [POSTGRES_PAVICS_USERNAME] is overriding [BIRDHOUSE_POSTGRES_USERNAME]. Check env.local file.
WARNING:  Deprecated variable [POSTGRES_PAVICS_PASSWORD] is overriding [BIRDHOUSE_POSTGRES_PASSWORD]. Check env.local file.
WARNING:  Deprecated variable [SSL_CERTIFICATE] is overriding [BIRDHOUSE_SSL_CERTIFICATE]. Check env.local file.
WARNING:  Deprecated variable [DOC_URL] is overriding [BIRDHOUSE_DOC_URL]. Check env.local file.
WARNING:  Deprecated variable [SUPPORT_EMAIL] is overriding [BIRDHOUSE_SUPPORT_EMAIL]. Check env.local file.
WARNING:  Deprecated variable [EXTRA_CONF_DIRS] is overriding [BIRDHOUSE_EXTRA_CONF_DIRS]. Check env.local file.
WARNING:  Deprecated variable [AUTODEPLOY_NOTEBOOK_FREQUENCY] is overriding [BIRDHOUSE_AUTODEPLOY_NOTEBOOK_FREQUENCY]. Check env.local file.
WARNING:  Deprecated variable [ALLOW_UNSECURE_HTTP] is overriding [BIRDHOUSE_ALLOW_UNSECURE_HTTP]. Check env.local file.
WARNING:  Deprecated variable [DOCKER_NOTEBOOK_IMAGES] is overriding [JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES]. Check env.local file.
WARNING:  Deprecated variable [ENABLE_JUPYTERHUB_MULTI_NOTEBOOKS] is overriding [JUPYTERHUB_ENABLE_MULTI_NOTEBOOKS]. Check env.local file.
WARNING:  Deprecated variable [VERIFY_SSL] is overriding [BIRDHOUSE_VERIFY_SSL]. Check env.local file.
WARNING:  Deprecated variable [SMTP_SERVER] is overriding [ALERTMANAGER_SMTP_SERVER]. Check env.local file.
WARNING:  Deprecated variable [SERVER_NAME] is overriding [BIRDHOUSE_NAME]. Check env.local file.
WARNING:  Deprecated variable [SERVER_DESCRIPTION] is overriding [BIRDHOUSE_DESCRIPTION]. Check env.local file.
WARNING:  Deprecated variable [SERVER_INSTITUTION] is overriding [BIRDHOUSE_INSTITUTION]. Check env.local file.
WARNING:  Deprecated variable [SERVER_SUBJECT] is overriding [BIRDHOUSE_SUBJECT]. Check env.local file.
WARNING:  Deprecated variable [SERVER_TAGS] is overriding [BIRDHOUSE_TAGS]. Check env.local file.
WARNING:  Variable [POSTGRES_PAVICS_USERNAME] employs a deprecated default value recommended for override. Check env.local file.
WARNING:  Required variable [MAGPIE_SECRET] employs a default recommended for override. The security of your deployment may be compromised unless it is changed. Check env.local file.
WARNING:  Required variable [MAGPIE_ADMIN_USERNAME] employs a default recommended for override. The security of your deployment may be compromised unless it is changed. Check env.local file.
WARNING:  Required variable [MAGPIE_ADMIN_PASSWORD] employs a default recommended for override. The security of your deployment may be compromised unless it is changed. Check env.local file.
WARNING:  Required variable [POSTGRES_MAGPIE_USERNAME] employs a default recommended for override. The security of your deployment may be compromised unless it is changed. Check env.local file.
WARNING:  Required variable [POSTGRES_MAGPIE_PASSWORD] employs a default recommended for override. The security of your deployment may be compromised unless it is changed. Check env.local file.
WARNING:  Required variable [GEOSERVER_ADMIN_PASSWORD] employs a default recommended for override. The security of your deployment may be compromised unless it is changed. Check env.local file.
WARNING:  Required variable [BIRDHOUSE_POSTGRES_PASSWORD] employs a default recommended for override. The security of your deployment may be compromised unless it is changed. Check env.local file.
WARNING:  Optional variable [BIRDHOUSE_DOCUMENTATION_URL] employs a default recommended for override. Check env.local file.
WARNING:  Optional variable [BIRDHOUSE_RELEASE_NOTES_URL] employs a default recommended for override. Check env.local file.
WARNING:  Optional variable [BIRDHOUSE_SUPPORT_URL] employs a default recommended for override. Check env.local file.
WARNING:  Optional variable [BIRDHOUSE_LICENSE_URL] employs a default recommended for override. Check env.local file.
WARNING:  Optional variable [RAVEN_GEO_URL] employs a default recommended for override. Check env.local file.
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
INFO:     Resolved docker-compose directory: [/home/ubuntu/birdhouse-deploy/birdhouse]
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
INFO:     Resolved docker-compose directory: [/home/ubuntu/birdhouse-deploy/birdhouse]
WARNING:  No services in BIRDHOUSE_DEFAULT_CONF_DIRS and BIRDHOUSE_EXTRA_CONF_DIRS. SERVICES JSON list will be empty!
INFO:     Found compose directory [/home/ubuntu/birdhouse-deploy/birdhouse]
INFO:     Moving to [/home/ubuntu/birdhouse-deploy/birdhouse]
INFO:     Displaying resolved compose configurations:
COMPOSE_CONF_LIST=
docker-compose.yml
./components/proxy/docker-compose-extra.yml
./components/magpie/docker-compose-extra.yml
./components/magpie/config/proxy/docker-compose-extra.yml
./components/twitcher/docker-compose-extra.yml
./components/twitcher/config/proxy/docker-compose-extra.yml
./components/wps_outputs-volume/docker-compose-extra.yml
./components/wps_outputs-volume/config/proxy/docker-compose-extra.yml
./components/cowbird/docker-compose-extra.yml
./components/cowbird/config/magpie/docker-compose-extra.yml
./components/cowbird/config/proxy/docker-compose-extra.yml
./components/stac/docker-compose-extra.yml
./components/stac/config/magpie/docker-compose-extra.yml
./components/stac/config/proxy/docker-compose-extra.yml
./components/stac/config/twitcher/docker-compose-extra.yml
./components/magpie/config/canarie-api/docker-compose-extra.yml
./components/twitcher/config/canarie-api/docker-compose-extra.yml
./components/wps_outputs-volume/config/canarie-api/docker-compose-extra.yml
./components/cowbird/config/canarie-api/docker-compose-extra.yml
./components/stac/config/canarie-api/docker-compose-extra.yml
./components/cowbird/config/geoserver/docker-compose-extra.yml
./components/cowbird/config/jupyterhub/docker-compose-extra.yml
./components/magpie/config/canarie-api/docker-compose-extra.yml
./components/twitcher/config/canarie-api/docker-compose-extra.yml
./components/wps_outputs-volume/config/canarie-api/docker-compose-extra.yml
./components/cowbird/config/canarie-api/docker-compose-extra.yml
./components/stac/config/canarie-api/docker-compose-extra.yml
./components/canarie-api/config/proxy/docker-compose-extra.yml
./components/geoserver/docker-compose-extra.yml
./components/cowbird/config/geoserver/docker-compose-extra.yml
./components/geoserver/config/canarie-api/docker-compose-extra.yml
./components/geoserver/config/magpie/docker-compose-extra.yml
./components/geoserver/config/proxy/docker-compose-extra.yml
./components/postgres/docker-compose-extra.yml
./components/finch/docker-compose-extra.yml
./components/finch/config/canarie-api/docker-compose-extra.yml
./components/finch/config/magpie/docker-compose-extra.yml
./components/finch/config/wps_outputs-volume/docker-compose-extra.yml
./components/raven/docker-compose-extra.yml
./components/raven/config/canarie-api/docker-compose-extra.yml
./components/raven/config/magpie/docker-compose-extra.yml
./components/raven/config/wps_outputs-volume/docker-compose-extra.yml
./components/data-volume/docker-compose-extra.yml
./components/hummingbird/docker-compose-extra.yml
./components/hummingbird/config/canarie-api/docker-compose-extra.yml
./components/hummingbird/config/data-volume/docker-compose-extra.yml
./components/hummingbird/config/magpie/docker-compose-extra.yml
./components/hummingbird/config/wps_outputs-volume/docker-compose-extra.yml
./components/thredds/docker-compose-extra.yml
./components/thredds/config/canarie-api/docker-compose-extra.yml
./components/thredds/config/magpie/docker-compose-extra.yml
./components/thredds/config/proxy/docker-compose-extra.yml
./components/portainer/docker-compose-extra.yml
./components/portainer/config/proxy/docker-compose-extra.yml
./components/jupyterhub/docker-compose-extra.yml
./components/cowbird/config/jupyterhub/docker-compose-extra.yml
./components/jupyterhub/config/canarie-api/docker-compose-extra.yml
./components/jupyterhub/config/magpie/docker-compose-extra.yml
./components/jupyterhub/config/proxy/docker-compose-extra.yml
./components/monitoring/docker-compose-extra.yml
./components/monitoring/config/magpie/docker-compose-extra.yml
./components/monitoring/config/proxy/docker-compose-extra.yml
./components/weaver/docker-compose-extra.yml
./components/weaver/config/canarie-api/docker-compose-extra.yml
./components/weaver/config/magpie/docker-compose-extra.yml
./components/weaver/config/proxy/docker-compose-extra.yml
./components/weaver/config/twitcher/docker-compose-extra.yml
./optional-components/canarie-api-full-monitoring/config/canarie-api/docker-compose-extra.yml
./optional-components/canarie-api-full-monitoring/config/cowbird/docker-compose-extra.yml
./optional-components/canarie-api-full-monitoring/config/finch/docker-compose-extra.yml
./optional-components/canarie-api-full-monitoring/config/hummingbird/docker-compose-extra.yml
./optional-components/canarie-api-full-monitoring/config/raven/docker-compose-extra.yml
./optional-components/canarie-api-full-monitoring/config/thredds/docker-compose-extra.yml
./optional-components/canarie-api-full-monitoring/config/weaver/docker-compose-extra.yml
./optional-components/all-public-access/config/finch/docker-compose-extra.yml
./optional-components/all-public-access/config/geoserver/docker-compose-extra.yml
./optional-components/all-public-access/config/hummingbird/docker-compose-extra.yml
./optional-components/all-public-access/config/jupyterhub/docker-compose-extra.yml
./optional-components/all-public-access/config/raven/docker-compose-extra.yml
./optional-components/all-public-access/config/thredds/docker-compose-extra.yml
./optional-components/testthredds/docker-compose-extra.yml
./optional-components/testthredds/config/canarie-api/docker-compose-extra.yml
./optional-components/testthredds/config/proxy/docker-compose-extra.yml
./optional-components/secure-thredds/config/magpie/docker-compose-extra.yml
./optional-components/all-public-access/config/secure-data-proxy/docker-compose-extra.yml
./optional-components/secure-data-proxy/config/magpie/docker-compose-extra.yml
./optional-components/secure-data-proxy/config/proxy/docker-compose-extra.yml
./optional-components/stac-data-proxy/config/proxy/docker-compose-extra.yml
./optional-components/stac-data-proxy/config/secure-data-proxy/docker-compose-extra.yml
./optional-components/stac-populator/docker-compose-extra.yml
./optional-components/wps-healthchecks/config/finch/docker-compose-extra.yml
./optional-components/wps-healthchecks/config/hummingbird/docker-compose-extra.yml
./optional-components/wps-healthchecks/config/raven/docker-compose-extra.yml
./optional-components/database-external-ports/config/magpie/docker-compose-extra.yml
./optional-components/database-external-ports/config/postgres/docker-compose-extra.yml
./optional-components/test-weaver/docker-compose-extra.yml
./optional-components/test-geoserver-secured-access/docker-compose-extra.yml
./optional-components/test-cowbird-jupyter-access/docker-compose-extra.yml
./optional-components/x-robots-tag-header/config/proxy/docker-compose-extra.yml
./optional-components/backwards-compatible-overrides/config/jupyterhub/docker-compose-extra.yml
INFO:     Executing './components/proxy/pre-docker-compose-up'
INFO:     Executing './components/geoserver/pre-docker-compose-up'
INFO:     Executing './components/thredds/pre-docker-compose-up'
thredds_persistence
INFO:     Executing './components/jupyterhub/pre-docker-compose-up'
Error response from daemon: network with name jupyterhub_network already exists
jupyterhub_data_persistence
INFO:     Executing './components/monitoring/pre-docker-compose-up'
prometheus_persistence
grafana_persistence
alertmanager_persistence
INFO:     Executing docker-compose with extra options: up -d 
time="2024-03-26T22:02:50Z" level=warning msg="volume thredds_persistence: volume.external.name is deprecated in favor of volume.name"
time="2024-03-26T22:02:50Z" level=warning msg="network jupyterhub_network: network.external.name is deprecated. Please set network.name with external: true"
time="2024-03-26T22:02:50Z" level=warning msg="volume jupyterhub_data_persistence: volume.external.name is deprecated in favor of volume.name"
time="2024-03-26T22:02:50Z" level=warning msg="volume prometheus_persistence: volume.external.name is deprecated in favor of volume.name"
time="2024-03-26T22:02:50Z" level=warning msg="volume grafana_persistence: volume.external.name is deprecated in favor of volume.name"
time="2024-03-26T22:02:50Z" level=warning msg="volume alertmanager_persistence: volume.external.name is deprecated in favor of volume.name"
service "proxy" refers to undefined volume ${BIRDHOUSE_DATA_PERSIST_SHARED_ROOT}/wps_outputs/weaver: invalid compose project
ERROR:    docker-compose error, exit code 15

Is it possible that BIRDHOUSE_DATA_PERSIST_SHARED_ROOT is not backward-supported correctly, or that the value is read as literal ${BIRDHOUSE_DATA_PERSIST_SHARED_ROOT} in the compose? The env.local does not contain any DATA_PERSIST_ROOT, DATA_PERSIST_SHARED_ROOT, BIRDHOUSE_DATA_PERSIST_ROOT or similar. Everything should use defaults, which I would expect to be /data. The /data dir does exist, and seems to contain the expected geoserver, wps_outputs, magpie_persist and so on.

When I run pavics-compose ps on that node, it ends up with the same error:

service "proxy" refers to undefined volume ${BIRDHOUSE_DATA_PERSIST_SHARED_ROOT}/wps_outputs/weaver: invalid compose project
ERROR:    docker-compose error, exit code 15

If I run pavics-compose config (or ./birdhouse -b -e ../birdhouse/env.local compose config), I end up with error:

service "cowbird" refers to undefined network default: invalid compose project
ERROR:    docker-compose error, exit code 15

Oh, and let me know if you want me to investigate for more things. The instance is not deleted, so I can try patches.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2564/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[mishaschwartz]

@fmigneault

Ok I think that I've found the issue that caused that specific error and fixed it. The tests are still failing though. Do you mind pulling in the latest commit and trying again with your instance?

[fmigneault]

After pulling the branch, the previous error seems fixed. I will try launching another instance from scratch. For some reason, the one with the pulled branch fails during the weaver docker script while trying to get a response from WPS providers.

Self-note: run http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2566

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2566/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1564/

NOTEBOOK TEST RESULTS

    
[2024-03-28T01:20:06.546Z] ============================= test session starts ==============================
[2024-03-28T01:20:06.546Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-03-28T01:20:06.547Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-03-28T01:20:06.547Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-03-28T01:20:06.547Z] collected 270 items
[2024-03-28T01:20:06.547Z] 
[2024-03-28T01:20:17.261Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-03-28T01:20:32.091Z] notebooks-auth/test_cowbird_jupyter.ipynb ......FFF.                     [ 10%]
[2024-03-28T01:20:39.925Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-03-28T01:21:29.105Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-03-28T01:21:38.338Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb ..F..F.        [ 19%]
[2024-03-28T01:21:48.044Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-03-28T01:21:56.307Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-03-28T01:29:08.051Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-03-28T01:29:08.051Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-03-28T01:29:11.325Z] ...............                                                          [ 34%]
[2024-03-28T01:29:20.756Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-03-28T01:29:27.896Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-03-28T01:29:43.561Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-03-28T01:30:00.271Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2024-03-28T01:30:05.165Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-03-28T01:33:32.034Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-03-28T01:34:45.104Z] .............                                                            [ 55%]
[2024-03-28T01:34:47.016Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-03-28T01:34:49.243Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-03-28T01:35:06.151Z] .................                                                        [ 66%]
[2024-03-28T01:35:13.329Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 68%]
[2024-03-28T01:35:14.714Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-03-28T01:35:16.087Z] .FFFFFFFF                                                                [ 72%]
[2024-03-28T01:35:26.664Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-03-28T01:35:35.405Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 78%]
[2024-03-28T01:35:37.315Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-03-28T01:35:40.128Z] ......                                                                   [ 81%]
[2024-03-28T01:35:46.717Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2024-03-28T01:36:03.718Z] .............                                                            [ 86%]
[2024-03-28T01:36:13.729Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-03-28T01:37:06.638Z] ....s.                                                                   [ 89%]
[2024-03-28T01:37:13.241Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-03-28T01:37:28.799Z] ...                                                                      [ 90%]
[2024-03-28T01:37:41.047Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-03-28T01:38:05.556Z] ......                                                                   [ 93%]
[2024-03-28T01:38:06.655Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-03-28T01:40:46.271Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-03-28T01:40:46.271Z] 
[2024-03-28T01:40:46.271Z] =================================== FAILURES ===================================
    
  

[fmigneault]

One of the errors for Cowbird:

test-cowbird-jupyter-access  | Failed to connect to Magpie [HTTPSConnectionPool(host='host-140-216.rdext.crim.ca', port=443): Max retries exceeded with url: /magpie (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f419c785340>: Failed to establish a new connection: [Errno 111] Connection refused'))]. 
weaver-worker                | whether broker connection retries are made during startup in Celery 6.0 and above.
test-cowbird-jupyter-access  | Attempting again (2)...
test-cowbird-jupyter-access  | Magpie availability checked successfully.
hummingbird                  |     Updating '/opt/birdhouse/etc/gunicorn',
test-cowbird-jupyter-access  | Failed to connect to Cowbird []. 
test-cowbird-jupyter-access  | Attempting again (1)...
hummingbird                  |     mode 755, user 'root', group 'root'
test-cowbird-jupyter-access  | Cowbird availability checked successfully.
test-cowbird-jupyter-access  | Creating test user on Magpie...
test-cowbird-jupyter-access  | Failed to find user workspace at path [/data/user_workspaces/testcowbirdjupyter]. Attempting again (1)...
test-cowbird-jupyter-access  | User workspace successfully found at path `/data/user_workspaces/testcowbirdjupyter`.
test-cowbird-jupyter-access  | Copying shapefile data to user workspace...
test-cowbird-jupyter-access  | Creating test WPS outputs data...
test-cowbird-jupyter-access  | Secure-data-proxy service exists. Checking that the user has access to the wpsoutputs resource...
test-cowbird-jupyter-access  | User access to secure-data-proxy is allowed.
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (1)...
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (2)...
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (3)...
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (4)...
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (5)...
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (6)...
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (7)...
test-cowbird-jupyter-access  | Failed to find user hardlink at path [/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt]. Attempting again (8)...
test-cowbird-jupyter-access  |     raise RuntimeError(f"Failed to create `{expected_user_wps_outputs_hardlink}` hardlink in the user workspace.")
test-cowbird-jupyter-access  | RuntimeError: Failed to create `/data/user_workspaces/testcowbirdjupyter/wps_outputs/weaver/test_user_file.txt` hardlink in the user workspace.

Another error related to Cowbird:

cowbird-worker               | [2024-03-28 01:28:16,376: ERROR/ForkPoolWorker-8] Task cowbird.handlers.impl.geoserver.publish_shapefile[d0f3aea9-a920-4ac8-84c5-11cf1ee00703] raised unexpected: RequestException('Connection to Geoserver failed using [http://host-140-216.rdext.crim.ca:8765/geoserver]')
cowbird-worker               | Traceback (most recent call last):
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 203, in _new_conn
cowbird-worker               |     sock = connection.create_connection(
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
cowbird-worker               |     raise err
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
cowbird-worker               |     sock.connect(sa)
cowbird-worker               | ConnectionRefusedError: [Errno 111] Connection refused
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | The above exception was the direct cause of the following exception:
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | Traceback (most recent call last):
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
cowbird-worker               |     response = self._make_request(
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request
cowbird-worker               |     conn.request(
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 395, in request
cowbird-worker               |     self.endheaders()
cowbird-worker               |   File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
cowbird-worker               |     self._send_output(message_body, encode_chunked=encode_chunked)
cowbird-worker               |   File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
cowbird-worker               |     self.send(msg)
cowbird-worker               |   File "/usr/local/lib/python3.10/http/client.py", line 976, in send
cowbird-worker               |     self.connect()
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 243, in connect
cowbird-worker               |     self.sock = self._new_conn()
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 218, in _new_conn
cowbird-worker               |     raise NewConnectionError(
cowbird-worker               | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fb4835a4a00>: Failed to establish a new connection: [Errno 111] Connection refused
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | The above exception was the direct cause of the following exception:
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | Traceback (most recent call last):
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
cowbird-worker               |     resp = conn.urlopen(
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
cowbird-worker               |     retries = retries.increment(
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
cowbird-worker               |     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
cowbird-worker               | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='host-140-216.rdext.crim.ca', port=8765): Max retries exceeded with url: /geoserver/rest/workspaces/testcowbirdjupyter/datastores/shapefile_datastore_testcowbirdjupyter/featuretypes (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb4835a4a00>: Failed to establish a new connection: [Errno 111] Connection refused'))
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | During handling of the above exception, another exception occurred:
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | Traceback (most recent call last):
cowbird-worker               |   File "/opt/local/src/cowbird/cowbird/handlers/impl/geoserver.py", line 112, in wrapper
cowbird-worker               |     response = func(geoserver, **kwargs)  # type: ignore[arg-type,misc]  # since args are not named explicitly
cowbird-worker               |   File "/opt/local/src/cowbird/cowbird/handlers/impl/geoserver.py", line 868, in _publish_shapefile_request
cowbird-worker               |     response = requests.post(url=request_url, json=payload, auth=self.auth,
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 115, in post
cowbird-worker               |     return request("post", url, data=data, json=json, **kwargs)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
cowbird-worker               |     return session.request(method=method, url=url, **kwargs)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
cowbird-worker               |     resp = self.send(prep, **send_kwargs)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
cowbird-worker               |     r = adapter.send(request, **kwargs)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
cowbird-worker               |     raise ConnectionError(e, request=request)
cowbird-worker               | requests.exceptions.ConnectionError: HTTPConnectionPool(host='host-140-216.rdext.crim.ca', port=8765): Max retries exceeded with url: /geoserver/rest/workspaces/testcowbirdjupyter/datastores/shapefile_datastore_testcowbirdjupyter/featuretypes (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb4835a4a00>: Failed to establish a new connection: [Errno 111] Connection refused'))
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | During handling of the above exception, another exception occurred:
cowbird-worker               | 
cowbird-worker               | 
cowbird-worker               | Traceback (most recent call last):
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
cowbird-worker               |     R = retval = fun(*args, **kwargs)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
cowbird-worker               |     return self.run(*args, **kwargs)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/celery/app/autoretry.py", line 60, in run
cowbird-worker               |     ret = task.retry(exc=exc, **retry_kwargs)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 736, in retry
cowbird-worker               |     raise_with_context(exc)
cowbird-worker               |   File "/usr/local/lib/python3.10/site-packages/celery/app/autoretry.py", line 38, in run
cowbird-worker               |     return task._orig_run(*args, **kwargs)
cowbird-worker               |   File "/opt/local/src/cowbird/cowbird/handlers/impl/geoserver.py", line 921, in publish_shapefile
cowbird-worker               |     return Geoserver.get_instance().publish_shapefile(workspace_name, shapefile_name)
cowbird-worker               |   File "/opt/local/src/cowbird/cowbird/handlers/impl/geoserver.py", line 581, in publish_shapefile
cowbird-worker               |     self._publish_shapefile_request(workspace_name=workspace_name,
cowbird-worker               |   File "/opt/local/src/cowbird/cowbird/handlers/impl/geoserver.py", line 115, in wrapper
cowbird-worker               |     raise requests.RequestException(f"Connection to Geoserver failed using [{geoserver.url}]")
cowbird-worker               | requests.exceptions.RequestException: Connection to Geoserver failed using [http://host-140-216.rdext.crim.ca:8765/geoserver]

[mishaschwartz]

For some reason, the one with the pulled branch fails during the weaver docker script while trying to get a response from WPS providers.

This can happen if you're using the default postgres settings and you haven't updated your env.local file to explicitly request the old hardcoded values. This happens when you have previously created a postgres instance with the old settings and now you're trying to reconnect with the new settings.

For example, the default postgres db name was previously hardcoded to pavics, now it is set with the BIRDHOUSE_POSTGRES_DB variable and set to birdhouse by default.

I've made another commit that will set these sort of variables to their hardcoded defaults in backwards compatible mode (if they are not overridden in the env.local file).

Let me know if that helps.

One way to check if this is a problem is to look at the logs for one of the birds: docker logs -f finch and it should give an error message about not being able to connect to the database if that is the issue.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2567/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2568/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2569/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

[fmigneault]

For example, the default postgres db name was previously hardcoded to pavics, now it is set with the BIRDHOUSE_POSTGRES_DB variable and set to birdhouse by default.

I've made another commit that will set these sort of variables to their hardcoded defaults in backwards compatible mode (if they are not overridden in the env.local file).

That's a good fix for migrating instances, but shouldn't that be a non issue for CI instances recreated from scratch?