bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform

https://birdhouse-deploy.readthedocs.io/en/latest/

Apache License 2.0

4 stars 7 forks source link

Rename variables, constants and files from PAVICS to Birdhouse #428

Closed mishaschwartz closed 5 months ago

mishaschwartz commented 8 months ago

Overview

For historical reasons the name PAVICS was used in variable names, constants and filenames in this repo to refer to the software stack in general. This was because, for a long time, the PAVICS deployment of this stack was the only one that was being used in production. However, now that multiple deployments of this software exist in production (that are not named PAVICS), we remove unnecessary references to PAVICS in order to reduce confusion for maintainers and developers who may not be aware of the historical reasons for the PAVICS name.

This update makes the following changes:

The string PAVICS in environment variables, constant values, and file names have been changed to BIRDHOUSE (case has been preserved where possible).
- For example:
  - PAVICS_FQDN -> BIRDHOUSE_FQDN
  - pavics_compose.sh -> birdhouse_compose.sh
  - THREDDS_DATASET_LOCATION_ON_CONTAINER='/pavics-ncml' -> THREDDS_DATASET_LOCATION_ON_CONTAINER='/birdhouse-ncml'
Comment strings and documentation that refers to the software stack as PAVICS have been changed to use Birdhouse.
Recreated the pavics-compose.sh script that runs birdhouse-compose.sh in backwards compatible mode.
- Backwards compatible mode means that variables in env.local that contain the string PAVICS will be used to set the equivalent variable that contains BIRDHOUSE. For example, the PAVICS_FQDN variable set in the env.local file will be used to set the value of BIRDHOUSE_FQDN.
Create a new entrypoint in bin/birdhouse that can be used to invoke pavics-compose.sh or birdhouse-compose.sh from one convenient location. This script also includes some useful options and provides a generic entrypoint to the stack that can be extended in the future.
Removed unused variables:
- CMIP5_THREDDS_ROOT

Migration Guide

Update env.local file to replace all variables that contain PAVICS with BIRDHOUSE. Variable names have also been updated to ensure that they start with the prefix BIRDHOUSE_.
- see env.local.example to see new variable names
- see the BACKWARDS_COMPATIBLE_VARIABLES variable (defined in default.env) for a full list of changed environment variable names.
Update any external scripts that access the old variable names directly to use the updated variable names.

Update any external scripts that access any of the following files to use the new file name:

old file name	new file name
pavics-compose.sh	birdhouse-compose.sh
PAVICS-deploy.logrotate	birdhouse-deploy.logrotate
configure-pavics.sh	configure-birdhouse.sh
trigger-pavicscrawler	trigger-birdhousecrawler

The following default values have changed. If your deployment was using the old default value, update your env.local file to explicitly set the old default values.

old variable name	new variable name	old default value	new default value
POSTGRES_PAVICS_USERNAME	BIRDHOUSE_POSTGRES_USERNAME	postgres-pavics	postgres-birdhouse
THREDDS_DATASET_LOCATION_ON_CONTAINER	(no change)	/pavics-ncml	/birdhouse-datasets
THREDDS_SERVICE_DATA_LOCATION_ON_CONTAINER	(no change)	/pavics-data	/birdhouse-service-data
THREDDS_DATASET_LOCATION_ON_HOST	(no change)	'${DATA_PERSIST_ROOT}/ncml'	'${BIRDHOUSE_DATA_PERSIST_ROOT}/thredds-datasets'
THREDDS_SERVICE_DATA_LOCATION_ON_HOST	(no change)	'${DATA_PERSIST_ROOT}/datasets'	'${BIRDHOUSE_DATA_PERSIST_ROOT}/thredds-service-data'
(hardcoded)	BIRDHOUSE_POSTGRES_DB	pavics	birdhouse
PAVICS_LOG_DIR	BIRDHOUSE_LOG_DIR	/var/log/PAVICS	/var/log/birdhouse
(hardcoded)	GRAFANA_DEFAULT_PROVIDER_FOLDER	Local-PAVICS	Local-Birdhouse
(hardcoded)	GRAFANA_DEFAULT_PROVIDER_FOLDER_UUID	local-pavics	local-birdhouse
(hardcoded)	GRAFANA_PROMETHEUS_DATASOURCE_UUID	local_pavics_prometheus	local_birdhouse_prometheus

Update any jupyter notebooks that make use of the PAVICS_HOST_URL environment variable to use the new BIRDHOUSE_HOST_URL instead.
Set the BIRDHOUSE_POSTGRES_DB variable to pavics in the env.local file. This value was previously hardcoded to the string pavics so to maintain backwards compatibility with any existing databases this should be kept the same. If you do want to update to the new database name, you will need to rename the existing database. For example, the following will update the existing database named pavics to birdhouse (assuming the old default values for the postgres username):
```
docker exec -it postgres psql -U postgres-pavics -d postgres -c 'ALTER DATABASE pavics RENAME TO birdhouse'
```
You can then update the env.local file to the new variable name and restart the stack
Set the BIRDHOUSE_POSTGRES_USER variable to postgres-pavics in the env.local file if you would like to preserve the old default value. If you would like to change the value of BIRDHOUSE_POSTGRES_USER then also update the name for any running postgres instances. For example, the following will update the user named postgres-pavics to postgres-birdhouse:
```
docker exec -it postgres psql -U postgres-pavics -d postgres -c 'CREATE USER "tmpsuperuser" WITH SUPERUSER'
docker exec -it postgres psql -U tmpsuperuser -d postgres -c 'ALTER ROLE "postgres-pavics" RENAME TO "postgres-birdhouse"'
docker exec -it postgres psql -U tmpsuperuser -d postgres -c 'ALTER ROLE "postgres-birdhouse" WITH PASSWORD '\''postgres-qwerty'\'
docker exec -it postgres psql -U postgres-birdhouse -d postgres -c 'DROP ROLE "tmpsuperuser"'
```
Note that the postgres-qwerty value is meant just for illustration, you should replace this with the value of the BIRDHOUSE_POSTGRES_PASSWORD variable. Note that you'll need to do the same for the stac-db service as well (assuming that you weren't previously overriding the STAC_POSTGRES_USER with a custom value).

Changes

Non-breaking changes

comment changes
most changes are backwards compatible (but not all)

Breaking changes

variable, file, constant name changes

Related Issue / Discussion

Resolves #414

Additional Information

Links to other issues or sources.

[x] Options/flags that modify how the compose script behaves should all use BIRDHOUSE_ as prefix
[x] manually test upgrading an instance using the migration guide (above)
[x] manually test autodeploy (note: we need to automate this somehow)
[ ] update associated projects:
- [x] https://github.com/DACCS-Climate/marble_client_python
- [x] https://github.com/bird-house/pavics-jupyter-base
- [ ] https://github.com/ouranosinc/pavics-datacatalog (deprecated, low priority)

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

mishaschwartz commented 8 months ago

That's a good fix for migrating instances, but shouldn't that be a non issue for CI instances recreated from scratch?

Yeah you're right.

I'm not really seeing any of the same error messages that you're reporting on my end. Could you share your env.local file? (with any private info removed of course) so I can try to properly replicate the issue?

crim-jenkins-bot commented 8 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2570/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

fmigneault commented 8 months ago

@mishaschwartz This is the config:

#############################################################################
# Mandatory vars (will be enforced by pavics-compose.sh)
# Can add new vars but do not remove, else automated deployment will break
#
# Do NOT use environment variables in here since when pavics-compose.sh runs
# inside a container, the environment vars do not have the same value.
#############################################################################

[REDACTED]

#############################################################################
# Optional vars
#############################################################################

#----------------------------------------------------------------------------
# Server Identification Detail
#----------------------------------------------------------------------------
export SERVER_NAME=CRIM-TEST-CI
export SERVER_INSTITUTION=CRIM
export SERVER_SUBJECT=Testing
export SERVER_TAGS=test,ci,validation,integration,end-2-end
export SERVER_DESCRIPTION="
Platform to test functionatities of birdhouse-deploy (https://github.com/bird-house/birdhouse-deploy)
employed by multiple institutions to provide the core configuration of components for PAVICS/DACCS/Marble nodes
(https://github.com/DACCS-Climate/Marble-node-registry).
"
export SUPPORT_EMAIL=support-geo@crim.ca

# Extra dirs containing docker-compose-extra.yml file to override the default
# docker-compose.yml file.  Inside each extra dir, there must be a
# docker-compose-extra.yml file.  If this file is not found, the dir is
# ignored.
#
# Useful to split configs into different dirs leveraging docker-compose
# override capabilities, see https://docs.docker.com/compose/extends/.
#
# Possible use-cases:
#
# * Split config to have all private and sensitive infos in a different
#   config dir.
#
# * Manage different env (prod, stating, dev, ...) with each env specific
#   config in its own dir leaving the default config dir with generic/common
#   config only.
#
# * Assemble different combinations of components/functionalities by including
#   only the config/docker-compose fragment necessary.
#
# Last dir/docker-compose fragment in the list have highest override precedence.
# Ex: last docker-compose volume mount to same destination win over the
# previous docker-compose volume mount.
#
# Format: space separated list of dirs
#
#export EXTRA_CONF_DIRS="/path/to/dir1 ./path/to/dir2 dir3 dir4"
export EXTRA_CONF_DIRS="
    ./config/proxy
    ./config/magpie
    ./config/twitcher
    ./config/canarie-api
    ./config/geoserver
    ./config/finch
    ./config/raven
    ./config/hummingbird
    ./config/thredds
    ./config/portainer
    ./config/jupyterhub
    ./components/proxy
    ./components/magpie
    ./components/twitcher
    ./components/canarie-api
    ./components/geoserver
    ./components/finch
    ./components/raven
    ./components/hummingbird
    ./components/thredds
    ./components/portainer
    ./components/jupyterhub
    ./components/monitoring
    ./components/weaver
    ./components/cowbird
    ./components/stac
    ./optional-components/canarie-api-full-monitoring
    ./optional-components/all-public-access
    ./optional-components/testthredds
    ./optional-components/secure-thredds
    ./optional-components/secure-data-proxy
    ./optional-components/stac-data-proxy
    ./optional-components/stac-populator
    ./optional-components/wps-healthchecks
    ./optional-components/database-external-ports
    ./optional-components/test-weaver
    ./optional-components/test-cowbird-jupyter-access
    ./optional-components/test-geoserver-secured-access
    ./optional-components/x-robots-tag-header
"
# must toggle the option to have pavics-sdi optional weaver component tested:
# https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/master/Jenkinsfile
export TEST_PAVICS_SDI_WEAVER=true
# in turn, the following variable must also be set to force to have pavics-sdi optional component notebook added:
# https://github.com/Ouranosinc/pavics-sdi/blob/master/docs/source/notebook-components/weaver_example.ipynb
# but since it is executed via 'PAVICS-e2e-workflow-tests', it must be passed indirectly with the extra variable
export EXTRA_TEST_ENV_VAR="${EXTRA_TEST_ENV_VAR} WEAVER_TEST_ENABLED=true"

# Extra repos, than the current repo, the autodeploy should keep up-to-date.
# Any changes to these extra repos will also trigger autodeploy.
#
# Useful to save the instanciated version of this env.local config file and
# any custom docker-compose-extra.yml from the previous EXTRA_CONF_DIRS var.
#
# Note:
#
# * These extra repos should be git repos for the out-of-date detection to
# work to trigger autodeploy.  If you just have a regular folder, do not add
# it here, out-of-date detection currently only works for git repos.
#
# * To preserve write permissions for your user, run once for this repo and
# once each time AUTODEPLOY_EXTRA_REPOS changes:
#     deployment/fix-write-perm
#
# Format: space separated list of full path to dirs
#export AUTODEPLOY_EXTRA_REPOS="/path/to/dir1 /path/to/dir2 /path/to/dir3"
#export AUTODEPLOY_EXTRA_REPOS="/path/to/private-config-containing-env.local"

# For each git repo in AUTODEPLOY_EXTRA_REPOS that use ssh to clone/fetch
# instead of https, provide its corresponding ssh deploy key in this dir.
#
# See instructions in deployment/deploy.sh or
# https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys for
# how to create deploy key for your git repos.
#
# The autodeploy mechanism runs inside its own container so environment
# variables are not the same inside and outside the container.  Do not use
# any environment vars, use their fully resolved values.
#
# Format of keys inside the dir: {repo-name-1}_deploy_key,
#   {repo-name-2}_deploy_key, ...
#
# If '{repo-name}_deploy_key' file is not found, default to
# 'id_rsa_git_ssh_read_only' so if multiple private repos share the same ssh
# deploy key, you can just name that shared key id_rsa_git_ssh_read_only and
# create {repo-name}_deploy_key only for repo specific key.
#
# Example of keys inside the dir: dir1_deploy_key, dir2_deploy_key,
#   private-config-containing-env.local_deploy_key,
#   id_rsa_git_ssh_read_only
#
#export AUTODEPLOY_DEPLOY_KEY_ROOT_DIR="/path/to/ssh-deploy-keys-for-all-repos"

# Frequency to trigger the various autodeploy tasks.
# See common.env for default.
#
# For all possible syntax, see implementation at
# https://github.com/Ouranosinc/docker-crontab/blob/3ac8cfa363b3f2ffdd0ead6089d355ff84521dc9/docker-entrypoint#L137-L184
#
# Ex:
# - daily at 5:07 AM: "7 5 * * *"
# - daily at midnight: "0 0 * * *" or "@daily"/"@midnight"
# - hourly: ""0 * * * *" or "@hourly"
# - every 2 hours: "*/120 * * * *" or "@every 2h"
# - every 5 minutes: "*/5 * * * *" or "@every 5m"
#
#export AUTODEPLOY_PLATFORM_FREQUENCY="@every 5m"
export AUTODEPLOY_NOTEBOOK_FREQUENCY="@every 5m"

# Add more jobs to ./components/scheduler/config.yml
#
# Potential usages: other deployment, backup jobs on the same machine
#
#export AUTODEPLOY_EXTRA_SCHEDULER_JOBS=""

# Public (on the internet) fully qualified domain name of this Pavics
# installation.  This is optional so default to the same internal PAVICS_FQDN if
# not set.
#export PAVICS_FQDN_PUBLIC="$PAVICS_FQDN"

# If ALLOW_UNSECURE_HTTP is enabled, port 80 will not redirect to 443
# anymore and will have the same service definitions as 443.
#
# This is so we can use pagekite proper SSL certificate.  Pagekite will
# hit port 80 instead of 443 and will be the one providing the SSL cert.
#
# Port 443 is still active with whatever the SSL cert it was given before.
#
# Set to "True" to allow traffic on unsecure port 80.
# This is for debugging only, do Not set this on a Production server.
#
export ALLOW_UNSECURE_HTTP="True"

# Docker images name to be used in the jupyterhub
# For PAVICS image, use default from birdhouse-deploy to obtain latest reference automatically.
DOCKER_NOTEBOOK_IMAGES_DEFAULT="${DOCKER_NOTEBOOK_IMAGES}"
if [ "$(echo "${DOCKER_NOTEBOOK_IMAGES}" | grep -c 'pavics/workflow-tests' || true)" -eq 0 ]; then
  echo "WARNING: Fixing missing default 'DOCKER_NOTEBOOK_IMAGES'..."
  #  backward compat
  COMPOSE_DIR="${COMPOSE_DIR:-$(dirname "$(realpath $0)")}"
  echo "WARNING: Resolved 'COMPOSE_DIR=${COMPOSE_DIR}'"
  if [ -f "${COMPOSE_DIR}/config/jupyterhub/default.env" ]; then
    DOCKER_NOTEBOOK_IMAGES_DEFAULT=$(cat "${COMPOSE_DIR}/config/jupyterhub/default.env" | grep 'DOCKER_NOTEBOOK_IMAGES=' | cut -d '=' -f 2)
  elif [ -f "${COMPOSE_DIR}/components/jupyterhub/default.env" ]; then
    DOCKER_NOTEBOOK_IMAGES_DEFAULT=$(cat "${COMPOSE_DIR}/components/jupyterhub/default.env" | grep 'DOCKER_NOTEBOOK_IMAGES=' | cut -d '=' -f 2)
  fi
  echo "WARNING: Resolved 'DOCKER_NOTEBOOK_IMAGES_DEFAULT=${DOCKER_NOTEBOOK_IMAGES_DEFAULT}'"
fi
export PAVICS_IMAGE="$(echo ${DOCKER_NOTEBOOK_IMAGES_DEFAULT} | grep pavics/workflow-tests | xargs)"
export PAVICS_VERSION="$(echo ${PAVICS_IMAGE} | cut -d ':' -f2)"
export PAVICS_NAME="$(echo ${PAVICS_IMAGE} | cut -d ':' -f1 | sed 's/[^[:alnum:]-]/-/g')"

# Jupyter single-user server images
export DOCKER_NOTEBOOK_IMAGES="${PAVICS_IMAGE}:${PAVICS_VERSION}"

# Name of the images displayed on the JupyterHub image selection page
# The name order must correspond to the order of the DOCKER_NOTEBOOK_IMAGES variable,
# and both variables should have the same number of entries.
# Note that the selection names are also used as directory names for the tutorial-notebooks directories mounted when
# starting the corresponding image. The name can use the '<name>' or the '<name>:<version>' format. The version will be
# excluded when mounting the corresponding directory.
export JUPYTERHUB_IMAGE_SELECTION_NAMES="${PAVICS_NAME}:${PAVICS_VERSION}"

echo "INFO: Resolved 'DOCKER_NOTEBOOK_IMAGES=${DOCKER_NOTEBOOK_IMAGES}'"
echo "INFO: Resolved 'JUPYTERHUB_IMAGE_SELECTION_NAMES=${JUPYTERHUB_IMAGE_SELECTION_NAMES}'"

# allow jupyterhub user selection of which notebook image to run
# see https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html
export ENABLE_JUPYTERHUB_MULTI_NOTEBOOKS="
c.DockerSpawner.image_whitelist = {
  **{
    _name: _img for _name, _img in
    zip(os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split(), os.environ['DOCKER_NOTEBOOK_IMAGES'].split())
  },
  'jupyter/scipy-notebook': 'jupyter/scipy-notebook',
  'jupyter/r-notebook': 'jupyter/r-notebook',
  'jupyter/tensorflow-notebook': 'jupyter/tensorflow-notebook',
  'jupyter/datascience-notebook': 'jupyter/datascience-notebook',
  'jupyter/pyspark-notebook': 'jupyter/pyspark-notebook',
  'jupyter/all-spark-notebook': 'jupyter/all-spark-notebook',
}
"

# Load jobs to automatically deploy the custom notebooks from the specific images
#
# Ensure we always use the "latest" version of the "cronjob generation code"
# Path to a checked out repo of "pavics-jupyter-base" (https://github.com/bird-house/pavics-jupyter-base)
# which contains the config required for the cronjob generation
#CHECKOUT_PAVICS_JUPYTER_BASE="/path/to/checkout/pavics-jupyter-base"
#export AUTODEPLOY_EXTRA_REPOS="$AUTODEPLOY_EXTRA_REPOS $CHECKOUT_PAVICS_JUPYTER_BASE"

# Config for the generation of cronjobs, found on external repo
#DEPLOY_DATA_PAVICS_JUPYTER_ENV="$CHECKOUT_PAVICS_JUPYTER_BASE/scheduler-jobs/deploy_data_pavics_jupyter.env"

# Generates a cronjob for each image found in DOCKERNOTEBOOK_IMAGES
#if [ -f "$DEPLOY_DATA_PAVICS_JUPYTER_ENV" ]; then
#  . $DEPLOY_DATA_PAVICS_JUPYTER_ENV
#fi

# Activates mounting a tutorial-notebooks subfolder that has the same name as the spawned image on JupyterHub
# This variable is only useful if there are more than one image in DOCKER_NOTEBOOK_IMAGES
# and ENABLE_JUPYTERHUB_MULTI_NOTEBOOKS is set with a proper c.DockerSpawner.image_whitelist
# matching the images in DOCKER_NOTEBOOK_IMAGES and their corresponding JUPYTERHUB_IMAGE_SELECTION_NAMES.
#export MOUNT_IMAGE_SPECIFIC_NOTEBOOKS=true

# The parent folder where all the user notebooks will be stored.
# For example, a user named "bob" will have his data in $JUPYTERHUB_USER_DATA_DIR/bob
# and this folder will be mounted when he logs into JupyterHub.
#export JUPYTERHUB_USER_DATA_DIR=/data/jupyterhub_user_data

# Path to the file containing the clientID for the google drive extension for jupyterlab
# This file will be mounted into JupyterLab instances.
# It should contain the following data : {"clientId":"<add_client_id_here>"}
# To setup a project and find the clientID, check the doc at : https://github.com/jupyterlab/jupyterlab-google-drive/blob/master/docs/setup.md
export JUPYTER_GOOGLE_DRIVE_SETTINGS=    #/home/ubuntu/daccs-configs/test/jupyter/drive_daccs.jupyterlab-settings

# Extra PyWPS config for **all** WPS services (currently only Flyingpigeon, Finch and Raven supported).
# export EXTRA_PYWPS_CONFIG="
# [logging]
# level = DEBUG
# "

# Thredds server customization
#
# Name of organization hosting the Thredds server
#export THREDDS_ORGANIZATION="Birdhouse"

# Allow using Github as external AuthN/AuthZ provider with Magpie
#   To setup Github as login, goto <https://github.com/settings/developers> under section [OAuth Apps]
#   and create a new Magpie application with configurations:
#
#       Homepage URL:                   https://${PAVICS_FQDN}
#       Authorization callback URL:     https://${PAVICS_FQDN}/magpie/providers/github/signin
#
#   Then, specify obtained Github client ID/Secret for this Magpie OAuth App with following variables.
#
#export GITHUB_CLIENT_ID=####
#export GITHUB_CLIENT_SECRET=####

# Magpie DB name
#export MAGPIE_DB_NAME="magpiedb"

# Set to 'false' if using self-signed SSL certificate
export VERIFY_SSL="false"

# Jupyter public demo account with limited computing resources for security reasons
#export JUPYTER_DEMO_USER="demo"
# Changing any limits requires restarting the jupyter user server
#export JUPYTER_DEMO_USER_MEM_LIMIT="2G"  # ex: 2G, 500M

# See config/jupyterhub/custom_templates/login.html.template
#export JUPYTER_LOGIN_BANNER_TOP_SECTION=""
#export JUPYTER_LOGIN_BANNER_BOTTOM_SECTION=""

#############################################################################
# Monitoring components configs
#############################################################################
#
# Below are Mandatory if monitoring component is enabled:
export GRAFANA_ADMIN_PASSWORD=[REDACTED]
export ALERTMANAGER_ADMIN_EMAIL_RECEIVER=[REDACTED]
export SMTP_SERVER=[REDACTED]

# Below are optional for monitoring component
#export ALERTMANAGER_EXTRA_GLOBAL=""
#export ALERTMANAGER_EXTRA_ROUTES=""
#export ALERTMANAGER_EXTRA_INHIBITION=""
#export ALERTMANAGER_EXTRA_RECEIVERS=""

#############################################################################
# Emu optional vars
#############################################################################

# To enable emu: add './optional-components/emu' to EXTRA_CONF_DIRS above.

# Emu WPS service image if that testing component is enabled
#export EMU_IMAGE="tlvu/emu:watchdog"

crim-jenkins-bot commented 8 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2571/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

fmigneault commented 8 months ago

@mishaschwartz I think an issue is related to the JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES variable that is passed in the docker-compose. Since env.local can define some overrides such as the following, the Jupyter image ends up raising a KeyError on the missing DOCKER_NOTEBOOK_IMAGES.

export ENABLE_JUPYTERHUB_MULTI_NOTEBOOKS="
c.DockerSpawner.image_whitelist = {
  **{
    _name: _img for _name, _img in
    zip(os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split(), os.environ['DOCKER_NOTEBOOK_IMAGES'].split())
  },
  'jupyter/scipy-notebook': 'jupyter/scipy-notebook',
  'jupyter/r-notebook': 'jupyter/r-notebook',
  'jupyter/tensorflow-notebook': 'jupyter/tensorflow-notebook',
  'jupyter/datascience-notebook': 'jupyter/datascience-notebook',
  'jupyter/pyspark-notebook': 'jupyter/pyspark-notebook',
  'jupyter/all-spark-notebook': 'jupyter/all-spark-notebook',
}
"

crim-jenkins-bot commented 8 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2572/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

crim-jenkins-bot commented 8 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2573/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

mishaschwartz commented 8 months ago

I think an issue is related to the JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES variable that is passed in the docker-compose.

There's a component to fix that. It looks like it was not enabled in time. I've made another fix.

(The stack works as expected on my test machine using your env.local now)

crim-jenkins-bot commented 7 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2579/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

crim-jenkins-bot commented 7 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2580/
Result :warning: ABORTED

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://host-140-216.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

crim-jenkins-bot commented 7 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2583/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : fix-ci-test-nb-images
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

crim-jenkins-bot commented 7 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2584/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_CONFIGS_BRANCH : test-daccs-config-branch
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference pavics-to-birdhouse.

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2621/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1588/

NOTEBOOK TEST RESULTS

    
[2024-05-01T13:45:40.774Z] ============================= test session starts ==============================
[2024-05-01T13:45:40.774Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-05-01T13:45:40.774Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-01T13:45:40.774Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-05-01T13:45:40.774Z] collected 259 items
[2024-05-01T13:45:40.774Z] 
[2024-05-01T13:45:50.711Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-01T13:46:30.952Z] notebooks-auth/test_cowbird_jupyter.ipynb ........F.                     [ 10%]
[2024-05-01T13:46:34.113Z] notebooks-auth/test_thredds.ipynb ...........                            [ 15%]
[2024-05-01T13:47:22.465Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 17%]
[2024-05-01T13:47:31.445Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 20%]
[2024-05-01T13:47:40.123Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 22%]
[2024-05-01T13:47:46.771Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 25%]
[2024-05-01T13:55:09.333Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 30%]
[2024-05-01T13:55:11.882Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 30%]
[2024-05-01T13:55:20.779Z] ...............                                                          [ 36%]
[2024-05-01T13:55:29.729Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 38%]
[2024-05-01T13:55:37.117Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 40%]
[2024-05-01T13:55:52.898Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 42%]
[2024-05-01T13:56:21.972Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 45%]
[2024-05-01T13:56:23.747Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 47%]
[2024-05-01T14:00:50.221Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 53%]
[2024-05-01T14:02:07.162Z] .............                                                            [ 58%]
[2024-05-01T14:02:10.281Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 59%]
[2024-05-01T14:02:12.421Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 62%]
[2024-05-01T14:02:30.242Z] .................                                                        [ 69%]
[2024-05-01T14:02:37.242Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 71%]
[2024-05-01T14:02:48.816Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-05-01T14:02:58.179Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 78%]
[2024-05-01T14:02:59.559Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2024-05-01T14:03:02.857Z] ......                                                                   [ 81%]
[2024-05-01T14:03:11.004Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2024-05-01T14:03:26.048Z] .............                                                            [ 86%]
[2024-05-01T14:03:36.060Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2024-05-01T14:04:18.189Z] ....s.                                                                   [ 89%]
[2024-05-01T14:04:26.329Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-01T14:04:41.108Z] ..                                                                       [ 90%]
[2024-05-01T14:04:56.034Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2024-05-01T14:05:18.575Z] ......                                                                   [ 93%]
[2024-05-01T14:05:20.871Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-01T14:08:01.404Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-01T14:08:01.404Z] 
[2024-05-01T14:08:01.404Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2626/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1591/

NOTEBOOK TEST RESULTS

    
[2024-05-01T15:21:51.519Z] ============================= test session starts ==============================
[2024-05-01T15:21:51.519Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-05-01T15:21:51.519Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-01T15:21:51.519Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-05-01T15:21:51.519Z] collected 269 items
[2024-05-01T15:21:51.519Z] 
[2024-05-01T15:22:02.317Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-01T15:23:16.695Z] notebooks-auth/test_cowbird_jupyter.ipynb ........F.                     [ 10%]
[2024-05-01T15:23:18.852Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-01T15:24:07.121Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-01T15:24:17.008Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-05-01T15:24:26.073Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-01T15:24:36.856Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-05-01T15:32:14.131Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-05-01T15:32:16.672Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-05-01T15:32:23.849Z] ...............                                                          [ 34%]
[2024-05-01T15:32:32.885Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-01T15:32:41.135Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 39%]
[2024-05-01T15:32:57.354Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-05-01T15:33:02.271Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2024-05-01T15:33:07.032Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-05-01T15:36:06.250Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-05-01T15:37:22.782Z] .............                                                            [ 56%]
[2024-05-01T15:37:25.487Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-05-01T15:37:27.695Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-05-01T15:37:44.714Z] .................                                                        [ 66%]
[2024-05-01T15:37:52.869Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 69%]
[2024-05-01T15:37:54.797Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-05-01T15:38:13.974Z] .........                                                                [ 72%]
[2024-05-01T15:38:22.673Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-05-01T15:38:31.999Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 79%]
[2024-05-01T15:38:33.386Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-05-01T15:38:37.064Z] ......                                                                   [ 81%]
[2024-05-01T15:38:47.066Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2024-05-01T15:39:01.608Z] .............                                                            [ 86%]
[2024-05-01T15:39:11.614Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-05-01T15:39:48.737Z] ....s.                                                                   [ 89%]
[2024-05-01T15:39:56.892Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-01T15:40:11.657Z] ..                                                                       [ 90%]
[2024-05-01T15:40:26.576Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-05-01T15:40:46.924Z] ......                                                                   [ 93%]
[2024-05-01T15:40:49.428Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-01T15:43:35.081Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-01T15:43:35.081Z] 
[2024-05-01T15:43:35.081Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2629/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1592/

NOTEBOOK TEST RESULTS

    
[2024-05-02T20:22:54.769Z] ============================= test session starts ==============================
[2024-05-02T20:22:54.769Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-05-02T20:22:54.769Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-02T20:22:54.769Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-05-02T20:22:54.769Z] collected 269 items
[2024-05-02T20:22:54.769Z] 
[2024-05-02T20:23:06.272Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-02T20:23:41.015Z] notebooks-auth/test_cowbird_jupyter.ipynb ........F.                     [ 10%]
[2024-05-02T20:23:46.135Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-02T20:24:35.525Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-02T20:24:44.497Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-05-02T20:24:52.928Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-02T20:25:04.404Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-05-02T20:32:35.492Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-05-02T20:32:37.420Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-05-02T20:32:45.768Z] ...............                                                          [ 34%]
[2024-05-02T20:32:54.854Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-02T20:33:01.867Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 39%]
[2024-05-02T20:33:17.925Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-05-02T20:33:23.477Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2024-05-02T20:33:27.990Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-05-02T20:37:09.976Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-05-02T20:38:25.190Z] .............                                                            [ 56%]
[2024-05-02T20:38:26.874Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-05-02T20:38:29.460Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-05-02T20:38:48.637Z] .................                                                        [ 66%]
[2024-05-02T20:38:56.730Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 69%]
[2024-05-02T20:38:58.110Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-05-02T20:39:16.721Z] .........                                                                [ 72%]
[2024-05-02T20:39:25.841Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-05-02T20:39:35.167Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 79%]
[2024-05-02T20:39:37.074Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-05-02T20:39:39.885Z] ......                                                                   [ 81%]
[2024-05-02T20:39:48.033Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2024-05-02T20:40:03.308Z] .............                                                            [ 86%]
[2024-05-02T20:40:15.558Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-05-02T20:40:58.211Z] ....s.                                                                   [ 89%]
[2024-05-02T20:41:06.355Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-02T20:41:19.668Z] ..                                                                       [ 90%]
[2024-05-02T20:41:34.583Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-05-02T20:41:55.822Z] ......                                                                   [ 93%]
[2024-05-02T20:41:58.570Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-02T20:44:32.735Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-02T20:44:32.735Z] 
[2024-05-02T20:44:32.735Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2631/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1593/

NOTEBOOK TEST RESULTS

    
[2024-05-03T16:10:38.029Z] ============================= test session starts ==============================
[2024-05-03T16:10:38.029Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-05-03T16:10:38.029Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-03T16:10:38.029Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-05-03T16:10:38.029Z] collected 269 items
[2024-05-03T16:10:38.029Z] 
[2024-05-03T16:10:50.146Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-03T16:11:24.874Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-05-03T16:11:28.395Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-03T16:12:17.597Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-03T16:12:26.835Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-05-03T16:12:36.918Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-03T16:12:43.619Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-05-03T16:20:13.780Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-05-03T16:20:14.365Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-05-03T16:20:23.283Z] ...............                                                          [ 34%]
[2024-05-03T16:20:32.696Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-03T16:20:39.801Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 39%]
[2024-05-03T16:20:56.481Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-05-03T16:21:02.382Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2024-05-03T16:21:06.895Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-05-03T16:25:25.344Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-05-03T16:26:41.324Z] .............                                                            [ 56%]
[2024-05-03T16:26:43.240Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-05-03T16:26:45.467Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-05-03T16:27:02.815Z] .................                                                        [ 66%]
[2024-05-03T16:27:10.517Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 69%]
[2024-05-03T16:27:11.907Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-05-03T16:27:29.695Z] .........                                                                [ 72%]
[2024-05-03T16:27:39.023Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-05-03T16:27:47.755Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 79%]
[2024-05-03T16:27:49.671Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-05-03T16:27:52.110Z] ......                                                                   [ 81%]
[2024-05-03T16:28:00.248Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2024-05-03T16:28:15.277Z] .............                                                            [ 86%]
[2024-05-03T16:28:27.535Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-05-03T16:29:04.095Z] ....s.                                                                   [ 89%]
[2024-05-03T16:29:12.235Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-03T16:29:27.753Z] ..                                                                       [ 90%]
[2024-05-03T16:29:42.676Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-05-03T16:30:06.005Z] ......                                                                   [ 93%]
[2024-05-03T16:30:07.359Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-03T16:32:32.527Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-03T16:32:32.527Z] 
[2024-05-03T16:32:32.527Z] ================= 268 passed, 1 skipped in 1314.91s (0:21:54) ==================

mishaschwartz commented 6 months ago

@fmigneault

Just FYI, from the jenkins logs:

fatal: Authentication failed for 'https://gitlab.com/crim.ca/clients/daccs/daccs-e2e-jenkins.git/'

fmigneault commented 6 months ago

run tests

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2647/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1602/

NOTEBOOK TEST RESULTS

    
[2024-05-15T22:59:20.021Z] ============================= test session starts ==============================
[2024-05-15T22:59:20.021Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-05-15T22:59:20.021Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-15T22:59:20.022Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-05-15T22:59:20.022Z] collected 268 items
[2024-05-15T22:59:20.022Z] 
[2024-05-15T22:59:31.333Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-15T22:59:53.576Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-05-15T23:00:00.356Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-15T23:00:55.665Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-15T23:01:04.379Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-05-15T23:01:12.784Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-15T23:01:15.022Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 24%]
[2024-05-15T23:08:22.241Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 29%]
[2024-05-15T23:08:22.241Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-05-15T23:08:26.337Z] ...............                                                          [ 35%]
[2024-05-15T23:08:33.915Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-15T23:08:41.181Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 39%]
[2024-05-15T23:08:55.641Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-05-15T23:09:00.587Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 44%]
[2024-05-15T23:09:04.827Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-05-15T23:12:58.293Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-05-15T23:13:59.870Z] .............                                                            [ 56%]
[2024-05-15T23:14:02.321Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-05-15T23:14:04.200Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-05-15T23:14:21.474Z] .................                                                        [ 67%]
[2024-05-15T23:14:28.445Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 69%]
[2024-05-15T23:14:29.388Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-05-15T23:14:47.412Z] .........                                                                [ 73%]
[2024-05-15T23:14:56.255Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 77%]
[2024-05-15T23:15:05.029Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 79%]
[2024-05-15T23:15:06.421Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-05-15T23:15:09.490Z] ......                                                                   [ 82%]
[2024-05-15T23:15:16.070Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2024-05-15T23:15:28.914Z] .............                                                            [ 87%]
[2024-05-15T23:15:37.052Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-05-15T23:15:57.231Z] ....s                                                                    [ 89%]
[2024-05-15T23:16:03.801Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-15T23:16:15.264Z] ..                                                                       [ 90%]
[2024-05-15T23:16:27.500Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-05-15T23:17:07.170Z] ......                                                                   [ 93%]
[2024-05-15T23:17:07.431Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-15T23:19:46.542Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-15T23:19:46.542Z] 
[2024-05-15T23:19:46.542Z] =================================== FAILURES ===================================

mishaschwartz commented 6 months ago

Tests are failing because https://neo.gsfc.nasa.gov/wms/wms?service=WMS&request=GetCapabilities&version=1.1.1 was temporarily down. I guess let's just wait a bit and try to run it again.

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2656/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1608/

NOTEBOOK TEST RESULTS

    
[2024-05-22T16:30:43.409Z] ============================= test session starts ==============================
[2024-05-22T16:30:43.409Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-05-22T16:30:43.409Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-22T16:30:43.409Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-05-22T16:30:43.409Z] collected 268 items
[2024-05-22T16:30:43.409Z] 
[2024-05-22T16:30:52.591Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-22T16:31:11.173Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-05-22T16:31:18.492Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-22T16:32:03.626Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-22T16:32:12.208Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-05-22T16:32:20.373Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-22T16:32:32.438Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-05-22T16:40:38.225Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 29%]
[2024-05-22T16:40:39.608Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-05-22T16:40:52.515Z] ...............                                                          [ 35%]
[2024-05-22T16:41:02.658Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-22T16:41:14.358Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 39%]
[2024-05-22T16:41:32.867Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-05-22T16:41:59.222Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 44%]
[2024-05-22T16:42:03.938Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-05-22T16:45:49.074Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-05-22T16:46:54.903Z] .............                                                            [ 56%]
[2024-05-22T16:46:57.645Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-05-22T16:46:59.389Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-05-22T16:47:15.103Z] .................                                                        [ 67%]
[2024-05-22T16:47:22.581Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 69%]
[2024-05-22T16:47:23.982Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-05-22T16:47:41.509Z] .........                                                                [ 73%]
[2024-05-22T16:47:50.595Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 77%]
[2024-05-22T16:47:59.564Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 79%]
[2024-05-22T16:48:00.963Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-05-22T16:48:04.299Z] ......                                                                   [ 82%]
[2024-05-22T16:48:12.434Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2024-05-22T16:48:24.614Z] .............                                                            [ 87%]
[2024-05-22T16:48:32.747Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-05-22T16:48:55.388Z] ....s                                                                    [ 89%]
[2024-05-22T16:49:03.542Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-22T16:49:15.486Z] ..                                                                       [ 90%]
[2024-05-22T16:49:25.504Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-05-22T16:50:08.475Z] ......                                                                   [ 93%]
[2024-05-22T16:50:10.513Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-22T16:52:44.618Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-22T16:52:44.618Z] 
[2024-05-22T16:52:44.618Z] ================= 267 passed, 1 skipped in 1318.97s (0:21:58) ==================

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2657/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1609/

NOTEBOOK TEST RESULTS

    
[2024-05-22T16:35:18.803Z] ============================= test session starts ==============================
[2024-05-22T16:35:18.803Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-05-22T16:35:18.803Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2024-05-22T16:35:18.803Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-05-22T16:35:18.803Z] collected 268 items
[2024-05-22T16:35:18.803Z] 
[2024-05-22T16:35:29.091Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-22T16:35:54.291Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-05-22T16:35:59.948Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-22T16:38:30.471Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-22T16:38:44.649Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-05-22T16:39:01.307Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-22T16:39:09.195Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-05-22T16:46:39.262Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 29%]
[2024-05-22T16:46:39.262Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-05-22T16:46:43.425Z] ...............                                                          [ 35%]
[2024-05-22T16:46:51.775Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-22T16:46:59.135Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 39%]
[2024-05-22T16:47:12.345Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-05-22T16:47:19.123Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 44%]
[2024-05-22T16:47:23.364Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-05-22T16:49:50.806Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-05-22T16:51:04.177Z] .............                                                            [ 56%]
[2024-05-22T16:51:06.610Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-05-22T16:51:08.513Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-05-22T16:51:24.193Z] .................                                                        [ 67%]
[2024-05-22T16:51:32.078Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 69%]
[2024-05-22T16:51:33.013Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-05-22T16:51:50.797Z] .........                                                                [ 73%]
[2024-05-22T16:51:59.867Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 77%]
[2024-05-22T16:52:09.488Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 79%]
[2024-05-22T16:52:10.893Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-05-22T16:52:14.441Z] ......                                                                   [ 82%]
[2024-05-22T16:52:22.567Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2024-05-22T16:52:34.594Z] .............                                                            [ 87%]
[2024-05-22T16:52:42.723Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-05-22T16:53:07.526Z] ....s                                                                    [ 89%]
[2024-05-22T16:53:15.658Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-22T16:53:26.592Z] ..                                                                       [ 90%]
[2024-05-22T16:53:38.827Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-05-22T16:54:28.137Z] ......                                                                   [ 93%]
[2024-05-22T16:54:28.137Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-22T16:57:01.468Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-22T16:57:01.469Z] 
[2024-05-22T16:57:01.469Z] ================= 267 passed, 1 skipped in 1303.07s (0:21:43) ==================

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2658/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1610/

NOTEBOOK TEST RESULTS

    
[2024-05-22T18:28:41.478Z] ============================= test session starts ==============================
[2024-05-22T18:28:41.478Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-05-22T18:28:41.478Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-22T18:28:41.478Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-05-22T18:28:41.478Z] collected 268 items
[2024-05-22T18:28:41.478Z] 
[2024-05-22T18:28:51.027Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-22T18:29:11.888Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-05-22T18:29:19.526Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-22T18:30:05.645Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-22T18:30:16.285Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-05-22T18:30:25.016Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-22T18:30:28.429Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 24%]
[2024-05-22T18:37:55.795Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 29%]
[2024-05-22T18:37:55.795Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-05-22T18:38:03.623Z] ...............                                                          [ 35%]
[2024-05-22T18:38:11.314Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-22T18:38:19.458Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 39%]
[2024-05-22T18:38:34.731Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-05-22T18:38:41.831Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 44%]
[2024-05-22T18:38:46.342Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-05-22T18:42:09.228Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-05-22T18:43:17.297Z] .............                                                            [ 56%]
[2024-05-22T18:43:18.082Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-05-22T18:43:19.881Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-05-22T18:43:35.613Z] .................                                                        [ 67%]
[2024-05-22T18:43:43.160Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 69%]
[2024-05-22T18:43:44.108Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-05-22T18:44:01.489Z] .........                                                                [ 73%]
[2024-05-22T18:44:10.545Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 77%]
[2024-05-22T18:44:19.279Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 79%]
[2024-05-22T18:44:20.662Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-05-22T18:44:23.727Z] ......                                                                   [ 82%]
[2024-05-22T18:44:33.728Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2024-05-22T18:44:44.834Z] .............                                                            [ 87%]
[2024-05-22T18:44:52.968Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-05-22T18:45:25.907Z] ....s                                                                    [ 89%]
[2024-05-22T18:45:32.501Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-05-22T18:45:45.832Z] ..                                                                       [ 90%]
[2024-05-22T18:45:55.828Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-05-22T18:46:38.121Z] ......                                                                   [ 93%]
[2024-05-22T18:46:40.376Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-22T18:49:14.093Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-22T18:49:14.093Z] 
[2024-05-22T18:49:14.093Z] =================================== FAILURES ===================================

mishaschwartz commented 6 months ago

@tlvu this PR has been tested with autodeploy: from master to this branch, and from this branch to a future update

tlvu commented 6 months ago

@tlvu this PR has been tested with autodeploy: from master to this branch, and from this branch to a future update

I let @tlvu see if anything else still needs to be addressed.

@mishaschwartz @fmigneault yes my bad, extremely sorry. I was too consumed lately on other issues on Ouranos side. If I can not get to this next week, I think you can merge and new findings from me will just be in a followup PR.

mishaschwartz commented 6 months ago

@tlvu

Ok thanks for the follow up. I'll aim to merge this on Friday May 31st if you haven't had a chance to look at it before then.

crim-jenkins-bot commented 6 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2666/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1614/

NOTEBOOK TEST RESULTS

    
[2024-05-27T13:43:14.389Z] ============================= test session starts ==============================
[2024-05-27T13:43:14.389Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-05-27T13:43:14.389Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-27T13:43:14.389Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-05-27T13:43:14.389Z] collected 275 items
[2024-05-27T13:43:14.389Z] 
[2024-05-27T13:43:23.377Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-27T13:43:42.315Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-05-27T13:43:50.176Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-27T13:44:35.334Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-27T13:44:43.987Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-05-27T13:44:52.393Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-27T13:45:01.140Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-05-27T13:52:11.342Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-05-27T13:52:11.342Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-05-27T13:52:19.199Z] ...............                                                          [ 34%]
[2024-05-27T13:52:26.905Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-27T13:52:34.596Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-05-27T13:52:48.473Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-05-27T13:52:54.569Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-05-27T13:52:58.910Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 44%]
[2024-05-27T13:56:37.403Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-05-27T13:57:47.987Z] .............                                                            [ 54%]
[2024-05-27T13:57:50.686Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-05-27T13:57:52.457Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-05-27T13:58:07.287Z] .................                                                        [ 65%]
[2024-05-27T13:58:14.742Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2024-05-27T13:58:15.687Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 68%]
[2024-05-27T13:58:33.484Z] .........                                                                [ 71%]
[2024-05-27T13:58:42.521Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2024-05-27T13:58:52.321Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2024-05-27T13:58:53.706Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-05-27T13:58:56.793Z] .....                                                                    [ 79%]
[2024-05-27T13:59:11.686Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2024-05-27T13:59:28.290Z] ............                                                             [ 84%]
[2024-05-27T13:59:43.225Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-05-27T14:00:03.005Z] .....s.                                                                  [ 87%]
[2024-05-27T14:00:11.148Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-05-27T14:00:25.022Z] ..                                                                       [ 88%]
[2024-05-27T14:00:39.933Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-05-27T14:01:34.222Z] ........                                                                 [ 91%]
[2024-05-27T14:01:44.240Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-05-27T14:06:34.903Z] ....                                                                     [ 93%]
[2024-05-27T14:06:34.903Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-27T14:08:58.109Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-27T14:08:58.109Z] 
[2024-05-27T14:08:58.109Z] =============================== warnings summary ===============================

crim-jenkins-bot commented 5 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2669/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1615/

NOTEBOOK TEST RESULTS

    
[2024-05-29T12:32:48.429Z] ============================= test session starts ==============================
[2024-05-29T12:32:48.429Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-05-29T12:32:48.429Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-05-29T12:32:48.429Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-05-29T12:32:48.429Z] collected 275 items
[2024-05-29T12:32:48.429Z] 
[2024-05-29T12:32:57.741Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-05-29T12:33:18.635Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-05-29T12:33:26.579Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-05-29T12:34:12.562Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-05-29T12:34:21.237Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-05-29T12:34:29.344Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-05-29T12:34:38.632Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-05-29T12:42:02.361Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-05-29T12:42:02.362Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-05-29T12:42:06.836Z] ...............                                                          [ 34%]
[2024-05-29T12:42:14.530Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-05-29T12:42:21.204Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-05-29T12:42:35.076Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-05-29T12:42:42.180Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-05-29T12:42:46.420Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 44%]
[2024-05-29T12:46:11.956Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-05-29T12:47:22.006Z] .............                                                            [ 54%]
[2024-05-29T12:47:23.470Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-05-29T12:47:24.951Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-05-29T12:47:40.212Z] .................                                                        [ 65%]
[2024-05-29T12:47:47.163Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2024-05-29T12:47:48.553Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 68%]
[2024-05-29T12:48:05.820Z] .........                                                                [ 71%]
[2024-05-29T12:48:14.367Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2024-05-29T12:48:23.364Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2024-05-29T12:48:24.754Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-05-29T12:48:27.826Z] .....                                                                    [ 79%]
[2024-05-29T12:48:42.739Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2024-05-29T12:48:58.352Z] ............                                                             [ 84%]
[2024-05-29T12:49:13.280Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-05-29T12:49:33.439Z] .....s.                                                                  [ 87%]
[2024-05-29T12:49:40.015Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-05-29T12:49:54.816Z] ..                                                                       [ 88%]
[2024-05-29T12:50:09.731Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-05-29T12:51:04.060Z] ........                                                                 [ 91%]
[2024-05-29T12:51:14.076Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-05-29T12:55:57.716Z] ....                                                                     [ 93%]
[2024-05-29T12:55:57.716Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-05-29T12:58:30.491Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-05-29T12:58:30.491Z] 
[2024-05-29T12:58:30.491Z] =============================== warnings summary ===============================

mishaschwartz commented 5 months ago

@fmigneault

I'd appreciate your opinion:

I'm tempted to bump this a major version since:

MAJOR version when the API or user facing UI changes that requires significant documentation update

And the documentation has significantly changed even though the UI hasn't been changed at all. But that's not strictly necessary because everything should be fully backwards compatible. What do you think?

tlvu commented 5 months ago

I'm tempted to bump this a major version

I am doing code only quick review today, not enough time to perform a real deploy review. I'll let you know from a fresh eye perspective that not see how extensive this change is.

fmigneault commented 5 months ago

@tlvu Beside the mandatory vars section, only the following are configured.


export SERVER_NAME=CRIM-TEST-CI
export SERVER_INSTITUTION=CRIM
export SERVER_SUBJECT=Testing
export SERVER_TAGS=test,ci,validation,integration,end-2-end
export SERVER_DESCRIPTION="
Platform to test functionatities of birdhouse-deploy (https://github.com/bird-house/birdhouse-deploy)
employed by multiple institutions to provide the core configuration of components for PAVICS/DACCS/Marble nodes
(https://github.com/DACCS-Climate/Marble-node-registry).
"
export SUPPORT_EMAIL=support-geo@crim.ca

export EXTRA_CONF_DIRS="
    ./config/proxy
    ./config/magpie
    ./config/twitcher
    ./config/canarie-api
    ./config/geoserver
    ./config/finch
    ./config/raven
    ./config/hummingbird
    ./config/thredds
    ./config/portainer
    ./config/jupyterhub
    ./components/proxy
    ./components/magpie
    ./components/twitcher
    ./components/canarie-api
    ./components/geoserver
    ./components/finch
    ./components/raven
    ./components/hummingbird
    ./components/thredds
    ./components/portainer
    ./components/jupyterhub
    ./components/monitoring
    ./components/weaver
    ./components/cowbird
    ./components/stac
    ./optional-components/canarie-api-full-monitoring
    ./optional-components/all-public-access
    ./optional-components/testthredds
    ./optional-components/secure-thredds
    ./optional-components/secure-data-proxy
    ./optional-components/stac-data-proxy
    ./optional-components/stac-populator
    ./optional-components/wps-healthchecks
    ./optional-components/database-external-ports
    ./optional-components/test-weaver
    ./optional-components/test-cowbird-jupyter-access
    ./optional-components/test-geoserver-secured-access
    ./optional-components/x-robots-tag-header
"
# must toggle the option to have pavics-sdi optional weaver component tested:
# https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/master/Jenkinsfile
export TEST_PAVICS_SDI_WEAVER=true
# in turn, the following variable must also be set to force to have pavics-sdi optional component notebook added:
# https://github.com/Ouranosinc/pavics-sdi/blob/master/docs/source/notebook-components/weaver_example.ipynb
# but since it is executed via 'PAVICS-e2e-workflow-tests', it must be passed indirectly with the extra variable
export EXTRA_TEST_ENV_VAR="${EXTRA_TEST_ENV_VAR} WEAVER_TEST_ENABLED=true"

# Docker images name to be used in the jupyterhub
# For PAVICS image, use default from birdhouse-deploy to obtain latest reference automatically.
DOCKER_NOTEBOOK_IMAGES_DEFAULT="${DOCKER_NOTEBOOK_IMAGES}"
if [ "$(echo "${DOCKER_NOTEBOOK_IMAGES}" | grep -c 'pavics/workflow-tests' || true)" -eq 0 ]; then
  echo "WARNING: Fixing missing default 'DOCKER_NOTEBOOK_IMAGES'..."
  #  backward compat
  COMPOSE_DIR="${COMPOSE_DIR:-$(dirname "$(realpath $0)")}"
  # in case of execution by jenkins worker or ubuntu service, consider sourced/exec method can cause $0 to be the runner
  # patch the path manually if invalid
  if [ "$(basename "${COMPOSE_DIR}")" != "birdhouse" ]; then
    echo "WARNING: Detected invalid 'COMPOSE_DIR=${COMPOSE_DIR}'. Overriding."
    COMPOSE_DIR="/home/ubuntu/${GH_REPO:-birdhouse-deploy}/birdhouse"
  fi
  echo "WARNING: Resolved 'COMPOSE_DIR=${COMPOSE_DIR}'"
  if [ -f "${COMPOSE_DIR}/config/jupyterhub/default.env" ]; then
    DOCKER_NOTEBOOK_IMAGES_DEFAULT=$(cat "${COMPOSE_DIR}/config/jupyterhub/default.env" | grep 'DOCKER_NOTEBOOK_IMAGES=' | cut -d '=' -f 2)
  elif [ -f "${COMPOSE_DIR}/components/jupyterhub/default.env" ]; then
    DOCKER_NOTEBOOK_IMAGES_DEFAULT=$(cat "${COMPOSE_DIR}/components/jupyterhub/default.env" | grep 'DOCKER_NOTEBOOK_IMAGES=' | cut -d '=' -f 2)
  fi
  echo "WARNING: Resolved 'DOCKER_NOTEBOOK_IMAGES_DEFAULT=${DOCKER_NOTEBOOK_IMAGES_DEFAULT}'"
fi
export PAVICS_IMAGE="$(echo ${DOCKER_NOTEBOOK_IMAGES_DEFAULT} | grep pavics/workflow-tests | xargs)"
export PAVICS_VERSION="$(echo ${PAVICS_IMAGE} | cut -d ':' -f2)"
export PAVICS_NAME="$(echo ${PAVICS_IMAGE} | cut -d ':' -f1 | sed 's/[^[:alnum:]-]/-/g')"

# Jupyter single-user server images
export DOCKER_NOTEBOOK_IMAGES="${PAVICS_IMAGE}"

export JUPYTERHUB_IMAGE_SELECTION_NAMES="${PAVICS_NAME}${PAVICS_IMAGE:+:}${PAVICS_VERSION}"

echo "INFO: Resolved 'DOCKER_NOTEBOOK_IMAGES=${DOCKER_NOTEBOOK_IMAGES}'"
echo "INFO: Resolved 'JUPYTERHUB_IMAGE_SELECTION_NAMES=${JUPYTERHUB_IMAGE_SELECTION_NAMES}'"

# allow jupyterhub user selection of which notebook image to run
# see https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html
export ENABLE_JUPYTERHUB_MULTI_NOTEBOOKS="
c.DockerSpawner.image_whitelist = {
  **{
    _name: _img for _name, _img in
    zip(os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split(), os.environ['DOCKER_NOTEBOOK_IMAGES'].split())
  },
  'jupyter/scipy-notebook': 'jupyter/scipy-notebook',
  'jupyter/r-notebook': 'jupyter/r-notebook',
  'jupyter/tensorflow-notebook': 'jupyter/tensorflow-notebook',
  'jupyter/datascience-notebook': 'jupyter/datascience-notebook',
  'jupyter/pyspark-notebook': 'jupyter/pyspark-notebook',
  'jupyter/all-spark-notebook': 'jupyter/all-spark-notebook',
}
"

fmigneault commented 5 months ago

@mishaschwartz

I'd appreciate your opinion:

I'm tempted to bump this a major version

I think MINOR could be sufficient since it is possible (at least from the CI's perspective) to run the stack without any change and tests working end-to-end, but I'm not against MAJOR either if @tlvu thinks the defaults (which TBD?) cause sufficient changes on his end.

I think the changes to the documentation adds a lot of precision and details, but they are not themselves introducing breaking changes. I don't think they reflect the semver mention "changes that requires significant documentation update" hinting that the documentation updates must be caused by/resulting from the stack changes initially.

I would consider MAJOR necessary if the introduced bin/birdhouse becomes the de facto runner, and that we phase out the use of birdhouse/pavics-compose.sh. However, since the birdhouse/read-configs.include.sh still relies on finding birdhouse-compose.sh, MINOR seems sufficient.

Again, not a strong feeling about either MINOR or MAJOR for this case. Same result from my perspective.

@tlvu

So for vars that we never had to set in env.local because we were relying on the old default values, we now have to set them to get back the default values.

Let me know which vars you refer to (is it only the THREDDS ones mentioned in https://github.com/bird-house/birdhouse-deploy/pull/428#discussion_r1622769717 ?). I have not applied any change in env.local or the compose script to run this PR. It is the same CI and config that runs on master, other PRs and this one.

mishaschwartz commented 5 months ago

@tlvu

Now the breaking part: some old default values have changed. So even if the old config var name still work, their default value have changed.

In some cases the old default is preserved (see BIRDHOUSE_BACKWARDS_COMPATIBLE_HARDCODED_DEFAULTS in default.env).

So for vars that we never had to set in env.local because we were relying on the old default values, we now have to set them to get back the default values.

Yes, in the migration guide we write:

https://github.com/bird-house/birdhouse-deploy/blob/2c3140f46545bef99fbfee34794098e177e6d9e5/CHANGES.md?plain=1#L68-L69

could I have the list of vars you had to set to keep the github pipeline and all your existing hosts running?

See the table in the migration guide for a list of variables that need to be changed to keep defaults backwards compatible:

https://github.com/bird-house/birdhouse-deploy/blob/2c3140f46545bef99fbfee34794098e177e6d9e5/CHANGES.md?plain=1#L71-L80

If you want to update all variables so that you don't have to run in backwards compatible mode, check out the migration guide for details on what to change.

mishaschwartz commented 5 months ago

@fmigneault

I'm happy to go with a minor bump as well.

I would consider MAJOR necessary if the introduced bin/birdhouse becomes the de facto runner, and that we phase out the use of birdhouse/pavics-compose.sh. However, since the birdhouse/read-configs.include.sh still relies on finding birdhouse-compose.sh, MINOR seems sufficient.

The goal of creating bin/birdhouse was that it would eventually become the de facto runner/interface for everything so that we could phase out or change any internal scripts without breaking backwards compatibility in the future. I'm introducing it here and presenting it as the proper interface to use for any external scripts in the future. Then we can give people lots of time to change their scripts to use bin/birdhouse before we remove pavics-compose.sh in a later version (with a major bump). For now, we need to keep pavics-compose.sh in order to maintain backwards compatiblity.

mishaschwartz commented 5 months ago

@tlvu @fmigneault given the slight changes I've made based on @tlvu's review I'll give this another 24 hours or so before I pull this in.

tlvu commented 5 months ago

Again, not a strong feeling about either MINOR or MAJOR for this case. Same result from my perspective.

Same here. I'll let @mishaschwartz decide on Minor or Major bump. On my side, I'll prepare my production env.local in advance so the upgrade is completely seemless.

So for vars that we never had to set in env.local because we were relying on the old default values, we now have to set them to get back the default values.

Let me know which vars you refer to (is it only the THREDDS ones mentioned in #428 (comment) ?). I have not applied any change in env.local or the compose script to run this PR. It is the same CI and config that runs on master, other PRs and this one.

Oh that probably because the CI pipeline is only testing a fresh deployment case.

For existing production deployments we won't be so lucky. I believe outarde is considered as your production hosts? You will at least have to care for that one.

This is a reminder that the CI pipeline is not enough catch all errors in a PR because it does not test the upgrade scenario. Furthermore the upgrade scenario is actually the most critical path because all existing production deployments will take the upgrade scenario path and not the fresh deployment from scratch path. This is why we currently have the issue that Cowbird do not play nice with existing Magpie users because the upgrade path has never been considered during Cowbird development and testing.

I think all the vars described here by @mishaschwartz will have to be set https://github.com/bird-house/birdhouse-deploy/blob/2c3140f46545bef99fbfee34794098e177e6d9e5/CHANGES.md?plain=1#L71-L80. Hopefully all the additional components from external repos that we have on ourside do not require more !

crim-jenkins-bot commented 5 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2675/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1619/

NOTEBOOK TEST RESULTS

    
[2024-06-03T15:51:15.771Z] ============================= test session starts ==============================
[2024-06-03T15:51:15.771Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-03T15:51:15.771Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-03T15:51:15.771Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-03T15:51:15.771Z] collected 276 items
[2024-06-03T15:51:15.771Z] 
[2024-06-03T15:51:26.600Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-06-03T15:51:47.061Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-06-03T15:51:50.243Z] notebooks-auth/test_thredds.ipynb ......FFFF.                            [ 14%]
[2024-06-03T15:52:36.547Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-06-03T15:52:45.147Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-06-03T15:52:53.810Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-06-03T15:53:04.548Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 23%]
[2024-06-03T16:00:35.546Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-06-03T16:00:37.463Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-06-03T16:00:45.498Z] ...............                                                          [ 34%]
[2024-06-03T16:00:53.636Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-06-03T16:01:01.071Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-06-03T16:01:16.138Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-06-03T16:01:21.011Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-06-03T16:01:25.448Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .F..F       [ 44%]
[2024-06-03T16:01:53.884Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb .FF..FFFFFFFF.. [ 50%]
[2024-06-03T16:02:38.220Z] F.FFFF.......                                                            [ 54%]
[2024-06-03T16:02:40.397Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb FFFF             [ 56%]
[2024-06-03T16:02:42.375Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-06-03T16:03:01.438Z] .................                                                        [ 65%]
[2024-06-03T16:03:08.618Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb F..FF.          [ 67%]
[2024-06-03T16:03:10.012Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2024-06-03T16:03:33.447Z] ........F                                                                [ 71%]
[2024-06-03T16:03:39.217Z] finch-master/docs/source/notebooks/dap_subset.ipynb .FFFFFFFFF.          [ 75%]
[2024-06-03T16:03:45.345Z] finch-master/docs/source/notebooks/finch-usage.ipynb ..FFFF              [ 77%]
[2024-06-03T16:03:46.749Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-06-03T16:03:50.318Z] .....                                                                    [ 79%]
[2024-06-03T16:04:08.451Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2024-06-03T16:04:23.844Z] ............                                                             [ 84%]
[2024-06-03T16:04:41.975Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-06-03T16:05:01.553Z] .....s.                                                                  [ 86%]
[2024-06-03T16:05:09.690Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-06-03T16:05:26.505Z] ..                                                                       [ 88%]
[2024-06-03T16:05:34.667Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-06-03T16:06:39.062Z] F........                                                                [ 91%]
[2024-06-03T16:06:49.082Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-06-03T16:11:49.080Z] F...                                                                     [ 93%]
[2024-06-03T16:11:49.080Z] notebooks/hummingbird.ipynb .........F..                                 [ 97%]
[2024-06-03T16:14:27.873Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-03T16:14:27.873Z] 
[2024-06-03T16:14:27.873Z] =================================== FAILURES ===================================

mishaschwartz commented 5 months ago

I'm reverting the changes to the thredds variables because they break the tests and I don't want to go through the effort of fixing the tests again. If we want to change these we can do it in a future update.

crim-jenkins-bot commented 5 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2676/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1620/

NOTEBOOK TEST RESULTS

    
[2024-06-03T17:31:43.349Z] ============================= test session starts ==============================
[2024-06-03T17:31:43.349Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-03T17:31:43.349Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-03T17:31:43.349Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-03T17:31:43.349Z] collected 276 items
[2024-06-03T17:31:43.349Z] 
[2024-06-03T17:31:54.855Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-06-03T17:32:15.750Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-06-03T17:32:21.706Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-06-03T17:33:08.522Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-06-03T17:33:17.855Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-06-03T17:33:26.278Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-06-03T17:33:43.530Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 23%]
[2024-06-03T17:41:05.940Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-06-03T17:41:05.941Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-06-03T17:41:09.861Z] ...............                                                          [ 34%]
[2024-06-03T17:41:17.455Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-06-03T17:41:24.494Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-06-03T17:41:38.353Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-06-03T17:42:20.540Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-06-03T17:42:23.826Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 44%]
[2024-06-03T17:46:17.742Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-06-03T17:47:19.461Z] .............                                                            [ 54%]
[2024-06-03T17:47:22.764Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-06-03T17:47:24.688Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-06-03T17:47:40.114Z] .................                                                        [ 65%]
[2024-06-03T17:47:46.795Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2024-06-03T17:47:47.739Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2024-06-03T17:49:01.608Z] ......FFF                                                                [ 71%]
[2024-06-03T17:49:12.689Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2024-06-03T17:49:20.980Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2024-06-03T17:49:22.365Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-06-03T17:49:25.175Z] .....                                                                    [ 79%]
[2024-06-03T17:49:40.100Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2024-06-03T17:49:55.575Z] ............                                                             [ 84%]
[2024-06-03T17:50:22.167Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-06-03T17:50:42.532Z] .....s.                                                                  [ 86%]
[2024-06-03T17:50:49.121Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-06-03T17:51:05.941Z] ..                                                                       [ 88%]
[2024-06-03T17:51:12.534Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-06-03T17:52:09.190Z] F........                                                                [ 91%]
[2024-06-03T17:52:17.326Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-06-03T17:57:09.344Z] F...                                                                     [ 93%]
[2024-06-03T17:57:09.344Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-06-03T17:59:32.564Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-03T17:59:32.564Z] 
[2024-06-03T17:59:32.564Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 5 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2678/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1622/

NOTEBOOK TEST RESULTS

    
[2024-06-04T15:21:49.722Z] ============================= test session starts ==============================
[2024-06-04T15:21:49.722Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-04T15:21:49.722Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-04T15:21:49.722Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-04T15:21:49.722Z] collected 276 items
[2024-06-04T15:21:49.722Z] 
[2024-06-04T15:21:59.637Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-06-04T15:22:23.212Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-06-04T15:22:30.542Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-06-04T15:23:16.029Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-06-04T15:23:25.074Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-06-04T15:23:34.807Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-06-04T15:23:43.804Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 23%]
[2024-06-04T15:31:11.084Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-06-04T15:31:11.084Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-06-04T15:31:12.985Z] ...............                                                          [ 34%]
[2024-06-04T15:31:20.926Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-06-04T15:31:29.195Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-06-04T15:31:42.461Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-06-04T15:31:47.691Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-06-04T15:31:52.345Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 44%]
[2024-06-04T15:35:59.765Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-06-04T15:37:02.353Z] .............                                                            [ 54%]
[2024-06-04T15:37:06.435Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-06-04T15:38:14.157Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb EEEEEEEE [ 59%]
[2024-06-04T15:38:27.807Z] EEEEEEEEEEEEEEEEE                                                        [ 65%]
[2024-06-04T15:38:36.739Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2024-06-04T15:38:37.690Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2024-06-04T15:38:55.496Z] .........                                                                [ 71%]
[2024-06-04T15:39:04.499Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2024-06-04T15:39:13.153Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2024-06-04T15:39:14.102Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-06-04T15:39:17.443Z] .....                                                                    [ 79%]
[2024-06-04T15:39:32.352Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2024-06-04T15:39:47.964Z] ............                                                             [ 84%]
[2024-06-04T15:40:02.864Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-06-04T15:40:40.001Z] .....s.                                                                  [ 86%]
[2024-06-04T15:40:46.588Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-06-04T15:41:04.057Z] ..                                                                       [ 88%]
[2024-06-04T15:41:12.213Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-06-04T15:42:14.646Z] F........                                                                [ 91%]
[2024-06-04T15:42:24.642Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-06-04T15:47:15.298Z] F...                                                                     [ 93%]
[2024-06-04T15:47:15.298Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-06-04T15:49:38.536Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-04T15:49:38.536Z] 
[2024-06-04T15:49:38.536Z] ==================================== ERRORS ====================================

mishaschwartz commented 5 months ago

aaand test are failing again. Nothing in the code has changed with this PR so I'm going to just assume that the flaky tests got more flaky and I'm going to pull this in.

tlvu commented 5 months ago

Before pulling this in, can you revert to the existing behavior of git status --untracked-files=all, see this comment thread https://github.com/bird-house/birdhouse-deploy/pull/428#discussion_r1622777292

crim-jenkins-bot commented 5 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2680/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1623/

NOTEBOOK TEST RESULTS

    
[2024-06-04T17:57:08.663Z] ============================= test session starts ==============================
[2024-06-04T17:57:08.663Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-04T17:57:08.663Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-06-04T17:57:08.663Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-04T17:57:08.663Z] collected 276 items
[2024-06-04T17:57:08.663Z] 
[2024-06-04T17:57:18.123Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-06-04T17:57:40.320Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-06-04T17:57:47.083Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-06-04T17:58:35.342Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-06-04T17:58:52.502Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-06-04T17:59:01.922Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-06-04T17:59:18.523Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 23%]
[2024-06-04T18:12:41.804Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-06-04T18:12:44.353Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-06-04T18:12:55.530Z] ...............                                                          [ 34%]
[2024-06-04T18:13:03.865Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-06-04T18:13:10.690Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-06-04T18:13:26.285Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-06-04T18:13:32.294Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-06-04T18:13:37.234Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 44%]
[2024-06-04T18:17:46.538Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-06-04T18:18:49.658Z] .............                                                            [ 54%]
[2024-06-04T18:18:52.670Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-06-04T18:18:54.449Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-06-04T18:19:10.503Z] .................                                                        [ 65%]
[2024-06-04T18:19:18.351Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2024-06-04T18:19:19.297Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2024-06-04T18:19:36.831Z] .........                                                                [ 71%]
[2024-06-04T18:19:45.367Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2024-06-04T18:19:54.684Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2024-06-04T18:19:57.219Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-06-04T18:20:00.063Z] .....                                                                    [ 79%]
[2024-06-04T18:20:22.022Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2024-06-04T18:20:41.668Z] ............                                                             [ 84%]
[2024-06-04T18:20:59.756Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-06-04T18:21:27.427Z] .....s.                                                                  [ 86%]
[2024-06-04T18:21:34.006Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-06-04T18:21:51.519Z] ..                                                                       [ 88%]
[2024-06-04T18:21:59.650Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-06-04T18:23:01.251Z] F........                                                                [ 91%]
[2024-06-04T18:23:11.237Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-06-04T18:30:26.294Z] F...                                                                     [ 93%]
[2024-06-04T18:30:26.294Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-06-04T18:32:59.023Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-04T18:32:59.023Z] 
[2024-06-04T18:32:59.023Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 5 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2679/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : pavics-to-birdhouse
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1624/

NOTEBOOK TEST RESULTS

    
[2024-06-04T17:57:39.710Z] ============================= test session starts ==============================
[2024-06-04T17:57:39.710Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-06-04T17:57:39.710Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2024-06-04T17:57:39.710Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-06-04T17:57:39.710Z] collected 276 items
[2024-06-04T17:57:39.710Z] 
[2024-06-04T17:57:49.985Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-06-04T17:58:10.890Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-06-04T17:58:18.091Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-06-04T17:59:06.375Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-06-04T17:59:14.684Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 18%]
[2024-06-04T17:59:22.718Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-06-04T17:59:28.679Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 23%]
[2024-06-04T18:12:43.958Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-06-04T18:12:45.345Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 28%]
[2024-06-04T18:12:55.459Z] ...............                                                          [ 34%]
[2024-06-04T18:13:03.564Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 35%]
[2024-06-04T18:13:10.901Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-06-04T18:13:26.015Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 40%]
[2024-06-04T18:13:32.798Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 42%]
[2024-06-04T18:13:37.323Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 44%]
[2024-06-04T18:17:39.437Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 50%]
[2024-06-04T18:18:52.120Z] .............                                                            [ 54%]
[2024-06-04T18:18:55.603Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 56%]
[2024-06-04T18:18:57.562Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 59%]
[2024-06-04T18:19:13.559Z] .................                                                        [ 65%]
[2024-06-04T18:19:21.256Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2024-06-04T18:19:22.199Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2024-06-04T18:19:41.115Z] .........                                                                [ 71%]
[2024-06-04T18:19:50.419Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2024-06-04T18:19:59.737Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2024-06-04T18:20:00.677Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2024-06-04T18:20:04.000Z] .....                                                                    [ 79%]
[2024-06-04T18:20:25.957Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2024-06-04T18:20:45.920Z] ............                                                             [ 84%]
[2024-06-04T18:21:00.849Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2024-06-04T18:21:30.779Z] .....s.                                                                  [ 86%]
[2024-06-04T18:21:37.359Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2024-06-04T18:21:55.573Z] ..                                                                       [ 88%]
[2024-06-04T18:22:02.152Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 88%]
[2024-06-04T18:23:08.572Z] F........                                                                [ 91%]
[2024-06-04T18:23:18.556Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 92%]
[2024-06-04T18:31:33.544Z] F...                                                                     [ 93%]
[2024-06-04T18:31:33.544Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-06-04T18:34:06.328Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-06-04T18:34:06.328Z] 
[2024-06-04T18:34:06.328Z] =================================== FAILURES ===================================

bird-house / birdhouse-deploy

Rename variables, constants and files from PAVICS to Birdhouse #428

Overview

Migration Guide

Changes

Related Issue / Discussion

Additional Information

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results