bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform

https://birdhouse-deploy.readthedocs.io/en/latest/

Apache License 2.0

4 stars 7 forks source link

add `optional-components/stac-data-proxy` #403

Closed fmigneault closed 12 months ago

fmigneault commented 1 year ago

Overview

Provide a way to host local data that STAC API can refer to for use/download.

Currently, any STAC Asset that is referenced within responses by STAC-API Collections/Items must either be already hosted by another service of the stack (eg: CMIP6 netCDF in THREDDS), or point at some other external resource not on the server.

Instead of having a custom config and mount point for each node, this optional component defines a standard way to define it.

Changes

Non-breaking changes

optional-components/stac-data-proxy: add a new feature to allow hosting of local STAC assets.

The new component defines variables STAC_DATA_PROXY_DIR_PATH (default ${DATA_PERSIST_ROOT}/stac-data) and STAC_DATA_PROXY_URL_PATH (default /data/stac) that are aliased (mapped) under nginx to provide a URL where locally hosted STAC assets can be downloaded from. This allows a server node to be a proper data provider, where its STAC-API can return Catalog, Collection and Item definitions that points at these local assets available through the STAC_DATA_PROXY_URL_PATH endpoint.

When enabled, this component can be combined with optional-components/secure-data-proxy to allow per-resource access control of the contents under STAC_DATA_PROXY_DIR_PATH by setting relevant Magpie permissions under service secure-data-proxy for children resources that correspond to STAC_DATA_PROXY_URL_PATH. Otherwise, the path and all of its contents are publicly available, in the same fashion that WPS outputs are managed without optional-components/secure-data-proxy.

More details are provided in https://github.com/bird-house/birdhouse-deploy/blob/stac-data-proxy/birdhouse/optional-components/README.rst#provide-a-proxy-for-local-stac-asset-hosting

Breaking changes

Related Issue / Discussion

Relates to https://github.com/crim-ca/stac-populator/pull/31
Relates to contents in https://github.com/ai-extensions/stac-data-loader/tree/main/data/EuroSAT/stac
Relates to https://github.com/ai-extensions/stac-data-loader/blob/main/notebooks/stac_eurosat.ipynb

STAC metadata generated from above notebook (see subset for example), will be able to use a location such as https://${PAVICS_FQDN_PUBLIC}${STAC_DATA_PROXY_URL_PATH}/EuroSAT/... instead of the temporary raw-GitHub content URLs. The STAC populator (with DirectoryLoading implementation), will be able to push the STAC Collection/Items toward that instances. The STAC Assets that they refer to will be placed under ${STAC_DATA_PROXY_DIR_PATH}/EuroSAT to make them accessible externally.

fmigneault commented 1 year ago

@ahandan @ChaamC @francisPLT @Nazim-crim Feel free to review or not. Just giving you the pointer as FYI for future geo projects.

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2244/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1417/

NOTEBOOK TEST RESULTS

    
[2023-11-15T04:22:24.691Z] ============================= test session starts ==============================
[2023-11-15T04:22:24.691Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-15T04:22:24.691Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-15T04:22:24.691Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-15T04:22:24.691Z] collected 255 items
[2023-11-15T04:22:24.691Z] 
[2023-11-15T04:22:36.874Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-15T04:22:44.820Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-15T04:22:56.132Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-15T04:23:05.855Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-15T04:23:13.771Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-15T04:30:38.093Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-15T04:30:38.671Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-15T05:04:08.638Z] ........FxxXXXX                                                          [ 30%]
[2023-11-15T05:04:17.156Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-15T05:04:24.435Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-15T05:04:40.798Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-15T05:04:42.455Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-15T05:04:51.098Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-15T05:04:55.873Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-15T05:09:22.858Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-15T05:10:25.560Z] .............                                                            [ 53%]
[2023-11-15T05:10:30.290Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-15T05:10:32.526Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-15T05:10:48.282Z] .................                                                        [ 64%]
[2023-11-15T05:10:56.142Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-15T05:10:57.532Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-15T05:10:58.841Z] .FFFFFFFF                                                                [ 70%]
[2023-11-15T05:11:09.479Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-15T05:11:18.774Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-15T05:11:20.160Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-15T05:11:22.961Z] ......                                                                   [ 80%]
[2023-11-15T05:11:29.541Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-15T05:11:43.436Z] .............                                                            [ 85%]
[2023-11-15T05:11:53.452Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-15T05:12:27.875Z] ....s.                                                                   [ 88%]
[2023-11-15T05:12:36.019Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-15T05:12:50.793Z] ...                                                                      [ 90%]
[2023-11-15T05:13:03.033Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-15T05:13:28.294Z] ......                                                                   [ 92%]
[2023-11-15T05:13:30.014Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-15T05:16:09.636Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-15T05:16:09.636Z] 
[2023-11-15T05:16:09.637Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2245/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1418/

NOTEBOOK TEST RESULTS

    
[2023-11-15T15:22:12.561Z] ============================= test session starts ==============================
[2023-11-15T15:22:12.561Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-15T15:22:12.561Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-15T15:22:12.561Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-15T15:22:12.561Z] collected 255 items
[2023-11-15T15:22:12.561Z] 
[2023-11-15T15:22:23.257Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-15T15:22:31.669Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-15T15:22:39.512Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-15T15:22:49.612Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-15T15:22:59.310Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-15T15:33:11.595Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-15T15:33:11.595Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-15T15:33:16.133Z] ...............                                                          [ 30%]
[2023-11-15T15:33:25.362Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-15T15:34:15.566Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-15T15:34:17.571Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb EEEEEE           [ 37%]
[2023-11-15T15:34:18.958Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-15T15:34:43.785Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-15T15:34:48.467Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-15T15:38:49.038Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-15T15:40:03.553Z] .............                                                            [ 53%]
[2023-11-15T15:40:05.468Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-15T15:40:07.686Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-15T15:40:25.354Z] .................                                                        [ 64%]
[2023-11-15T15:40:33.703Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-15T15:40:35.084Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-15T15:40:36.405Z] .FFFFFFFF                                                                [ 70%]
[2023-11-15T15:40:46.543Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-15T15:40:55.643Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-15T15:40:57.567Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-15T15:41:00.395Z] ......                                                                   [ 80%]
[2023-11-15T15:41:06.991Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-15T15:41:21.664Z] .............                                                            [ 85%]
[2023-11-15T15:41:31.675Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-15T15:42:05.583Z] ....s.                                                                   [ 88%]
[2023-11-15T15:42:13.720Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-15T15:42:27.853Z] ...                                                                      [ 90%]
[2023-11-15T15:42:40.094Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-15T15:43:04.621Z] ......                                                                   [ 92%]
[2023-11-15T15:43:07.636Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-15T15:45:42.011Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-15T15:45:42.011Z] 
[2023-11-15T15:45:42.011Z] ==================================== ERRORS ====================================

fmigneault commented 1 year ago

run tests

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2252/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1422/

NOTEBOOK TEST RESULTS

    
[2023-11-16T21:18:20.710Z] ============================= test session starts ==============================
[2023-11-16T21:18:20.710Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-16T21:18:20.710Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-16T21:18:20.710Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-16T21:18:20.710Z] collected 255 items
[2023-11-16T21:18:20.710Z] 
[2023-11-16T21:18:31.765Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-16T21:18:39.650Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-16T21:18:48.213Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-16T21:18:56.640Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-16T21:19:03.706Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ......FF       [ 19%]
[2023-11-16T21:26:40.307Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-16T21:26:42.850Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-16T21:26:52.589Z] ...............                                                          [ 30%]
[2023-11-16T21:27:02.489Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-16T21:27:09.861Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-16T21:27:26.298Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-16T21:27:27.679Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-16T21:27:32.697Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-16T21:27:37.203Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-16T21:31:34.586Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-16T21:32:41.366Z] .............                                                            [ 53%]
[2023-11-16T21:32:43.283Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-16T21:32:45.614Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-16T21:33:01.830Z] .................                                                        [ 64%]
[2023-11-16T21:33:09.706Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-16T21:33:11.095Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-16T21:34:13.424Z] ......FFF                                                                [ 70%]
[2023-11-16T21:34:20.947Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-16T21:34:29.668Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-16T21:34:31.055Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-16T21:34:34.126Z] ......                                                                   [ 80%]
[2023-11-16T21:34:40.720Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-16T21:34:54.605Z] .............                                                            [ 85%]
[2023-11-16T21:35:04.598Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-16T21:35:48.633Z] ....s.                                                                   [ 88%]
[2023-11-16T21:35:56.764Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-16T21:36:11.566Z] ...                                                                      [ 90%]
[2023-11-16T21:36:23.802Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-16T21:36:43.552Z] ......                                                                   [ 92%]
[2023-11-16T21:36:46.838Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-16T21:39:21.255Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-16T21:39:21.255Z] 
[2023-11-16T21:39:21.255Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2286/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1426/

NOTEBOOK TEST RESULTS

    
[2023-11-21T17:59:38.082Z] ============================= test session starts ==============================
[2023-11-21T17:59:38.082Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-21T17:59:38.082Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-21T17:59:38.082Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-21T17:59:38.082Z] collected 255 items
[2023-11-21T17:59:38.082Z] 
[2023-11-21T17:59:48.218Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-21T17:59:56.645Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-21T18:00:07.683Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-21T18:00:18.408Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-21T18:00:25.515Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-21T18:07:59.580Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-21T18:07:59.580Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-21T18:08:00.102Z] ...............                                                          [ 30%]
[2023-11-21T18:08:09.746Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-21T18:08:17.649Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-21T18:08:34.671Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-21T18:08:36.318Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-21T18:08:41.328Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-21T18:08:46.086Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-21T18:12:33.845Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-21T18:13:49.866Z] .............                                                            [ 53%]
[2023-11-21T18:13:51.511Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-21T18:13:53.726Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-21T18:14:09.355Z] .................                                                        [ 64%]
[2023-11-21T18:14:17.692Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-21T18:14:19.076Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-21T18:14:42.470Z] .........                                                                [ 70%]
[2023-11-21T18:14:51.782Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-21T18:15:01.117Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-21T18:15:02.495Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-21T18:15:06.176Z] ......                                                                   [ 80%]
[2023-11-21T18:15:14.395Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-21T18:15:31.189Z] .............                                                            [ 85%]
[2023-11-21T18:15:41.176Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-21T18:16:14.814Z] ....s.                                                                   [ 88%]
[2023-11-21T18:16:22.957Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-21T18:16:36.236Z] ...                                                                      [ 90%]
[2023-11-21T18:16:51.230Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-21T18:17:13.756Z] ......                                                                   [ 92%]
[2023-11-21T18:17:15.111Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-21T18:19:54.705Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-21T18:19:54.705Z] 
[2023-11-21T18:19:54.705Z] ================= 254 passed, 1 skipped in 1214.79s (0:20:14) ==================

fmigneault commented 1 year ago

@tlvu Friendly reminder regarding this PR.

If everything looks good, that will help me synchronize merges with other STAC-related PRs to avoid conflicts.

dchandan commented 1 year ago

I am not able to see how this feature meshes with the functionality for controlling the mount points of THREDDS data, and support for multiple THREDDS catalog that was recently provided in #359. The variables STAC_DATA_PROXY_DIR_PATH and STAC_DATA_PROXY_URL_PATH appear to be doing something similar to what is done by THREDDS_DATASET_LOCATION_ON_HOST, THREDDS_SERVICE_DATA_LOCATION_ON_HOST, THREDDS_DATASET_URL_PATH and THREDDS_SERVICE_DATA_URL_PATH variables from #359. From my limited understanding so far, I get the feeling that this is meant to support access to data that is not stored on THREDDS? Could you provide some use cases that will further illustrate the use case?

fmigneault commented 1 year ago

@dchandan
Indeed. It is for other data not hosted through THREDDS.

An example would be the GeoTiff, GeoJSON and PNG asset files under https://github.com/ai-extensions/stac-data-loader/tree/main/data/EuroSAT that are referenced by the STAC Collection/Items. At the moment, it is already possible to push those STAC definitions (eg: https://hirondelle.crim.ca/stac/collections/EuroSAT-subset-train), but the assets they refer to must make use of an external host source (temporarily using GitHub raw content endpoint for now, which is not optimal). If the node is a data provider, it should have a way to host its own data without necessarily relying on an external hosting service.

dchandan commented 1 year ago

I see. But then why not put that data on THREDDS on the node itself? In your example, it would mean putting the asset files from EuroSAT onto hirondelle's THREDDS catalog.

fmigneault commented 1 year ago

THREDDS makes sense in the case of netCDF and NcML since there are built-in visualization and formatting services for them, but it is an additional complexity layer for simple EO imagery and other potential data sources that could be represented by STAC.

dchandan commented 1 year ago

Yeah, I can see that. Although any data can be served through THREDDS, it does do better with climate-date related formats. But then this opens up a more broader question of if (and I am in favour of doing this) and how we should provide a more comprehensive file server based on nginx. If this PR essentially and completely achieves that, then I think we should rename it to something more general rather than stac-data-proxy.

dchandan commented 1 year ago

Maybe something like birdhouse-fileserver-proxy?

tlvu commented 1 year ago

we should provide a more comprehensive file server based on nginx. If this PR essentially and completely achieves that, then I think we should rename it to something more general rather than stac-data-proxy.

Maybe something like birdhouse-fileserver-proxy?

I sort of agree with @dchandan here. Is this new URL and location on disk meant only for STAC data? From the implementation, it looks pretty generic so it could be used for all sort of static data, not just STAC data.

So, in the future, if we need for "some other kind of data" we won't need to again change the config, or to go through PRs review again.

It's like the new ability to add any new numbers of Thedds collections, there is no need to modify the code anymore. So more flexibility and less "paperwork time" in PR reviews.

This stack will hopefully be deployed by many different organisations with very different needs, we should care for flexibility to help meet their needs.

fmigneault commented 1 year ago

@dchandan @tlvu

how we should provide a more comprehensive file server based on nginx. If this PR essentially and completely achieves that, then I think we should rename it to something more general rather than stac-data-proxy.

In a way, there is nothing more that needs to be done than adding a custom alias to the nginx configs to have as many mount points as desired. Even using secured access does not need any additional service in Magpie (as this PR shows, secure-data-proxy is simply reused).

The issue with combining everything under a common location/directory/component is that it can limit configuration mounts of the server. For now, only a simple Nginx alias is defined, but it could get much more complicated than that to control specific min/max contents-length/type and HTTP methods, and I would prefer to have a dedicated nginx configuration for a controlled use case than many variables for each use case. In the future, I would prefer to avoid the kind of defaults as below because of 1 service's use case / limitations. It is better to leave the option to mount each service's data individually for maximum flexibility.

https://github.com/bird-house/birdhouse-deploy/blob/cc8dc1f7c1e62a8b8d7892f1c1714363fd002cd3/birdhouse/config/proxy/nginx.conf.template#L27-L34

Another reason that I named the component stac-data-proxy is due to the importance of STAC compared to other generic data endpoints (assuming stac is enabled by default following @mishaschwartz PR). For the moment, permissions are somewhat generic, but I foresee they could become much more STAC-specific in follow-up PRs.

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2309/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1439/

NOTEBOOK TEST RESULTS

    
[2023-11-27T23:24:12.802Z] ============================= test session starts ==============================
[2023-11-27T23:24:12.802Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-27T23:24:12.802Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-27T23:24:12.802Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-27T23:24:12.802Z] collected 255 items
[2023-11-27T23:24:12.802Z] 
[2023-11-27T23:24:23.299Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-27T23:24:31.741Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-27T23:24:40.217Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-27T23:24:50.197Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-27T23:24:57.258Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-27T23:32:28.996Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-27T23:32:30.908Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-27T23:32:38.405Z] ...............                                                          [ 30%]
[2023-11-27T23:32:48.123Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-27T23:32:59.739Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-27T23:33:15.480Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-27T23:33:17.123Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-27T23:33:22.355Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-27T23:33:26.878Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-27T23:37:24.339Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-27T23:38:40.454Z] .............                                                            [ 53%]
[2023-11-27T23:38:42.638Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-27T23:38:44.880Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-27T23:39:02.963Z] .................                                                        [ 64%]
[2023-11-27T23:39:10.158Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-27T23:39:12.082Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-27T23:39:30.818Z] .........                                                                [ 70%]
[2023-11-27T23:39:41.169Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-27T23:39:50.488Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-27T23:39:51.881Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-27T23:39:55.189Z] ......                                                                   [ 80%]
[2023-11-27T23:40:03.326Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-27T23:40:21.076Z] .............                                                            [ 85%]
[2023-11-27T23:40:31.086Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-27T23:41:08.238Z] ....s.                                                                   [ 88%]
[2023-11-27T23:41:16.370Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-27T23:41:31.883Z] ...                                                                      [ 90%]
[2023-11-27T23:41:44.121Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-27T23:42:08.639Z] ......                                                                   [ 92%]
[2023-11-27T23:42:11.122Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-27T23:44:45.257Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-27T23:44:45.257Z] 
[2023-11-27T23:44:45.257Z] ================= 254 passed, 1 skipped in 1233.29s (0:20:33) ==================

mishaschwartz commented 1 year ago

I think that it is misleading to name this component "stac" since it really doesn't have anything to do with STAC (it just serves files that could be referenced by a STAC catalog but doesn't need to be at all).

I think that we should use the birdhouse-fileserver-proxy name instead.

It is better to leave the option to mount each service's data individually for maximum flexibility.

I agree with this.

However, having to make a new optional component every single time we need a new mountpoint is going to be very confusing. It would be better to make this component configurable so that it can support multiple mountpoints with different nginx location definitions if needed.

dchandan commented 1 year ago

Agreed

fmigneault commented 1 year ago

I think that it is misleading to name this component "stac" since it really doesn't have anything to do with STAC (it just serves files that could be referenced by a STAC catalog but doesn't need to be at all).

Component wise, this is true for now with minimal nginx config. However, I believe STAC has an important role being an eventual default component that could justify it. As shown by the component combination table here, the multiple flavors in which STAC can be configured is already somewhat complex depending on use cases the node wants to support. I think we can help new node managers configure their server more efficiently and quickly by providing explicit references to STAC rather than more extra variables to configure.

It would be better to make this component configurable so that it can support multiple mountpoints with different nginx location definitions if needed.

Wouldn't that mean that each of those mount point and nginx configuration will require to define a new component anyway to apply relevant default.env and docker-compose volume mounts of (service-specific) nginx configs? So instead of having each component managing their own alias, data mount and nginx-specific optimizations, we would have a birdhouse-fileserver-proxy dependency with a generic location which the sole purpose is to be overridden by other components variables? I don't think that makes it easier.

Just to make sure my point is understood properly. The idea of having a separate nginx config and component for STAC data specifically is that any relevant optimization for accessing STAC data could be gradually defined for that path. Other optimizations or redirections could be applied as we investigate further what works best for STAC and how users interact with it. This is not the same as the case for THREDDS where all configurable datasets go through THREDDS. Since thredds service becomes the "bottleneck" in that case, it indeed makes more sense to make "pluggable configuration" with variables. Nginx's definitions however are directly the access point of the server. It is possible to provide much more fine-grained control of each endpoint. Because of this, I believe a new user should have something better than a generic BIRDHOUSE_FILESERVER_NGINX_EXTRA_CONF variable. My idea is to provide a predefined set of configurations optimized specifically for STAC (admittedly not much for now, but got to start somewhere), without asking each new node admin to try to optimize it themselves.

I think birdhouse-fileserver-proxy is a good idea on its own, and could be provided as well, but the purpose of this PR is not to have a generic data location. Note also that providing an extra data access point in nginx is as simple as adding:

location /more-custom-data/ {
    alias /data/more-custom-data/;
}

Therefore, a node admin could already very easily add more data access points as needed without the need of an additional component. They are also free to choose if the (eventually) predefined optimization that would be provided by stac-data-proxy are relevant for them or not. It is an optional-component after all.

mishaschwartz commented 1 year ago

Ok I'm fine with that in principle. I'll have a look in detail and leave a review shortly

tlvu commented 1 year ago

For the moment, permissions are somewhat generic, but I foresee they could become much more STAC-specific in follow-up PRs.

@fmigneault I wasn't aware the following up PRs that could be STAC-specific. In this case, I agree to make this new component STAC-specific.

Also I agree it is fairly easy to add new Nginx config to handle new URL so the value to have a generic component for static file is less "valuable".

crim-jenkins-bot commented 12 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2316/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1445/

NOTEBOOK TEST RESULTS

    
[2023-11-29T23:19:56.862Z] ============================= test session starts ==============================
[2023-11-29T23:19:56.862Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-29T23:19:56.862Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-29T23:19:56.862Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-29T23:19:56.862Z] collected 255 items
[2023-11-29T23:19:56.862Z] 
[2023-11-29T23:20:09.087Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-29T23:20:17.032Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-29T23:20:27.609Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-29T23:20:32.366Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb .FFFF.         [ 16%]
[2023-11-29T23:20:43.464Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-29T23:20:59.495Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb FFFFFFFFFFF.        [ 24%]
[2023-11-29T23:21:02.044Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-29T23:21:13.603Z] ...............                                                          [ 30%]
[2023-11-29T23:21:24.689Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-29T23:21:32.056Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-29T23:21:44.267Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb FFFFFF           [ 37%]
[2023-11-29T23:21:45.644Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-29T23:21:51.038Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-29T23:21:55.297Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-29T23:25:59.209Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ...FF.FF.FFF.FF [ 48%]
[2023-11-29T23:25:59.209Z] FFFFFFFFFFFFF                                                            [ 53%]
[2023-11-29T23:26:00.311Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-29T23:26:02.634Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-29T23:26:14.207Z] ....FFFFFF.FFFFFF                                                        [ 64%]
[2023-11-29T23:26:23.217Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-29T23:26:24.648Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-29T23:26:26.000Z] .FFFFFFFF                                                                [ 70%]
[2023-11-29T23:26:36.613Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-29T23:26:45.775Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-29T23:26:51.065Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb F [ 78%]
[2023-11-29T23:26:53.921Z] FFF.F.                                                                   [ 80%]
[2023-11-29T23:26:58.190Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb F [ 80%]
[2023-11-29T23:27:04.582Z] FFF.FF.FFFFF.                                                            [ 85%]
[2023-11-29T23:27:14.577Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb F [ 86%]
[2023-11-29T23:27:21.599Z] FFFFs.                                                                   [ 88%]
[2023-11-29T23:27:29.741Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-29T23:27:45.260Z] ...                                                                      [ 90%]
[2023-11-29T23:28:00.160Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-29T23:28:24.504Z] ......                                                                   [ 92%]
[2023-11-29T23:28:27.249Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-29T23:31:01.427Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-29T23:31:01.427Z] 
[2023-11-29T23:31:01.427Z] =================================== FAILURES ===================================

crim-jenkins-bot commented 12 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2317/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1446/

NOTEBOOK TEST RESULTS

    
[2023-11-29T23:20:30.049Z] ============================= test session starts ==============================
[2023-11-29T23:20:30.050Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-29T23:20:30.050Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-11-29T23:20:30.050Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-29T23:20:30.050Z] collected 255 items
[2023-11-29T23:20:30.050Z] 
[2023-11-29T23:20:40.251Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-29T23:20:48.255Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-29T23:20:58.275Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-29T23:21:04.651Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb .FFFF.         [ 16%]
[2023-11-29T23:21:11.727Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-29T23:21:26.688Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb FFFFFFFFFFF.        [ 24%]
[2023-11-29T23:21:29.260Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-29T23:21:38.352Z] ...............                                                          [ 30%]
[2023-11-29T23:21:49.338Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-29T23:21:56.243Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-29T23:22:08.843Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb FFFFFF           [ 37%]
[2023-11-29T23:22:10.487Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-29T23:22:15.544Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-29T23:22:20.558Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-29T23:25:53.205Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ...FF.FF.FFF.FF [ 48%]
[2023-11-29T23:25:55.195Z] FFFFFFFFFFFFF                                                            [ 53%]
[2023-11-29T23:26:00.172Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-29T23:26:02.387Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-29T23:26:14.298Z] ....FFFFFF.FFFFFF                                                        [ 64%]
[2023-11-29T23:26:23.320Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-29T23:26:24.697Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-29T23:26:42.628Z] .........                                                                [ 70%]
[2023-11-29T23:26:51.997Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-29T23:27:01.613Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-29T23:27:04.911Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb F [ 78%]
[2023-11-29T23:27:07.506Z] FFF.F.                                                                   [ 80%]
[2023-11-29T23:27:11.715Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb F [ 80%]
[2023-11-29T23:27:17.355Z] FFF.FF.FFFFF.                                                            [ 85%]
[2023-11-29T23:27:25.505Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb F [ 86%]
[2023-11-29T23:27:30.626Z] FFFFs.                                                                   [ 88%]
[2023-11-29T23:27:38.752Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-29T23:27:54.327Z] ...                                                                      [ 90%]
[2023-11-29T23:28:09.225Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-29T23:28:33.859Z] ......                                                                   [ 92%]
[2023-11-29T23:28:36.356Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-29T23:31:16.237Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-29T23:31:16.237Z] 
[2023-11-29T23:31:16.237Z] =================================== FAILURES ===================================

fmigneault commented 12 months ago

run tests

crim-jenkins-bot commented 12 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2323/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1451/

NOTEBOOK TEST RESULTS

    
[2023-11-30T17:37:30.791Z] ============================= test session starts ==============================
[2023-11-30T17:37:30.791Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-30T17:37:30.791Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-11-30T17:37:30.791Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-30T17:37:30.791Z] collected 255 items
[2023-11-30T17:37:30.791Z] 
[2023-11-30T17:37:41.475Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-30T17:37:49.638Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-30T17:37:57.490Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-30T17:38:07.552Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-30T17:38:13.513Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 19%]
[2023-11-30T17:47:22.645Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-30T17:47:24.554Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-30T17:47:33.387Z] ...............                                                          [ 30%]
[2023-11-30T17:47:43.844Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-30T17:47:51.148Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-30T17:48:08.682Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-30T17:48:10.332Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-30T17:48:15.624Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-30T17:48:20.831Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-30T17:50:51.359Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-30T17:52:09.524Z] .............                                                            [ 53%]
[2023-11-30T17:52:12.392Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-30T17:52:14.621Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-30T17:52:30.643Z] .................                                                        [ 64%]
[2023-11-30T17:52:37.770Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-30T17:52:39.158Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-30T17:53:16.867Z] .........                                                                [ 70%]
[2023-11-30T17:53:22.709Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-30T17:53:32.497Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-30T17:53:33.890Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-30T17:53:36.989Z] ......                                                                   [ 80%]
[2023-11-30T17:53:43.575Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-30T17:53:58.382Z] .............                                                            [ 85%]
[2023-11-30T17:54:08.375Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-30T17:54:55.255Z] ....s.                                                                   [ 88%]
[2023-11-30T17:55:03.412Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-30T17:55:18.175Z] ...                                                                      [ 90%]
[2023-11-30T17:55:30.413Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-30T17:55:54.919Z] ......                                                                   [ 92%]
[2023-11-30T17:55:57.164Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-30T17:58:31.301Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-30T17:58:31.301Z] 
[2023-11-30T17:58:31.301Z] ================= 254 passed, 1 skipped in 1260.21s (0:21:00) ==================

crim-jenkins-bot commented 12 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2325/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov

crim-jenkins-bot commented 12 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2324/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-data-proxy
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1452/

NOTEBOOK TEST RESULTS

    
[2023-11-30T18:41:26.981Z] ============================= test session starts ==============================
[2023-11-30T18:41:26.982Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-11-30T18:41:26.982Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-11-30T18:41:26.982Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-11-30T18:41:26.982Z] collected 255 items
[2023-11-30T18:41:26.982Z] 
[2023-11-30T18:41:36.780Z] notebooks-auth/geoserver.ipynb ..................                        [  7%]
[2023-11-30T18:41:45.203Z] notebooks-auth/test_thredds.ipynb ...........                            [ 11%]
[2023-11-30T18:41:53.779Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 14%]
[2023-11-30T18:42:03.767Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 16%]
[2023-11-30T18:42:07.114Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 19%]
[2023-11-30T18:59:48.226Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 24%]
[2023-11-30T18:59:48.226Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 24%]
[2023-11-30T18:59:52.361Z] ...............                                                          [ 30%]
[2023-11-30T19:00:02.523Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 32%]
[2023-11-30T19:00:10.265Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 34%]
[2023-11-30T19:00:26.607Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 37%]
[2023-11-30T19:00:28.004Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 37%]
[2023-11-30T19:00:33.406Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 40%]
[2023-11-30T19:00:38.630Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 42%]
[2023-11-30T19:03:50.836Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 48%]
[2023-11-30T19:04:59.985Z] .............                                                            [ 53%]
[2023-11-30T19:05:01.631Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 54%]
[2023-11-30T19:05:04.234Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 58%]
[2023-11-30T19:05:21.173Z] .................                                                        [ 64%]
[2023-11-30T19:05:28.358Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 67%]
[2023-11-30T19:05:29.739Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 67%]
[2023-11-30T19:05:47.806Z] .........                                                                [ 70%]
[2023-11-30T19:05:57.195Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 75%]
[2023-11-30T19:06:06.538Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-11-30T19:06:07.913Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 78%]
[2023-11-30T19:06:10.996Z] ......                                                                   [ 80%]
[2023-11-30T19:06:17.578Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-11-30T19:06:33.200Z] .............                                                            [ 85%]
[2023-11-30T19:06:43.207Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-11-30T19:07:20.057Z] ....s.                                                                   [ 88%]
[2023-11-30T19:07:28.200Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-11-30T19:07:43.742Z] ...                                                                      [ 90%]
[2023-11-30T19:07:58.640Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-11-30T19:08:22.540Z] ......                                                                   [ 92%]
[2023-11-30T19:08:25.299Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-11-30T19:10:59.974Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-11-30T19:10:59.974Z] 
[2023-11-30T19:10:59.974Z] =================================== FAILURES ===================================

bird-house / birdhouse-deploy

add `optional-components/stac-data-proxy` #403

Overview

Changes

Related Issue / Discussion

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

386

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results