search

bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform
https://birdhouse-deploy.readthedocs.io/en/latest/
Apache License 2.0
4 stars 7 forks source link

Docker compose version missing in ``stac/config/magpie/`` compose file #384

Closed mishaschwartz closed 1 year ago

mishaschwartz commented 1 year ago

Overview

The version: key was not set in the stac/config/magpie/docker-compose-extra.yml file which caused docker-compose to report a version mismatch and fail to start.

Changes

Non-breaking changes

Breaking changes None

Related Issue / Discussion

Additional Information

Links to other issues or sources.

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2084/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-compose-version-stac
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-20.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1325/

NOTEBOOK TEST RESULTS 
    
[2023-09-21T20:13:28.939Z] ============================= test session starts ==============================
[2023-09-21T20:13:28.939Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-21T20:13:28.939Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-21T20:13:28.939Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-21T20:13:28.939Z] collected 237 items
[2023-09-21T20:13:28.939Z] 
[2023-09-21T20:13:36.248Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-21T20:13:50.038Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-21T20:13:59.746Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-21T20:14:02.647Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 13%]
[2023-09-21T20:21:28.473Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-21T20:21:31.033Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 18%]
[2023-09-21T20:21:41.535Z] ...............                                                          [ 25%]
[2023-09-21T20:21:51.125Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-21T20:22:34.552Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 29%]
[2023-09-21T20:22:50.365Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-21T20:22:51.751Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-09-21T20:23:57.506Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-09-21T20:23:57.506Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-09-21T20:28:07.938Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-21T20:29:19.718Z] .............                                                            [ 49%]
[2023-09-21T20:29:24.168Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-21T20:29:26.398Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 54%]
[2023-09-21T20:29:41.774Z] .................                                                        [ 62%]
[2023-09-21T20:29:50.122Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-21T20:29:51.508Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 64%]
[2023-09-21T20:30:00.058Z] .........                                                                [ 68%]
[2023-09-21T20:30:10.626Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-21T20:30:19.363Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 75%]
[2023-09-21T20:30:21.271Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-21T20:30:24.076Z] ......                                                                   [ 78%]
[2023-09-21T20:30:30.698Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-21T20:30:46.196Z] .............                                                            [ 84%]
[2023-09-21T20:30:56.209Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-21T20:31:30.332Z] ....s.                                                                   [ 87%]
[2023-09-21T20:31:38.446Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-21T20:31:50.983Z] ...                                                                      [ 89%]
[2023-09-21T20:32:05.895Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2023-09-21T20:32:27.882Z] ......                                                                   [ 92%]
[2023-09-21T20:32:28.978Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-21T20:35:03.130Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-09-21T20:35:03.130Z] 
[2023-09-21T20:35:03.130Z] =================================== FAILURES ===================================
tlvu commented 1 year ago

Not against adding it explicitly, though. Just wanted to share why it was missing.

I agree the version key is optional. We should remove them all. However, when doing this, all external repos should be done at the same time. And I think we should do it eventually but communications has to very clear with all partners because this is a breaking change.

For now, if we do not include the version key, the entire stack will not start and we have to ensure all changes are not breaking change, unless it is scheduled and communicated very clearly in advance.

mishaschwartz commented 1 year ago

@fmigneault

This is something I intentionally left out

Which version of docker compose are you using though? Because older ones (like the version suggested in the vagrant installation scripts) will complain about the mismatch.

I'm very very happy to update the docker compose version though instead

tlvu commented 1 year ago

Which version of docker compose are you using though? Because older ones (like the version suggested in the vagrant installation scripts) will complain about the mismatch.

The physical host has the same docker-compose version as what vagrant installs, which is also the same version as the autodeploy container.

https://github.com/bird-house/birdhouse-deploy/blob/2045cc5083dc96a64cae41ef2e72caf8f54eeeb7/birdhouse/vagrant-utils/install-docker.sh#L102-L105

The reason to keep all the versions in-sync is documented in the comment above. When there is a mismatch, the containers will re-create themselves all the time, even for a simple ./pavics-compose.sh up -d that should do nothing because no change to any configs.

I'm very very happy to update the docker compose version though instead

Updating the docker-compose version means all the physical hosts of all existing deployments have to do it at the same time.

I was thinking of a solution less disruptive documented here https://github.com/bird-house/birdhouse-deploy/issues/246

All comments about docker-compose update should probably stay in that issue https://github.com/bird-house/birdhouse-deploy/issues/246 to keep everything organized.

fmigneault commented 1 year ago

@mishaschwartz

Which version of docker compose are you using though? Because older ones (like the version suggested in the vagrant installation scripts) will complain about the mismatch.

I'm currently running v2.17.2, but was running 1.2x before. Both complained the same that it could not merge YAMLs with different version values.

fmigneault commented 1 year ago

I've tested on a older instance using docker-compose v1.19 and I do see an invalid "version 1" being used by default due to the missing version. It doesn't break the instance startup, but we can avoid the unnecessary error by adding it explicitly.

crim-jenkins-bot commented 1 year ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2098/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-compose-version-stac
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-166.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov