search

bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform
https://birdhouse-deploy.readthedocs.io/en/latest/
Apache License 2.0
4 stars 6 forks source link

weaver: bump version to 5.1.1 #439

Closed mishaschwartz closed 2 months ago

mishaschwartz commented 5 months ago

Overview

See changes in the Weaver changes file

Changes

Non-breaking changes

Breaking changes

Related Issue / Discussion

Additional Information

CI Operations

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

mishaschwartz commented 5 months ago

run tests

mishaschwartz commented 5 months ago

@fmigneault any idea why the jenkins tests aren't running for this one? It would be useful to double check that none of the weaver tests fail with the update.

fmigneault commented 5 months ago

run tests

fmigneault commented 5 months ago

@mishaschwartz Can't figure out why it is not triggered for this PR. The Jenkins CI does not show any build referring to this PR. One would have to inspect the GitHub hooks if an event was sent, which is what the CI is listening for. I do not have the necessary access to inspect hooks from GitHub.

As a workaround in the meantime, I triggered it manually from Jenkins: http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2560/

crim-jenkins-bot commented 5 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2560/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : bump-weaver-5.1.0
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1562/

NOTEBOOK TEST RESULTS 
    
[2024-03-25T17:52:10.683Z] ============================= test session starts ==============================
[2024-03-25T17:52:10.683Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-03-25T17:52:10.683Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-03-25T17:52:10.683Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-03-25T17:52:10.683Z] collected 270 items
[2024-03-25T17:52:10.683Z] 
[2024-03-25T17:52:23.306Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-03-25T17:53:02.789Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-03-25T17:53:08.116Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-03-25T17:53:59.728Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-03-25T17:54:08.309Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-03-25T17:54:18.290Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-03-25T17:54:28.341Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-03-25T18:01:51.781Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-03-25T18:01:51.781Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-03-25T18:01:58.506Z] ...............                                                          [ 34%]
[2024-03-25T18:02:07.203Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-03-25T18:02:15.840Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-03-25T18:02:31.630Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-03-25T18:02:43.190Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2024-03-25T18:02:47.432Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-03-25T18:06:28.369Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-03-25T18:07:42.294Z] .............                                                            [ 55%]
[2024-03-25T18:07:44.469Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-03-25T18:07:46.427Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-03-25T18:08:03.176Z] .................                                                        [ 66%]
[2024-03-25T18:08:10.611Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 68%]
[2024-03-25T18:08:11.996Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-03-25T18:08:28.996Z] ........F                                                                [ 72%]
[2024-03-25T18:08:39.036Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-03-25T18:08:48.047Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 78%]
[2024-03-25T18:08:49.429Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-03-25T18:08:52.996Z] ......                                                                   [ 81%]
[2024-03-25T18:09:01.126Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2024-03-25T18:09:16.488Z] .............                                                            [ 86%]
[2024-03-25T18:09:26.516Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-03-25T18:10:11.143Z] ....s.                                                                   [ 89%]
[2024-03-25T18:10:17.739Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-03-25T18:10:33.277Z] ...                                                                      [ 90%]
[2024-03-25T18:10:45.504Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-03-25T18:11:07.818Z] ......                                                                   [ 93%]
[2024-03-25T18:11:10.551Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-03-25T18:13:44.928Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-03-25T18:13:44.928Z] 
[2024-03-25T18:13:44.928Z] =================================== FAILURES ===================================
fmigneault commented 5 months ago

@mishaschwartz Only the final cell of https://github.com/Ouranosinc/pavics-sdi/blob/master/docs/source/notebook-components/weaver_example.ipynb is failing. The cell checks that the status is succeeded from Weaver's point of view, so I assume the underlying Hummingbird ncdump process dumped a plain Error: generating ncdump failed. Output: [... actual NetCDF contents ...] in the text file, or that more stderr/stdout get captured somehow and piped to that file. Needs more investigation, but I don't have anything obvious in mind that can pinpoint the cause. 

14:13:44   mismatch 'stdout'
14:13:44  
14:13:44   assert reference_output == test_output failed:
14:13:44  
14:13:44    '\nJob was su...e" ;\n}\n\n\n' == '\nJob was su...e" ;\n}\n\n\n'
14:13:44    Skipping 194 identical leading characters in diff, use -v to show
14:13:44      content:
14:13:44      
14:13:44    - Error: generating ncdump failed. Output: netcdf ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc {
14:13:44    ? -----------------------------------------
14:13:44    + netcdf ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc {
14:13:44      dimensions:
14:13:44        time = UNLIMITED ; // (120 currently)
14:13:44        bnds = 2 ;
14:13:44        lat = 160 ;
14:13:44        lon = 320 ;
14:13:44        plev = 23 ;
14:13:44      variables:
14:13:44        double time(time) ;
14:13:44            time:bounds = "time_bnds" ;
14:13:44            time:units = "days since 1981-01-01" ;
14:13:44            time:calendar = "standard" ;
14:13:44            time:axis = "T" ;
14:13:44            time:long_name = "time" ;
14:13:44            time:standard_name = "time" ;
14:13:44        double time_bnds(time, bnds) ;
14:13:44        double plev(plev) ;
14:13:44            plev:units = "Pa" ;
14:13:44            plev:axis = "Z" ;
14:13:44            plev:positive = "down" ;
14:13:44            plev:long_name = "pressure" ;
14:13:44            plev:standard_name = "air_pressure" ;
14:13:44        double lat(lat) ;
14:13:44            lat:bounds = "lat_bnds" ;
14:13:44            lat:units = "degrees_north" ;
14:13:44            lat:axis = "Y" ;
14:13:44            lat:long_name = "latitude" ;
14:13:44            lat:standard_name = "latitude" ;
14:13:44        double lat_bnds(lat, bnds) ;
14:13:44        double lon(lon) ;
14:13:44            lon:bounds = "lon_bnds" ;
14:13:44            lon:units = "degrees_east" ;
14:13:44            lon:axis = "X" ;
14:13:44            lon:long_name = "longitude" ;
14:13:44            lon:standard_name = "longitude" ;
14:13:44        double lon_bnds(lon, bnds) ;
14:13:44        float ta(time, plev, lat, lon) ;
14:13:44            ta:standard_name = "air_temperature" ;
14:13:44            ta:long_name = "Air Temperature" ;
14:13:44            ta:units = "K" ;
14:13:44            ta:original_name = "T" ;
14:13:44            ta:cell_methods = "time: mean (interval: 30 minutes)" ;
14:13:44            ta:cell_measures = "area: areacella" ;
14:13:44            ta:history = "DATE_TIME_TZ altered by CMOR: replaced missing value flag (-9.99e+33) with standard missing value (1e+20)." ;
14:13:44            ta:missing_value = 1.e+20f ;
14:13:44            ta:_FillValue = 1.e+20f ;
14:13:44            ta:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_MRI-CGCM3_decadal1980_r0i0p0.nc areacella: areacella_fx_MRI-CGCM3_decadal1980_r0i0p0.nc" ;
14:13:44      
14:13:44      // global attributes:
14:13:44            :institution = "MRI (Meteorological Research Institute, Tsukuba, Japan)" ;
14:13:44            :institute_id = "MRI" ;
14:13:44            :experiment_id = "decadal1980" ;
14:13:44            :source = "MRI-CGCM3 2011 atmosphere: GSMUV (gsmuv-110112, TL159L48); ocean: MRI.COM3 (MRICOM-3_0-20101116, 1x0.5L51); sea ice: MRI.COM3; land: HAL (HAL_cmip5_v0.31_04); aerosol: MASINGAR-mk2 (masingar_mk2-20110111_0203, TL95L48)" ;
14:13:44            :model_id = "MRI-CGCM3" ;
14:13:44            :forcing = "GHG, SA, Oz, LU, Sl, Vl, BC, OC (GHG includes CO2, CH4, N2O, CFC-11, CFC-12, and HCFC-22)" ;
14:13:44            :parent_experiment_id = "N/A" ;
14:13:44            :parent_experiment_rip = "N/A" ;
14:13:44            :branch_time = 0. ;
14:13:44            :contact = "Seiji Yukimoto (yukimoto@mri-jma.go.jp)" ;
14:13:44            :history = "Output from /sharex3/cmip5/decadal1980/run-C3_decadal1980_01a/grads/atm_avr_mon.ctl DATE_TIME_TZ CMOR rewrote data to comply with CF standards and CMIP5 requirements." ;
14:13:44            :references = "Model described by Yukimoto et al. (Technical Report of the Meteorological Research Institute, 2011, 64, 83pp.)" ;
14:13:44            :initialization_method = 1 ;
14:13:44            :physics_version = 1 ;
14:13:44            :tracking_id = "ce91e727-5f22-44fc-b24d-5bb53393ac69" ;
14:13:44            :product = "output" ;
14:13:44            :experiment = "10- or 30-year run initialized in year 1980" ;
14:13:44            :frequency = "mon" ;
14:13:44            :creation_date = "DATE_TIME_TZ" ;
14:13:44            :Conventions = "CF-1.4" ;
14:13:44            :project_id = "CMIP5" ;
14:13:44            :table_id = "Table Amon (26 July 2011) 976b7fd1d9e1be31dddd28f5dc79b7a1" ;
14:13:44            :title = "MRI-CGCM3 model output prepared for CMIP5 10- or 30-year run initialized in year 1980" ;
14:13:44            :parent_experiment = "N/A" ;
14:13:44            :modeling_realm = "atmos" ;
14:13:44            :realization = 1 ;
14:13:44            :cmor_version = "2.7.1" ;
14:13:44            :DODS_EXTRA.Unlimited_Dimension = "time" ;
14:13:44      }
14:13:44
mishaschwartz commented 5 months ago

@fmigneault

Needs more investigation, but I don't have anything obvious in mind that can pinpoint the cause.

Ok thanks! I've been investigating on my side but I haven't been able to get a consistent result to debug. If you have some time in the near future could you please have a look as well?

mishaschwartz commented 3 months ago

run tests

crim-jenkins-bot commented 3 months ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2660/
Result :x: FAILURE

BIRDHOUSE_DEPLOY_BRANCH : bump-weaver-5.1.0
DACCS_IAC_BRANCH :
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

:warning: Infrastructure deployment failed. :warning:
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference bump-weaver-5.1.0.

mishaschwartz commented 3 months ago

@fmigneault

I looks like the tests may be failing because the test data was never actually downloaded due to a permission denied error. If you look at the output logs from the jenkins setup (https://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2560/console):

+ [ ! -f testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc ]
+ dirname testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc
+ PARENT_DIRS=testdata/flyingpigeon/cmip3
+ [ ! -d testdata/flyingpigeon/cmip3 ]
+ mkdir -p testdata/flyingpigeon/cmip3
mkdir: cannot create directory ���testdata���: Permission denied
+ curl https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/fileServer/birdhouse/testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc --output testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to create the file 
Warning: testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc: No 
Warning: such file or directory

  0  570k    0  3696    0     0  28253      0  0:00:20 --:--:--  0:00:20 28430
curl: (23) Failure writing output to destination

It seems that the user that runs this stage in the build does not have permission to create directories at the correct location so there is no directory to put the file when it is downloaded with curl.

This is just one example, if you look at the log output you can see that this same error occurs for every file downloaded through the bootstrap-testdata script.

Do you mind checking this out?

fmigneault commented 3 months ago

Doesn't look like there were any changes related to test data setup since https://github.com/bird-house/birdhouse-deploy/pull/439#issuecomment-2018615652, but that one ran. Not sure of the cause, but will wait to see if it can be replicated.

fmigneault commented 3 months ago

run tests http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2661/

mishaschwartz commented 3 months ago

Is it possible that running the tests with the "run tests" comment runs with a different user than if the test is triggered by GitHub?

fmigneault commented 3 months ago

Is it possible that running the tests with the "run tests" comment runs with a different user than if the test is triggered by GitHub?

Yes. There is a specific user for CI triggers. However, it is not that user that is used within the instance doing the operations. This should be the "same" ubuntu user created dynamically for respective instances.