search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

NetCDF checker - different behaviour on same machine #253

Closed lbesnard closed 3 years ago

lbesnard commented 7 years ago

This was tested on the po box. But aws10 could be the same

Running the checker manually on the attached file works fine

IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc.zip

vagrant@packer-virtualbox-iso-1449624312:~$ netcdf-checker -t=imos:1.4 IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc
Running Compliance Checker on the dataset from: IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc

--------------------------------------------------------------------------------
                    The dataset scored 231 out of 231 points                    
                           during the imos:1.4 check                            
--------------------------------------------------------------------------------
All tests passed!

However, 

cp IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc /var/incoming/SOOP/XBT/DM/
input_logf
Jan 11 11:20:06 packer-virtualbox-iso-1449624312 SOOP_XBT_DM: File did not pass all compliance checks, verbose log saved at '/vagrant/src/data-services/log/SOOP_XBT_DM/IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc.20170111-112000.log'
Jan 11 11:20:06 packer-virtualbox-iso-1449624312 SOOP_XBT_DM: Could not process file '/vagrant/src/data-services/tmp/tmp.GukOdR41LG/IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc': NetCDF file does not comply with 'imos:1.4' conventions
Jan 11 11:20:06 packer-virtualbox-iso-1449624312 SOOP_XBT_DM: Moving '/vagrant/src/data-services/tmp/tmp.GukOdR41LG/IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc' -> '/vagrant/src/error/SOOP_XBT_DM/IMOS_SOOP-XBT_T_20160220T155900Z_IX28_FV01_ID-89006621.nc.20170111-112000'

verbose log

Running Compliance Checker on the dataset from: /vagrant/src/data-services/tmp/tmp.rWvCQT9WPd

--------------------------------------------------------------------------------
                    The dataset scored 228 out of 232 points                    
                           during the imos:1.4 check                            
--------------------------------------------------------------------------------
                               Scoring Breakdown:                               

                                 High Priority                                  
--------------------------------------------------------------------------------
    Name                            :Priority: Score
globalattr                              :3:    70/74
qc_var                                  :3:    10/10
var                                     :3:   148/148

                                Medium Priority                                 
--------------------------------------------------------------------------------
    Name                            :Priority: Score
optional_global_attributes              :2:     0/0

--------------------------------------------------------------------------------
                  Reasoning for the failed tests given below:                   

Name                             Priority:     Score:Reasoning
--------------------------------------------------------------------------------
globalattr                             :3:    70/74 :  
    Conventions                        :3:     1/ 2 :  
                                       :3:     0/ 1 : Attribute Conventions
                                                      does't match expected
                                                      pattern '(.*,)?CF-1.6,IMOS
                                                      -1.4(,.*)?'
    data_centre                        :3:     1/ 2 :  
                                       :3:     0/ 1 : Attribute data_centre
                                                      should be equal to
                                                      Australian Ocean Data
                                                      Network (AODN)
    data_centre_email                  :3:     1/ 2 :  
                                       :3:     0/ 1 : Attribute
                                                      data_centre_email should
                                                      be equal to
                                                      info@aodn.org.au
    geospatial_vertical_max            :3:     1/ 2 :  
        match_data                     :3:     0/ 1 : geospatial_vertical_max
                                                      value (0.67) did not match
                                                      maximum value of any
                                                      vertical variable
                                                      {u'DEPTH': 1100.25}

This seems quite bizarre that the same file, running the same imos1.4 checker version gives a total different output depending if it's running manually or via the pipeline.

@smancini @pblain @lwgordonimos

ggalibert commented 7 years ago

To invoke the "real" netCDF checker, you should run compliance-checker rather than netcdf-checker.

Also, compliance-checker --version should show 2.3.1 as on 10-aws.

lbesnard commented 7 years ago

Thanks @ggalibert, I ran compliance-checker but had the same behaviour.