aodn / python-aodndata

Handlers, destination path functions and any other code specific to a given facility or pipeline
GNU General Public License v3.0
0 stars 0 forks source link

Switch to compliance-checker v3 breaks unittests #15

Closed mhidas closed 6 years ago

mhidas commented 6 years ago

We now have three unittests failing due to test files that were expected to pass the checks, but now fail with checker version 3.

======================================================================
FAIL: test_rsync_manifest_file (test_aodndata.aatams.test_aatams_nrt.TestAatamsNrtHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/aodn/python-aodndata/test_aodndata/aatams/test_aatams_nrt.py", line 29, in test_rsync_manifest_file
    check_params={'checks': ['cf']}
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/aodncore/testlib/handlertest.py", line 39, in run_handler
    self.assertIsNone(handler.error)
AssertionError: ComplianceCheckFailedError("the following files failed the check step: ['IMOS_AATAMS-SATTAG_TSP_20090208T184000Z_Q9900180_FV00.nc']",) is not None

======================================================================
FAIL: test_good_netcdf (test_aodndata.common.test_genericHandler.TestGenericHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/aodn/python-aodndata/test_aodndata/common/test_genericHandler.py", line 50, in test_good_netcdf
    dest_path_function=dest_path_anmn_nrs_realtime)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/aodncore/testlib/handlertest.py", line 39, in run_handler
    self.assertIsNone(handler.error)
AssertionError: ComplianceCheckFailedError("the following files failed the check step: ['IMOS_ANMN-NRS_MT_20161109T231108Z_NRSMAI_FV00_NRSMAI-Surface-21-2016-11-MET-realtime.nc']",) is not None

======================================================================
FAIL: test_good_netcdf (test_aodndata.soop.test_soop_xbt_dm.TestSoopXbtDmHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/aodn/python-aodndata/test_aodndata/soop/test_soop_xbt_dm.py", line 22, in test_good_netcdf
    check_params={'checks': ['cf', 'imos:1.4']}
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/aodncore/testlib/handlertest.py", line 39, in run_handler
    self.assertIsNone(handler.error)
AssertionError: ComplianceCheckFailedError("the following files failed the check step: ['IMOS_SOOP-XBT_T_20160106T110100Z_PX02_FV01_ID-150739.nc']",) is not None
mhidas commented 6 years ago

The specific issues the checker complains about are as below.

TestGenericHandler

§2.6.2 Recommended Global Attributes   :2:     1/ 2 : global attribute history
                                                      should exist and be a non-
                                                      empty string

This is easy to fix.

TestAatamsNrtHandler

§2.4 Dimension Order                   :2:     4/ 7 : PRES's dimensions are not
                                                      in the recommended order
                                                      T, Z, Y, X. They are PRES,
                                                      INSTANCE, TEMP's
                                                      dimensions are not in the
                                                      recommended order T, Z, Y,
                                                      X. They are PRES,
                                                      INSTANCE, PSAL's
                                                      dimensions are not in the
                                                      recommended order T, Z, Y,
                                                      X. They are PRES, INSTANCE

§5.0 multidimensional coordinate PRES s:2:     0/ 1 : PRES shares the same name
                                                      as one of its dimensions

The easiest way to fix both of these issues is to remove the INSTANCE dimension (leaving the variable). Since it has a length of 1, it's not needed. Alternatively, if you want to keep this dimension, allowing for the possibility of it being longer than 1, it needs to be the first dimension of all those variables, before any space or time dimensions (see CF for more info). In that case, however, the PRES dimension needs to be named differently, e.g. LEVEL or Z.

TestSoopXbtDmHandler

§2.4 Dimension Order                   :2:     9/11 : TEMP's dimensions are not
                                                      in the recommended order
                                                      T, Z, Y, X. They are
                                                      DEPTH, INSTANCE,
                                                      TEMP_quality_control's
                                                      dimensions are not in the
                                                      recommended order T, Z, Y,
                                                      X. They are DEPTH,
                                                      INSTANCE

§2.6.2 Recommended Attributes          :2:     4/ 5 : institution should be
                                                      defined

The first of these is the same as for AATAMS (remove INSTANCE dimension, or switch order). The second is simply a missing global attribute.

mhidas commented 6 years ago

@lbesnard @bpasquer @xhoenner I'm happy to fix up the test files, but the above fixes will also need to be applied to the actual data files coming through, otherwise they will all fail the CF checks.

mhidas commented 6 years ago

Fixed in #17