IOOS checker and plugins new installation makes AIMS scripts fail

[lbesnard]

FAIMMS, ANMN NRS DARWIN YONGALA and SOOP TRV scripts runs as CRONTAB everyday. They also use the NETCDF-CHECKER before pushing a new file to the $INCOMING_DIR to prevent bad files to be downloaded and pushed. This used to work with no issues until last tuesday night on both NSP14 and AWS10.

The issue seems to be related to a bad installation of the IOOS CHECKER/plugin system.

On NSP14, run ipython, and paste the following lines

import os
import sys
import tempfile
netcdf_file_path='/mnt/imos-test-data/IMOS/FAIMMS/Heron_Island/Relay_Pole_1/sea_water_temperature@1.7m_channel_26/2009/NO_QAQC/IMOS_FAIMMS_T_20090101T000000Z_HIRP1_FV00.nc'
netcdf_checker_path = os.path.dirname(os.path.realpath(os.environ.get('NETCDF_CHECKER')))
sys.path.insert(0, netcdf_checker_path)
import cchecker

tmp_json_checker_output = tempfile.mkstemp()
test='imos'
return_value, errors = cchecker.ComplianceChecker.run_checker(netcdf_file_path, [test] , 'None', 'normal', tmp_json_checker_output[1], 'json')

This will fail with the following message :

No valid checkers found for tests 'imos'
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
/home/lbesnard/data-services/FAIMMS/REALTIME/<ipython-input-17-eeb824a74047> in <module>()
----> 1 return_value, errors = cchecker.ComplianceChecker.run_checker(netcdf_file_path, [test] , 'None', 'normal', tmp_json_checker_output[1], 'json')

/var/lib/netcdf-checker/compliance_checker/runner.py in run_checker(cls, ds_loc, checker_names, verbose, criteria, output_filename, output_format)
     48 
     49         elif output_format == 'json':
---> 50             groups = cls.json_output(cs, score_groups, output_filename, ds_loc, limit)
     51 
     52         else:

/var/lib/netcdf-checker/compliance_checker/runner.py in json_output(cls, cs, score_groups, output_filename, ds_loc, limit)
    121                     cs.json_output(checker, groups, f, ds_loc, limit)
    122 
--> 123         return groups
    124 
    125     @classmethod

UnboundLocalError: local variable 'groups' referenced before assignment

A bit hard to debug, since the PO box doesn't reflect what is on NSP14 and AWS10. This again shows how important it is to have the PO box similar to what is on prod.

However the issue can be tracked down to this file and line /var/lib/netcdf-checker/compliance_checker/runner.py" Line 34

(Pdb) core_groups = cs.run(ds, 'imos')
No valid checkers found for tests 'imos'
(Pdb) core_groups = cs.run(ds, 'cf')
No valid checkers found for tests 'cf'

It looks like the tests are not known. The checker plugins was probably not installed in the correct way.

I think this should be raised as a VIT, since it means those 3 datasets won't run until this is fixed.

@smancini @pblain

[ghost]

@lbesnard Ok I can see what's happening :)

The following is finding the cchecker.py file and importing it as a module. In the past this happened to work because the script was importing the ComplianceChecker class into it's namespace and you were referencing it through that.

netcdf_checker_path = os.path.dirname(os.path.realpath(os.environ.get('NETCDF_CHECKER')))
sys.path.insert(0, netcdf_checker_path)
import cchecker

This can be greatly simplified by just using the compliance_checker library directly without the middle hop, as follows:

import tempfile

from compliance_checker.runner import ComplianceChecker

netcdf_file_path='/mnt/imos-test-data/IMOS/FAIMMS/Heron_Island/Relay_Pole_1/sea_water_temperature@1.7m_channel_26/2009/NO_QAQC/IMOS_FAIMMS_T_20090101T000000Z_HIRP1_FV00.nc'
tmp_json_checker_output = tempfile.mkstemp()
test='imos'
return_value, errors = ComplianceChecker.run_checker(netcdf_file_path, [test] , 'None', 'normal', tmp_json_checker_output[1], 'json')

[lbesnard]

hum thanks @lwgordonimos but does this work for you ? it doesnt work for me on nsp14

[ghost]

@lbesnard The "checkers" dictionary in the CheckSuite class looks like it is only populated when the _load_all_availablecheckers() class method is called, otherwise it leaves the 'checkers' dictionary as empty and it fails to run.

If I modify runner.py and add an explicit call to the load method(my change prepended with '+' below), it looks like the check passes.

class ComplianceChecker(object):
    """
    Compliance Checker runner class.

    Ties together the entire compliance checker framework, is used from
    the command line or can be used via import.
    """
    @classmethod
    def run_checker(cls, ds_loc, checker_names, verbose, criteria, output_filename='stdout', output_format='stdout'):
        """
        Static check runner.

        @param  ds_loc          Dataset location (url or file)
        @param  checker_names    List of string names to run, should match keys of checkers dict (empty list means run all)
        @param  verbose         Verbosity of the output (0, 1, 2)
        @param  criteria        Determines failure (lenient, normal, strict)
        @param  output_filename Path to the file for output
        @param  output_format   Format of the output

        @returns                If the tests failed (based on the criteria)
        """
        cs = CheckSuite()
+       cs.load_all_available_checkers()

Sure enough, the command line version contains a call to this method before running the check:

cchecker.py

def main():
    # Load all available checker classes
    check_suite = CheckSuite()
    check_suite.load_all_available_checkers()

I will raise it as an upstream bug :)

[ghost]

Hmm, actually, no I won't... it is mentioned near the bottom of this page in the README:

https://github.com/ioos/compliance-checker/

Looks like the expectation is that your code will run that method before running the check... not sure why this isn't built into the call to "run_checker", but it may be intentional for some reason!

With that in mind, try the following(adding the lines marked with '+'):

import tempfile

from compliance_checker.runner import ComplianceChecker, CheckSuite

+cs = CheckSuite()
+cs.load_all_available_checkers()

netcdf_file_path='/mnt/imos-test-data/IMOS/FAIMMS/Heron_Island/Relay_Pole_1/sea_water_temperature@1.7m_channel_26/2009/NO_QAQC/IMOS_FAIMMS_T_20090101T000000Z_HIRP1_FV00.nc'
tmp_json_checker_output = tempfile.mkstemp()
test='imos'
return_value, errors = ComplianceChecker.run_checker(netcdf_file_path, [test] , 'None', 'normal', tmp_json_checker_output[1], 'json')

[lbesnard]

thanks a lot @lwgordonimos Check this out https://github.com/ioos/compliance-checker/commit/cb32e33576003dcfdba7259c6705f537f9302091

This is why ! We pulled out a new version of the checker and it was behaving differently !

[lbesnard]

and for example, this doesn't work with the current version in the PO.

[ghost]

@lbesnard Yes I noticed that change in the code as well looking at the commit history. At least it neatly explains it!

Re PO: no worries, I'll get that sorted out today.