error checking up front

[schiebel]

Run_iclean : If you supply “ mask=’auto-multithresh’ “ by mistake, instead of “usemask=’’’’, it does not detect this error until after it completes the first major cycle. It exits with an error, but it would be good to check these things at the start …

[schiebel]

gclean should provide a static method (no self parameter) for checking parameters. For remote execution, this will allow parameters to be checked on the local system in advance of creating a gclean object on a remote system.

This function should throw an exception if there is a problem with any parameters, and it should return a dictionary whose keys are the identifying string for each image or field. These keys will be used as the tab labels in the GUI. The values should be a dictionary which contains all of the fields in the dictionary passed to the update member function along with these additional fields:

• image: path to image file
• mask: path to clean mask file

For remote execution, this function may eventually need to accept a host parameter so that the paths (and other parameters?) can be sensitive to the host where gclean will run (but maybe this is not necessary).

[Kitchi]

Hey @schiebel I just did some prototyping of this function, and it looks like it will be quite an involved and substantial function (that likely needs to be it's own class) in order to catch the intricacies of parameter dependencies.

At the moment these checks for CASA 6 tasks are done in the XML layer, so I suspect the checks here should be done in the same layer. What do you think?

[Kitchi]

If we want to do param checking in here, there are these functions in graphviper that @jrhosk implemented : https://github.com/casangi/graphviper/blob/main/src/graphviper/utils/parameter.py

Writing a wrapper to use some of those functions and return a dict similar to the update function shouldn't be too difficult

[schiebel]

Hey @schiebel I just did some prototyping of this function, and it looks like it will be quite an involved and substantial function (that likely needs to be it's own class) in order to catch the intricacies of parameter dependencies.

Hi @Kitchi , the API is really independent from the implementation so it's no problem to create a parameter checking class and just create an instance of that class in a static member function in gclean.

The case where it might be useful for the public API to be an object interface if there was a need to check parameters (outside of gclean et al) one at a time and creating the parameter checking mechanism is too expensive to do it for each call. I've been trying to think of some places where this might be the case but have not come up with any... so if you can think of some let me know...

[schiebel]

Hey @schiebel I just did some prototyping of this function, and it looks like it will be quite an involved and substantial function (that likely needs to be it's own class) in order to catch the intricacies of parameter dependencies.

At the moment these checks for CASA 6 tasks are done in the XML layer, so I suspect the checks here should be done in the same layer. What do you think? ... If we want to do param checking in here, there are these functions in graphviper

If most of the checking is done via checks generated from the tclean XML file. Then parameter checking can be done using a mustache template like this:

#TASK XML> tclean -argfilter=interactive,fullsummary
from casatools.typecheck import validator as _pc
from casatools.coercetype import coerce as _coerce
from casatools.errors import create_error_string

class gclean:

    def __init__( self, {{params}} ):
        print( "gclean validation results:" )
        print( gclean.validate( { {{# paramList}}'{{name}}': {{name}}, {{/ paramList}} } ) )

    def validate( params ):
        schema = { {{# paramList}}'{{name}}': {{schema}}, {{/ paramList}} }
        result = _pc.validate( params, { k:v for k,v in schema.items( ) if k in params } )
        return result, None if result else _pc.errors

Using this template, the tclean XML file and xml-casa you can generate the source below by running:

bash$ java -jar casagui/private/__java__/xml-casa-assembly-1.86.jar gclean.mustache casatasks/__xml__/tclean.xml

This generated code (which I put in casagui.private.casashell.gclean) can then be used like this to check one or all parameters against the XML specification:

In [1]: from casagui.private.casashell.gclean import gclean

In [2]: gclean.validate( { 'vis': 'refim_twopoints_twochan.ms' } )
Out[2]: (True, None)

In [3]: gclean.validate( { 'vis': 'Xrefim_twopoints_twochan.ms' } )
Out[3]: 
(False,
 {'Xrefim_twopoints_twochan.ms': ['path not found'],
  "['Xrefim_twopoints_twochan.ms']": ['path vector element not found'],
  'vis': ['must be of cReqPath type', 'must be of cReqPathVec type']})

In [4]: x = gclean( vis='Xrefim_twopoints_twochan.ms', imagename='test', usemask='auto-multithresh', imsize=512, calcres=True, calcpsf=True, stokes='I', cell='12.0arcsec', specmode='cube', interpolation='nearest', nchan=5, start='1.0GHz', width='0.2GHz', pblimit=-1e-05, deconvolver='hogbom', threshold='0.001Jy', niter=50, cycleniter=10, cyclefactor=3, scales=[0,3,10] )
gclean validation results:
(False, {'Xrefim_twopoints_twochan.ms': ['path not found'], "['Xrefim_twopoints_twochan.ms']": ['path vector element not found'], 'vis': ['must be of cReqPath type', 'must be of cReqPathVec type']})

In [5]: x = gclean( vis='refim_twopoints_twochan.ms', imagename='test', usemask='auto-multithresh', imsize=512, calcres=True, calcpsf=True, stokes='I', cell='12.0arcsec', specmode='cube', interpolation='nearest', nchan=5, start='1.0GHz', width='0.2GHz', pblimit=-1e-05, deconvolver='hogbom', threshold='0.001Jy', niter=50, cycleniter=10, cyclefactor=3, scales=[0,3,10] )
gclean validation results:
(True, None)

In [6]: 

Generated Python Source

from casatools.typecheck import validator as _pc
from casatools.coercetype import coerce as _coerce
from casatools.errors import create_error_string

class gclean:

    def __init__( self, vis='', selectdata=True, field='', spw='', timerange='', uvrange='', antenna='', scan='', observation='', intent='', datacolumn='corrected', imagename='', imsize=[ int(100) ], cell=[  ], phasecenter='', stokes='I', projection='SIN', startmodel='', specmode='mfs', reffreq='', nchan=int(-1), start='', width='', outframe='LSRK', veltype='radio', restfreq=[  ], interpolation='linear', perchanweightdensity=True, gridder='standard', facets=int(1), psfphasecenter='', wprojplanes=int(1), vptable='', mosweight=True, aterm=True, psterm=False, wbawp=True, conjbeams=False, cfcache='', usepointing=False, computepastep=float(360.0), rotatepastep=float(360.0), pointingoffsetsigdev=[  ], pblimit=float(0.2), normtype='flatnoise', deconvolver='hogbom', scales=[  ], nterms=int(2), smallscalebias=float(0.0), fusedthreshold=float(0.0), largestscale=int(-1), restoration=True, restoringbeam=[  ], pbcor=False, outlierfile='', weighting='natural', robust=float(0.5), noise='1.0Jy', npixels=int(0), uvtaper=[ '' ], niter=int(0), gain=float(0.1), threshold=float(0.0), nsigma=float(0.0), cycleniter=int(-1), cyclefactor=float(1.0), minpsffraction=float(0.05), maxpsffraction=float(0.8), nmajor=int(-1), usemask='user', mask='', pbmask=float(0.0), sidelobethreshold=float(3.0), noisethreshold=float(5.0), lownoisethreshold=float(1.5), negativethreshold=float(0.0), smoothfactor=float(1.0), minbeamfrac=float(0.3), cutthreshold=float(0.01), growiterations=int(75), dogrowprune=True, minpercentchange=float(-1.0), verbose=False, fastnoise=True, restart=True, savemodel='none', calcres=True, calcpsf=True, psfcutoff=float(0.35), parallel=False ):
        print( "gclean validation results:" )
        print( gclean.validate( { 'vis': vis, 'selectdata': selectdata, 'field': field, 'spw': spw, 'timerange': timerange, 'uvrange': uvrange, 'antenna': antenna, 'scan': scan, 'observation': observation, 'intent': intent, 'datacolumn': datacolumn, 'imagename': imagename, 'imsize': imsize, 'cell': cell, 'phasecenter': phasecenter, 'stokes': stokes, 'projection': projection, 'startmodel': startmodel, 'specmode': specmode, 'reffreq': reffreq, 'nchan': nchan, 'start': start, 'width': width, 'outframe': outframe, 'veltype': veltype, 'restfreq': restfreq, 'interpolation': interpolation, 'perchanweightdensity': perchanweightdensity, 'gridder': gridder, 'facets': facets, 'psfphasecenter': psfphasecenter, 'wprojplanes': wprojplanes, 'vptable': vptable, 'mosweight': mosweight, 'aterm': aterm, 'psterm': psterm, 'wbawp': wbawp, 'conjbeams': conjbeams, 'cfcache': cfcache, 'usepointing': usepointing, 'computepastep': computepastep, 'rotatepastep': rotatepastep, 'pointingoffsetsigdev': pointingoffsetsigdev, 'pblimit': pblimit, 'normtype': normtype, 'deconvolver': deconvolver, 'scales': scales, 'nterms': nterms, 'smallscalebias': smallscalebias, 'fusedthreshold': fusedthreshold, 'largestscale': largestscale, 'restoration': restoration, 'restoringbeam': restoringbeam, 'pbcor': pbcor, 'outlierfile': outlierfile, 'weighting': weighting, 'robust': robust, 'noise': noise, 'npixels': npixels, 'uvtaper': uvtaper, 'niter': niter, 'gain': gain, 'threshold': threshold, 'nsigma': nsigma, 'cycleniter': cycleniter, 'cyclefactor': cyclefactor, 'minpsffraction': minpsffraction, 'maxpsffraction': maxpsffraction, 'nmajor': nmajor, 'usemask': usemask, 'mask': mask, 'pbmask': pbmask, 'sidelobethreshold': sidelobethreshold, 'noisethreshold': noisethreshold, 'lownoisethreshold': lownoisethreshold, 'negativethreshold': negativethreshold, 'smoothfactor': smoothfactor, 'minbeamfrac': minbeamfrac, 'cutthreshold': cutthreshold, 'growiterations': growiterations, 'dogrowprune': dogrowprune, 'minpercentchange': minpercentchange, 'verbose': verbose, 'fastnoise': fastnoise, 'restart': restart, 'savemodel': savemodel, 'calcres': calcres, 'calcpsf': calcpsf, 'psfcutoff': psfcutoff, 'parallel': parallel,  } ) )

    def validate( params ):
        schema = { 'vis': {'anyof': [{'type': 'cReqPath', 'coerce': _coerce.expand_path}, {'type': 'cReqPathVec', 'coerce': [_coerce.to_list,_coerce.expand_pathvec]}]}, 'selectdata': {'type': 'cBool'}, 'field': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'spw': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'timerange': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'uvrange': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'antenna': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'scan': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'observation': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cInt'}]}, 'intent': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'datacolumn': {'type': 'cStr', 'coerce': _coerce.to_str}, 'imagename': {'anyof': [{'type': 'cInt'}, {'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'imsize': {'anyof': [{'type': 'cInt'}, {'type': 'cIntVec', 'coerce': [_coerce.to_list,_coerce.to_intvec]}]}, 'cell': {'anyof': [{'type': 'cIntVec', 'coerce': [_coerce.to_list,_coerce.to_intvec]}, {'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cFloat', 'coerce': _coerce.to_float}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}, {'type': 'cInt'}, {'type': 'cFloatVec', 'coerce': [_coerce.to_list,_coerce.to_floatvec]}]}, 'phasecenter': {'anyof': [{'type': 'cInt'}, {'type': 'cStr', 'coerce': _coerce.to_str}]}, 'stokes': {'type': 'cStr', 'coerce': _coerce.to_str, 'allowed': [ 'I', 'IQUV', 'UV', 'RRLL', 'IQ', 'V', 'pseudoI', 'QU', 'YY', 'RR', 'Q', 'U', 'IV', 'XX', 'XXYY', 'LL' ]}, 'projection': {'type': 'cStr', 'coerce': _coerce.to_str}, 'startmodel': {'type': 'cVariant', 'coerce': [_coerce.to_variant]}, 'specmode': {'type': 'cVariant', 'coerce': [_coerce.to_variant] # <allowed> IS NOT ALLOWED FOR A PARAMETER OF TYPE any
}, 'reffreq': {'type': 'cVariant', 'coerce': [_coerce.to_variant]}, 'nchan': {'type': 'cInt'}, 'start': {'type': 'cVariant', 'coerce': [_coerce.to_variant]}, 'width': {'type': 'cVariant', 'coerce': [_coerce.to_variant]}, 'outframe': {'type': 'cStr', 'coerce': _coerce.to_str}, 'veltype': {'type': 'cStr', 'coerce': _coerce.to_str}, 'restfreq': {'type': 'cVariant', 'coerce': [_coerce.to_variant]}, 'interpolation': {'type': 'cStr', 'coerce': _coerce.to_str, 'allowed': [ 'nearest', 'linear', 'cubic' ]}, 'perchanweightdensity': {'type': 'cBool'}, 'gridder': {'type': 'cStr', 'coerce': _coerce.to_str, 'allowed': [ 'widefield', 'wproject', 'imagemosaic', 'standard', 'awproject', 'wprojectft', 'mosaicft', 'ft', 'ftmosaic', 'mosaic', 'awprojectft', 'gridft' ]}, 'facets': {'type': 'cInt'}, 'psfphasecenter': {'anyof': [{'type': 'cInt'}, {'type': 'cStr', 'coerce': _coerce.to_str}]}, 'wprojplanes': {'type': 'cInt'}, 'vptable': {'type': 'cStr', 'coerce': _coerce.to_str}, 'mosweight': {'type': 'cBool'}, 'aterm': {'type': 'cBool'}, 'psterm': {'type': 'cBool'}, 'wbawp': {'type': 'cBool'}, 'conjbeams': {'type': 'cBool'}, 'cfcache': {'type': 'cStr', 'coerce': _coerce.to_str}, 'usepointing': {'type': 'cBool'}, 'computepastep': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'rotatepastep': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'pointingoffsetsigdev': {'anyof': [{'type': 'cIntVec', 'coerce': [_coerce.to_list,_coerce.to_intvec]}, {'type': 'cFloatVec', 'coerce': [_coerce.to_list,_coerce.to_floatvec]}]}, 'pblimit': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'normtype': {'type': 'cStr', 'coerce': _coerce.to_str}, 'deconvolver': {'type': 'cStr', 'coerce': _coerce.to_str, 'allowed': [ 'clarkstokes_exp', 'mtmfs', 'mem', 'clarkstokes', 'hogbom', 'clark_exp', 'clark', 'asp', 'multiscale' ]}, 'scales': {'anyof': [{'type': 'cIntVec', 'coerce': [_coerce.to_list,_coerce.to_intvec]}, {'type': 'cFloatVec', 'coerce': [_coerce.to_list,_coerce.to_floatvec]}]}, 'nterms': {'type': 'cInt'}, 'smallscalebias': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'fusedthreshold': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'largestscale': {'type': 'cInt'}, 'restoration': {'type': 'cBool'}, 'restoringbeam': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'pbcor': {'type': 'cBool'}, 'outlierfile': {'type': 'cStr', 'coerce': _coerce.to_str}, 'weighting': {'type': 'cStr', 'coerce': _coerce.to_str, 'allowed': [ 'briggsabs', 'briggs', 'briggsbwtaper', 'natural', 'radial', 'superuniform', 'uniform' ]}, 'robust': {'type': 'cFloat', 'coerce': _coerce.to_float, 'min': -2.0, 'max': 2.0}, 'noise': {'type': 'cVariant', 'coerce': [_coerce.to_variant]}, 'npixels': {'type': 'cInt'}, 'uvtaper': {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}, 'niter': {'type': 'cInt'}, 'gain': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'threshold': {'type': 'cVariant', 'coerce': [_coerce.to_variant]}, 'nsigma': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'cycleniter': {'type': 'cInt'}, 'cyclefactor': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'minpsffraction': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'maxpsffraction': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'nmajor': {'type': 'cInt'}, 'usemask': {'type': 'cStr', 'coerce': _coerce.to_str, 'allowed': [ 'user', 'pb', 'auto-multithresh' ]}, 'mask': {'anyof': [{'type': 'cStr', 'coerce': _coerce.to_str}, {'type': 'cStrVec', 'coerce': [_coerce.to_list,_coerce.to_strvec]}]}, 'pbmask': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'sidelobethreshold': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'noisethreshold': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'lownoisethreshold': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'negativethreshold': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'smoothfactor': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'minbeamfrac': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'cutthreshold': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'growiterations': {'type': 'cInt'}, 'dogrowprune': {'type': 'cBool'}, 'minpercentchange': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'verbose': {'type': 'cBool'}, 'fastnoise': {'type': 'cBool'}, 'restart': {'type': 'cBool'}, 'savemodel': {'type': 'cStr', 'coerce': _coerce.to_str, 'allowed': [ 'none', 'virtual', 'modelcolumn' ]}, 'calcres': {'type': 'cBool'}, 'calcpsf': {'type': 'cBool'}, 'psfcutoff': {'type': 'cFloat', 'coerce': _coerce.to_float}, 'parallel': {'type': 'cBool'},  }
        result = _pc.validate( params, { k:v for k,v in schema.items( ) if k in params } )
        return result, None if result else _pc.errors

[schiebel]

A couple extra notes... the line #TASK XML> tclean -argfilter=interactive,fullsummary in the header is for a script I wrote for casagui which will expand all of the mustache files in casagui so a script like this could be useful, but is not necessary and this line doesn't have any affect on the expansion of the template.

The casatools validation support classes for Cerberus parameter checking could easily be copied or moved to a different package to avoid the dependence on casatools . They are all pure python. My impression is that @jrhosk 's implementation also uses Cerberus but I don't know about the details...