util.py KeyError returned while indexing error codes

[bourque]

@sosey Myself and others (@cmartlin, @mrafelski, @ivastar) are running into an issue where wfc3tools.util.py crashes when attempting to return an error code, e.g.:

  File "/grp/hst/wfc3h/bourque/repository/pyql/pyql/logging/logging_functions.py", line 212, in wrapped
    func(*a, **kw)
  File "/grp/hst/wfc3h/bourque/repository/pyql/pyql/logging/logging_functions.py", line 175, in wrapped
    func(*a, **kw)
  File "cal_uvis_make_darks.py", line 276, in process_darks_main
    preprocess_data_main(paths, postflash)
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/preprocess_data.py", line 87, in preprocess_data_main
    make_blvs_main(directory)
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/make_blvs.py", line 175, in make_blvs_main
    run_calwf3()
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/make_blvs.py", line 126, in run_calwf3
    calwf3(input='dark_asn.fits', save_tmp=True)
  File "/grp/hst/wfc3p/bourque/envs/anaconda3/envs/astroconda3/lib/python3.5/site-packages/wfc3tools/calwf3.py", line 77, in calwf3
    ec = error_code(return_code)
  File "/grp/hst/wfc3p/bourque/envs/anaconda3/envs/astroconda3/lib/python3.5/site-packages/wfc3tools/util.py", line 64, in error_code
    return codes[code]
KeyError: -7

Digging into the valid error codes in wfc3tools.util.py, it does't look like -7 is a choice:

def error_code(code=None):
    """ return the error code text, or all if code is None."""

    codes = {2: "ERROR_RETURN",
             111: "OUT_OF_MEMORY",
             114: "OPEN_FAILED",
             115: "CAL_FILE_MISSING",
             116: "NOTHIN_TO_DO",
             117: "KEYWORD_MISSING",
             118: "ALLOCATION_PROBLEM",
             119: "HEADER_PROBLEM",
             120: "SIZE_MISMATCH",
             130: "CAL_STEP_NOT_DONE",
             141: "TABLE_ERROR",
             142: "COLUMN_NOT_FOUND",
             143: "ELEMENT_NOT_FOUND",
             144: "ROW_NOT_FOUND",
             151: "NO_GOOD_DATA",
             152: "NO_CHIP_FOUND",
             171: "REF_TOO_SMALL",
             999: "INTERNAL_ERROR",
             1001: "INVALID_EXPTIME",
             1011: "INVALID_FILENAME",
             1020: "WRITE_FAILED",
             1021: "INVALID_TEMP_FILE",
             1023: "FILE_NOT_READABLE",
             1025: "COPY_NOT_POSSIBLE",
             1111: "INVALID_VALUE",
             1030: "UNSUPPORTED_APERTURE",
             }

    if code is None:
        return codes
    else:
        return codes[code]

Any idea what might be going on here?

This prompted me to dig into why my call to calwf3 might be triggering an error in the first place. Upon digging, I noticed that though I explicitly set the input variable to point to an association table (which contains the list of files I want to process) e.g ...

from wfc3tools.calwf3 import calwf3
calwf3(input='dark_asn.fits', save_tmp=True)

... calwf3 appears to be gathering any FITS files in the CWD and setting them as the infiles, e.g.:

        infiles, dummy_out = parseinput.parseinput(input)
        logging.info(infiles)
        for image in infiles:
            if not os.path.exists(image):
                raise IOError('Input file not found: ' + image)

        call_list.append(','.join(infiles))

['id0k2wkrq_raw.fits', 'id0k2vi6q_raw.fits', 'id0k2wklq_raw.fits', 'id0k2yn7q_raw.fits', 'id0k2xmiq_raw.fits', 'id0k2vi0q_raw.fits', 'id0k2uclq_raw.fits', 'id0k2zsmq_raw.fits', 'id0k2yn3q_raw.fits', 'id0k2zskq_raw.fits', 'id0k3aupq_raw.fits', 'id0k2xmhq_raw.fits', 'id0k3auoq_raw.fits', 'id0k3fujq_raw.fits', 'id0k3fuiq_raw.fits', 'id0k3ifnq_raw.fits', 'id0k3hcxq_raw.fits', 'id0k3hcwq_raw.fits', 'id0k3ga5q_raw.fits', 'id0k3jfsq_raw.fits', 'id0k3ga1q_raw.fits', 'id0k3jfqq_raw.fits', 'id0k3lq6q_raw.fits', 'id0k3lq5q_raw.fits', 'id0k3qhyq_raw.fits', 'id0k3qhxq_raw.fits', 'id0k3pfhq_raw.fits', 'id0k3tbkq_raw.fits', 'id0k3tbgq_raw.fits', 'id0k3obsq_raw.fits', 'id0k3obrq_raw.fits', 'id0k4areq_raw.fits', 'id0k4ardq_raw.fits', 'id0k4dc1q_raw.fits', 'id0k4ec9q_raw.fits', 'id0k4ec5q_raw.fits', 'id0k4jknq_raw.fits', 'id0k4jkiq_raw.fits', 'id0k4ikbq_raw.fits', 'id0k4ijpq_raw.fits']

When I bypass this code and explicitly set call_list, I can avoid an error and calwf3 works fine:

call_list = ['calwf3.e', 'dark_asn.fits', '-s']
proc = subprocess.Popen(
    call_list,
    stderr=subprocess.STDOUT,
    stdout=subprocess.PIPE,
)

Could you advise?

[sosey]

yes, @ivastar emailed me earlier this week and I have a fix for this, but have been conferencing all week. I'll update this when I get a chance

[bourque]

@sosey Great, thanks!

[sosey]

@bourque so these are a couple different issues playing with each other. -7 is a unix return error, not a calwf3 return error. parseinput comes from stsci.tools. It's an attempt to take into account different kinds of input that the user might provide and return a parsed list of expected inputs but the code is then attaching the wrong string to the calwf3 call. I think this section of the code got messed up when I applied a recent pull request. In any case, the code was trying to be helpful and check to make sure the files you want to process actually exist on disk before kicking off, for ASN files, this means opening them and returning the list of members to check

infiles,out= parseinput.parseinput('*raw.fits')

returns this to infiles:

['iaao01k8q_raw.fits',
 'iaao01k9q_raw.fits',
 'iacr51ohq_raw.fits',
 'iacs01t4q_raw.fits',
 'ibh719grq_raw.fits']

while, using an asn file actually open the asn file:

 infiles,out= parseinput.parseinput('iaao01010_asn.fits')

In [34]: infiles
Out[34]: ['iaao01k8q_raw.fits', 'iaao01k9q_flt.fits']

In [35]: out
Out[35]: 'IAAO01011'

It's supposed to return the list of input files from the association table, I'm unsure why in this example, it's returning the raw file for the first member and the processed (flt) file for the second. The output looks good, which is the association output name. Because of this I'm switching to a different function to check the file existence and fixing the bug that uses the list instead of the user input...as well as the fact that the calwf3 executable should only accept one input file...multiple bugs! showing how often this has been used :/

multiple input files (when not in an ASN file) should be taken care of by the user by calling calwf3 from a list.

for image in mylist:
calwf3(image)

check latest pull request out for yourself and see if it's working now?

*\ parseinput starting returning expected output for me, no idea what was messed up so I switched back to that.

[sosey]

@ivastar check it out and see if it works for you as well?

[bourque]

@sosey Thanks for looking into this. The version in your pull request is working!

[sosey]

@bourque great.

[bourque]

@sosey I forgot to ask ... when can I expect to see the changes in astroconda?

[sosey]

@bourque I was going to give @ivastar a couple days to try it out but I can update the version information for astroconda today or tomorrow. Since you've already installed the update locally you shouldn't see any changes if you later do a conda update for wfc3tools. Anyone who wants the update today can pip install from the git master right now, or just do the update after I change the version in astroconda.

[bourque]

@sosey great, thanks!