Pipeline: Fix WFC3 drizzlepac error: index out of range in runastrodriz module

[stscijgbot-hstdp]

Issue HLA-1120 was created on JIRA by Steve Goldman:

ib3y01/HAP_ib3y01.err  (48 total)

 

Traceback (most recent call last):

  File "/user/mburger/hap/singlevisits/make_svhap.py", line 137, in

    make_svhap(visit, today, manifest)

  File "/user/mburger/hap/singlevisits/make_svhap.py", line 72, in make_svhap

    runastrodriz.process(asn, debug=True)

  File "/user/mburger/anaconda3/envs/caldp_2023_2_0/lib/python3.9/site-packages/drizzlepac/runastrodriz.py", line 537, in process

    adriz_pars = mdzhandler.getMdriztabParameters(_good_images)

  File "/user/mburger/anaconda3/envs/caldp_2023_2_0/lib/python3.9/site-packages/drizzlepac/mdzhandler.py", line 26, in getMdriztabParameters

    _fileName = files[0]

IndexError: list index out of range

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Matthew Burger I was able to process this dataset on my machine (MacOS) using the latest software.  I used your RAW, FLT, and ASN files.  The error is coming from the pre-SVM processing using runastrodriz and is complaining that it cannot read the calibration reference file from the image header!?  The MDRIZTAB keyword is present in the FLTs, but it is not in the ASN.  The code should be looking in the FLTs.   I cannot see what your make_svhap.py is doing.  Can you either give me permission, send the file via email, or attach the file to this ticket?

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Location of datasets: /ifs/archive/dev/processing/hla/home/mburger/singlevisits//results_2023-09-14_gssstest

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Essentially, Matt is running runastrodriz in debug mode on ib3y01c6q_raw.fits and the processing fails. I have run under Linux (vs Mac) and matched my versions of stwcs (upgraded version) and numpy (downgraded version) as layered on the current release candidate https://github.com/astroconda/astroconda-releases/tree/master/caldp/satandtools]. I cannot reproduce the error! I always get a valid processing.

The apriori and aposteriori subdirectories never get created. I have a question for Warren in Slack.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

@⁣mburger DUH.  I tend to run the processes via command line versus the way you are running the code.  The problem is ib3y01c6q is a singleton and is determined to have bad guiding since the do_verify_guiding parameter is True via the API.  As I was running the code command line, the "-v" for do_verify_guiding is set to False, so I never had a problem.  This inconsistency needs to be fixed.  Also, this bug does expose an issue which is not handled in the code.  Namely, if a singleton is found to have bad guiding AND the force parameter (specifies to create a drizzle product no matter what) is set to false, then there are no images upon which to continue the processing.

(1) Fix the inconsistency between the command line setting and the API for the do_verify_guiding parameter.

(2) Fix handling of singletons when the image was considered to have bad guiding.

I will note in this case, the image does NOT suffer from bad guiding.  The code must be detecting the diffraction spikes around a bright object in the field.

 

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Command line: runastrodriz.py ==> Verification of guiding is OFF be default. It is turned ON using the "-v"option.

API: runastrodriz.process(inFile, do_verify_guiding=True, force=False) ==> Verification of guiding is ON by default. However, if there are no images to process as the images are deemed BAD (i.e., BAD GUIDING), the current code will fail and not nicely as {}force=False{}.

EASY FIXES:

If do_verify_guiding=True and {}force=True{}, then the code will continue to generate a "pipeline" DRZ file.

Let the code fail as there are no images to process, but exit the code nicely and possibly a new exit code.

Matthew Burger Please think about the fixes you prefer for making the interfaces consistent and handling the "no good images"case. If either interface changes, there may be implications for any scripts/programs DMS/Operations uses

 

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

A question I am going to ask you when we have a tag-up (tomorrow after stand-up?) is how you invoke the drizzle portion of the calibration pipeline.  By this I mean do you call runastrodriz.process() or use the command line method runastrodriz.py or ??? I am most interested in any parameters or options, respectively, that you may use.  In particular, I am interested in the bools do_verify_guiding and force for the API.  Corresponding, on the command line these would be -v and -f.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Addressed by Git PR#⁠1682.