search

spacetelescope / wfc3tools

Python tools for HST WFC3 data calibration and analysis
https://wfc3tools.readthedocs.io/en/latest/
Other
6 stars 19 forks source link

calwf3 fails when attempting to resume processing on a 'flattened' IMA file if ASN_ID=None #38

Open mackjenn opened 5 years ago

mackjenn commented 5 years ago

Following the methodology described here, the WFC3 team has written a function to remove time-variable background from IR images and improve the ramp fitting. https://github.com/shannnnnyyy/wfc3tools/blob/ir_tools/wfc3tools/wfc3ir_tools.py

The procedure will partially process a RAW exposure through calwf3 (with CRCORR=OMIT), remove the median signal per read from the IMA, and then resume calwf3 on the corrected IMA file, this time with CRCORR=PERFORM, in order to produce an improved FLT image.

The problem is that calwf3 fails the last step when the file is not part of an association. (eg. ASN_ID=NONE). This appears to be a bug in calwf3, which hopefully could be a simple fix?? We need this in order to run our pipeline to generate sky flats using exposures impacted by non-linear background. Tagging individuals involved in this effort @shannnnnyyy @heatherkurtz

mackjenn commented 5 years ago

@mdlpstsci Others in the team suggested that you might be able to help with this fix? We want to release a new wfc3ir tool in the next few months but we need the calwf3 bug fix first. One of us can provide a sample dataset for testing if that helps. Thanks!

mdlpstsci commented 5 years ago

My division (DMD) has gone to a Scrum style of development. What this means to you is that we are currently working in two week sprints where tickets are written and prioritized for work to be done during the two week period. I am currently working on a high priority project, but we are about to deliver (within next two weeks) the first version of software. There are also some important ACS waiting in the queue. This is just FYI.

You have already written the Git Issue which is the start of the process. If you could provide me with a test dataset and a trailer file sitting about from one of your "failed" runs, this would be very helpful. I would like to take a quick look to estimate the amount of work to address this issue which will help in the planning.

mackjenn commented 5 years ago

The calwf3 fix is not urgent yet, and we will keep you informed of the timeline for delivering the new tooks to the user.

In the meantime, I have placed a sample dataset in the following directory: /user/mack/calwf3_asn_bug/

The code has a workaround for files with ASN_ID=None, but we will want to remove that part once the calwf3 bug is fixed.

The pull request is here.