Add DQ_DEF extension to science products

[hbushouse]

The pipeline working group would like to have a DQ_DEF table extension appear in all science products (as opposed to just reference files), as a source of information to the user of what the various DQ flag values represent.

https://jira.stsci.edu/browse/JP-135

[philhodge]

Would the DQ_DEF extension be added to _uncal.fits files, or any other file that could have either or both groupdq and pixeldq extensions?

On 03/30/2017 08:52 AM, Howard Bushouse wrote:

The pipeline working group would like to have a DQ_DEF table extension appear in all science products (as opposed to just reference files), as a source of information to the user of what the various DQ flag values represent.

https://jira.stsci.edu/browse/JP-135

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/STScI-JWST/jwst/issues/762, or mute the thread https://github.com/notifications/unsubscribe-auth/AB-FDKNsuDczKN4ICul5yegGN7X9-H1Cks5rq6V6gaJpZM4MuUGL.

[hbushouse]

Normally the _uncal.fits files don't have any DQ flag extensions in them, so I don't see the need to have the DQ_DEF extension in those either. On the rare occurrence when something bad happens to cause missing data, in which case a DQ extension is put into the _uncal.fits file by SDP, we can just make sure they use an established flag value, which will then get picked up and defined by products going through calwebb_sloper. DQ_DEF would only be needed for products that contain either a DQ, PIXELDQ, or GROUPDQ extension.

[stscirij]

I tried to write such a capability a while back (over a year ago). It involved hacking the from_fits and to_fits capabilities of the model infrastructure. When you read a science data file, it translates the DQ extension using the DQ_DEF if one exists, and when writing out a DQ extension, it only wrote bitplanes where bits were set and created a custom DQ_DEF for that file. It was clunky and I remember having some weird issues with unexpected behaviour, but I think the idea was sound.

[hbushouse]

If it would make the implementation easier, it wouldn't hurt to just always write the entire list of defined bits, rather than just the ones that were in use in that file. I think users would appreciate always seeing the full list anyway. No collapsing of the actual flag values in the dq extensions would be needed in that case either. Just leave them set to the "master list" values for each condition. The idea of adding the DQ_DEF is not to save space, by compressing the dq extensions, but rather just to provide information to the users (so that they don't have to go looking for documentation in some other place).