add the QUALITY header keyword to the primary header of reductions

[havok2063]

We have a QUALITY keyword that indicates whether an exposure is good or bad for reductions. It can take on three values, "excellent", "bad", or "test". Currently, this keyword is set, or looked up from hdrFix files, during the raw frame metadata curation. This keyword should be added to the primary header of the reductions and propagated down to all reduced files. It is currently missing and I think we want it. This keyword is different from the DRPQUAL keyword which tracks problems with the reductions.

[ndrory]

Brian,

I think this should be a bitfield to allow Dmitry’s QA scripts to indicate various quality-related details to the downstream software. Is this possible?

N.

— Niv Drory — McDonald Observatory, The University of Texas at Austin https://www.as.utexas.edu/~drory

On Jul 29, 2024, at 11:02, Brian Cherinka @.***> wrote:



We have a QUALITY keyword that indicates whether an exposure is good or bad for reductions. It can take on three values, "excellent", "bad", or "test". Currently, this keyword is set, or looked up from hdrFix files, during the raw frame metadata curation. This keyword should be added to the primary header of the reductions and propagated down to all reduced files. It is currently missing and I think we want it. This keyword is different from the DRPQUAL keyword which tracks problems with the reductions.

— Reply to this email directly, view it on GitHubhttps://github.com/sdss/lvmdrp/issues/125, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJXJJ7LHYFCRK5XON5N526DZOZRRDAVCNFSM6AAAAABLUSSB26VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQZTKOBQGI4DEMI. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[havok2063]

Hi Niv,

I think the bitfield that tracks the reduction quality throughout the pipeline for a given exposure is the DRPQUAL header keyword. Alfredo started setting that up here, but it's not being used yet. https://github.com/sdss/lvmdrp/blob/43eb6ffe309055682dbe484d6e03c89dc5647c16/python/lvmdrp/utils/bitmask.py#L160.

I think we want a separate keyword to indicate holistically whether or not an exposure is good or bad, and should even be reduced at all. That is the QUALITY keyword. The pipeline skips reducing exposures with a bad QUALITY. I thought that is what Dmitry needed. I think we should avoid setting the DRPQUAL flag manually inside the hdrFix files.

I thought Dmitry was working on some separate QA code to flag an exposure as good or bad. But if he's working on the main pipeline QA, then he should just write his QA code into the pipeline and have it update the existing DRPQUAL keyword.

[ndrory]

Brian,

I am referring to Dmitry's QA pipeline output, not the DRPQUAL flags.

Dmitry's QA output is more than just good/bad. We'd like to know why he flagged an exposure. The decision what is fit for reduction and what is not lies within the DRP, not the raw data QA pipeline.

N.

On Jul 30, 2024, at 10:13 AM, Brian Cherinka @.***> wrote:

Hi Niv,

I think the bitfield that tracks the reduction quality throughout the pipeline for a given exposure is the DRPQUAL header keyword. Alfredo started setting that up here, but it's not being used yet. https://github.com/sdss/lvmdrp/blob/43eb6ffe309055682dbe484d6e03c89dc5647c16/python/lvmdrp/utils/bitmask.py#L160.

I think we want a separate keyword to indicate holistically whether or not an exposure is good or bad, and should even be reduced at all. That is the QUALITY keyword. The pipeline skips reducing exposures with a bad QUALITY. I thought that is what Dmitry needed. I think we should avoid setting the DRPQUAL flag manually inside the hdrFix files.

I thought Dmitry was working on some separate QA code to flag an exposure as good or bad. But if he's working on the main pipeline QA, then he should just write his QA code into the pipeline and have it update the existing DRPQUAL keyword.

— Reply to this email directly, view it on GitHubhttps://github.com/sdss/lvmdrp/issues/125#issuecomment-2258596012, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJXJJ7NSLN2233LBUEIJLJDZO6USTAVCNFSM6AAAAABLUSSB26VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJYGU4TMMBRGI. You are receiving this because you commented.Message ID: @.***>

-- Niv Drory -- McDonald Observatory / Dept. of Astronomy The University of Texas at Austin Tel: +1 512 471 6197 http://www.as.utexas.edu/~drory

[havok2063]

I've missed some meetings, so I don't know the full idea behind the intended workflow. Where is his QA pipeline intended to run and intersect with the main pipeline? Is the idea to perform some QA that informs the main DRP whether to reduce that exposure at all?

If you want to capture more information, then it's fine to make it a bitfield. If the intended workflow is that his scripts run at LCO, perform some QA, then update a hdrFix file with this final quality bitfield, and the pipeline uses this info to decide to reduce the exposure or not, then I think that should work. I don't see this working if it's intended to run at Utah after or during regular DRP reductions.

We could repurpose the QUALITY flag into this new QA bitfield. We would only need to make a few small changes to the existing pipeline. In any case, it should still be added to the primary header of the reduced files.

[ndrory]

Brian,

On Jul 30, 2024, at 12:56 PM, Brian Cherinka @.***> wrote:

If you want to capture more information, then it's fine to make it a bitfield. If the intended workflow is that his scripts run at LCO, perform some QA, then update a hdrFix file with this final quality bitfield, and the pipeline uses this info to decide to reduce the exposure or not, then I think that should work.

This is the intended workflow. Just wanting to have more than a yes/no/unknown flag.

N.

-- Niv Drory -- McDonald Observatory / Dept. of Astronomy The University of Texas at Austin Tel: +1 512 471 6197 http://www.as.utexas.edu/~drory