Provide monitor that fiducials are turned off.

[deisenstein]

Exposure 71870 appears to have failed to turn off the fiducials, leaving a bump centered at 4600 A (about 4400-4850 in full) in 3 spots on the FP.

This should be easy to filter for, e.g., Flux(4550,4650) - Flux(4350,4400) - Flux(4850,4900).

A camfiber-style focal plane plot of that function would show the blobs. But we also should then look for some single statistic of the image that could be reflected on the exposure list or summary page.

This is an example of how we might want to generate other "bands" in the camfiber list besides just coadding whole cameras. For instance, the formal r-band might be useful. Or features associated to dome lights or to arc lamps templates.

[sbailey]

+1 for automating the identification of this problem, but it is so rare that I'm not sure it deserves its own plot or own column in the summary table. Perhaps we could introduce a new concept of QA to be displayed on the summary page, recommending that the DQS investigate more. e.g. if everything is fine, nothing is listed, but if this trips threshold, a bullet would be added saying

• 4600 A contamination on N fibers; check focal plane plots

Going beyond this ticket, other possible entries could be things like:

• very high readnoise in CCD B amp D; check CCD images
• spectrograph 3 has 50% lower throughput than expected; check Hartmann doors
• No arc lines identified in ARC exposure; are arc lamps on?
• 30% of fibers are reported as not being on target

[deisenstein]

I definitely agree that it would be better to keep the summary page as efficient as possible, and so when we can get things down to a single phrase of text rather than a full plot, that is better.

Whether that text should only appear when something is wrong, versus having a message that indicates good status, I'm not sure. Personally I like the all's well message, so that I know it hasn't silently failed. But obviously that can't be 20+ such messages.

On Tue, Jan 12, 2021 at 1:59 PM Stephen Bailey notifications@github.com wrote:

+1 for automating the identification of this problem, but it is so rare that I'm not sure it deserves its own plot or own column in the summary table. Perhaps we could introduce a new concept of QA to be displayed on the summary page, recommending that the DQS investigate more. e.g. if everything is fine, nothing is listed, but if this trips threshold, a bullet would be added saying

• 4600 A contamination on N fibers; check focal plane plots

Going beyond this ticket, other possible entries could be things like:

• very high readnoise in CCD B amp D; check CCD images
• spectrograph 3 has 50% lower throughput than expected; check Hartmann doors
• No arc lines identified in ARC exposure; are arc lamps on?
• 30% of fibers are reported as not being on target

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/desihub/nightwatch/issues/213#issuecomment-758868410, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPQRGRRIOVWV3JZPITV3LTSZSL2HANCNFSM4V7P7G4A .

[sbailey]

A table/grid with 20+ green check marks on the per-exposure status page with additional text provided only if they fail (red X, yellow !, + text) could be fine. That is more viable than adding 20+ columns to the top-level table of exposures.

[deisenstein]

Definitely. We could have one column on the top table that get marked if any warning fails (or that lists the number of warnings).

But this raises a data model question, in that I'm not clear how you would like such information propagated from the processing half of the code to the display half. I think it would be best for per-fiber or per-chip/amp numbers to end up in those HDU tables. But also the code should compute summary statistics. Do you want those as extra key/value pairs in the top HDU?

And while numbers obviously should get recorded, if we arrive at boolean values from half a dozen stats, would you prefer that these each get a separate keyword? Or that there be a single keyword that holds a bitfield?

On Tue, Jan 12, 2021 at 4:50 PM Stephen Bailey notifications@github.com wrote:

A table/grid with 20+ green check marks on the per-exposure status page with additional text provided only if they fail (red X, yellow !, + text) could be fine. That is more viable than adding 20+ columns to the top-level table of exposures.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/desihub/nightwatch/issues/213#issuecomment-759056526, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPQRGVCOXB5O2TROQ4AXLDSZS735ANCNFSM4V7P7G4A .

[sbailey]

Nightwatch qa-EXPID.fits files have HDUs for PER_AMP, PER_CAMFIBER, PER_CAMERA, PER_FIBER for tracking QA specific to each of those things. Following that structure, we could add a PER_EXPOSURE HDU with global metrics or booleans. That would be a one-row table per file with columns NIGHT, EXPID, +one column per metric (even though a single-row table seems a bit odd, they are designed to be able to stack QA tables across exposures for making timeseries etc.).

i.e. the calculation would belong in the QA portion of the code, would be written to the PER_* HDUs, and then the display code would use the info from that file without doing calculation itself. Does that make sense?

[deisenstein]

That seems reasonable to me. Question: since the main purpose is support stacking across exposures, should we also echo some of the keywords like exposure number or time into the table columns? Otherwise, I think one would not be able to make sense of a stack.

On Tue, Jan 12, 2021 at 5:31 PM Stephen Bailey notifications@github.com wrote:

Nightwatch qa-EXPID.fits files have HDUs for PER_AMP, PER_CAMFIBER, PER_CAMERA, PER_FIBER for tracking QA specific to each of those things. Following that structure, we could add a PER_EXPOSURE HDU with global metrics or booleans. That would be a one-row table per file with columns NIGHT, EXPID, +one column per metric (even though a single-row table seems a bit odd, they are designed to be able to stack QA tables across exposures for making timeseries etc.).

i.e. the calculation would belong in the QA portion of the code, would be written to the PER_* HDUs, and then the display code would use the info from that file without doing calculation itself. Does that make sense?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/desihub/nightwatch/issues/213#issuecomment-759078741, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPQRGT43SMTHNKBKFFY5UDSZTESHANCNFSM4V7P7G4A .

[sbailey]

Question: since the main purpose is support stacking across exposures, should we also echo some of the keywords like exposure number or time into the table columns?

Yes, e.g. the PER_CAMERA HDU has "metadata" columns NIGHT, EXPID, SPECTRO, CAM, FIBER to orient what is what including stacking across exposures, plus QA metric columns INTEG_RAW_FLUX, MEDIAN_RAW_FLUX, etc. Within a single file, the NIGHT and EXPID are all the same but they are still kept to facilitate stacking. In this case there are 103500 = 15000 rows, one per fiber per camera (b,r,z) per spectrograph (0-9).

For a PER_EXPOSURE HDU, the "metadata" columns would be just NIGHT, EXPID; plus as many QA metric columns as we need, and each file would have just a single row.

Adding a timestamp would be useful too, but that starts taking us down a slippery slope of N>>1 other columns (airmass, sky, seeing, temp, ...) which might be better kept in a separate table and merged as needed.

[deisenstein]

I think that's reasonable, with one "but", which is that we should probably be annotating the QA plots with some of that contextual information, since otherwise the DQS has to look in multiple places. And then it seems plausible to route that information through the qa file, so that it's all in one place, and if so, then the PER_EXPOSURE HDU is a reasonable location.

But this begs a different question, namely will any of the ETC information be available to NightWatch, or does it only go into telemetry database? I'll ask on slack about that.

On Tue, Jan 12, 2021 at 7:57 PM Stephen Bailey notifications@github.com wrote:

Question: since the main purpose is support stacking across exposures, should we also echo some of the keywords like exposure number or time into the table columns?

Yes, e.g. the PER_CAMERA HDU has "metadata" columns NIGHT, EXPID, SPECTRO, CAM, FIBER to orient what is what including stacking across exposures, plus QA metric columns INTEG_RAW_FLUX, MEDIAN_RAW_FLUX, etc. Within a single file, the NIGHT and EXPID are all the same but they are still kept to facilitate stacking. In this case there are 103500 = 15000 rows, one per fiber per camera (b,r,z) per spectrograph (0-9).

For a PER_EXPOSURE HDU, the "metadata" columns would be just NIGHT, EXPID; plus as many QA metric columns as we need, and each file would have just a single row.

Adding a timestamp would be useful too, but that starts taking us down a slippery slope of N>>1 other columns (airmass, sky, seeing, temp, ...) which might be better kept in a separate table and merged as needed.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/desihub/nightwatch/issues/213#issuecomment-759132541, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPQRGRO46UPLW2JLQTM3ADSZTVXZANCNFSM4V7P7G4A .

[sybenzvi]

This ticket came up today in a survey-ops discussion on slack due to the discovery of fiducials contaminating tile 80708 in exposures 75868-75870.

Nightwatch is not the first line of defense against this problem; ICS now prevents the launch of spectrograph exposures if the fiducials are on. But we should probably incorporate the check into Nightwatch in case we rebuild the exposure tables in the future.

[sbailey]

Comments from @djschlegel in separate ticket #234:

If the lights that front-illuminate the focal plane for FVC pictures (installed ~summer 2021?) are left on, they severely contaminate both the GFA images and the spectrographs. The GFAs are saturated. The spectrographs have contaminating blue light like the spectrum below from night 20211128 expid 111175.

NightWatch should trigger an error on any spectra where this is seen (or more generally, any large amount of contaminating light, although here we have the opportunity to diagnose the problem from the spectrum).

Good test cases are the evening cals for night 20211128, expid 111164 - 11197.