[MANUAL_TRIGGER] Problem with some targets missing in reduction

njcuk9999 / apero-utils

APERO affiliated utilities and tools

MIT License

3 stars 3 forks source link

[MANUAL_TRIGGER] Problem with some targets missing in reduction #225

Closed njcuk9999 closed 8 months ago

njcuk9999 commented 8 months ago

Seems some targets weren't even processed by APERO

i.e. 2024-03-05 NIRPS_2024_03_06T06_21_44_589.fits 2024-03-05 NIRPS_2024_03_06T06_41_32_142.fits

This may be because the manual_trigger was run before this data was on disk (First preprocesssing for that night ran at 2024-03-06 14:36:47.595 (Montreal time)

Can we have a flag that some raw data did not get processed? How do we know that we have all raw data for a night?

njcuk9999 commented 8 months ago

[x] Add PREV_REDUC to reduced checks that checks files in the last 7 days (all raw files have pp files?)
[x] Test

Questions:

Do all raw files have pp files?
When do we set the "last file we expect" cut off (i.e. if data is downloaded later do we then expect all files older than that to be reduced?)
Do we use the last modified time of raw files (quickest)? or do we have to read headers and get observed time (slower)

njcuk9999 commented 8 months ago

Targets on 2024-01-29 (HA) do not reprocess (no pp) are they on reject list?

njcuk9999 commented 8 months ago

[ ] Check reject list and don't report reject list files