search

NCAR / kcor-pipeline

Pipeline code for KCor
Other
3 stars 2 forks source link

Consider not processing the first image after mechanism moves finishes #362

Open bberkeyU opened 9 months ago

bberkeyU commented 9 months ago

We see some bright light patterns assoicated with the cover moving out of the beam in 20220902_222740_kcor.fts.gz which reports cover out. The pattern is considerably fainter than the image before in which the cover is mid.

Most of the image is ok, but the SE is funky; so the team should decide if it is worth excluding the cover (and probally diffuser/polarizer) edge case images.

Tasks

mgalloy commented 3 months ago

How to determine if a science file is the first after a cal file? The cal and science files are handled in separate loops. Maybe separate runs of the pipeline when running in realtime. A realtime run of the pipeline needs to know all the cal and science files before it to track this.

bberkeyU commented 3 months ago

Good question. When the real-time code gets a batch of files, is there any guarantee that it processes the oldest one first, and is it single-threaded? If they go in order and we only have one processing thread, we could add a "the last file was a science file" flag. Only fully process the current file if the flag is TRUE.

But it may not be worth implementing this on the realtime side.

jburkepile commented 1 month ago

Mike: Can you remove these in the 'eod' processing?

In the realtime pipeline you are logging the information you need in the realtime.log file, to identify images taken after a calibration or anytime a device is in the beam.

Here is the end of a calibration sequence from the 20170203.realtime log:

There is an image with "dev" (device in the beam), then two saturated images, and then an "OK" image that is not actually OK:

2022-01-13 08:01:31 INFO: KCOR_QUALITY: 646/1858: 20170203_205732_kcor.fts.gz [eng] (dev)
2022-01-13 08:01:32 INFO: KCOR_QUALITY: 647/1858: 20170203_205747_kcor.fts.gz [eng] (sat)
2022-01-13 08:01:32 INFO: KCOR_QUALITY: 648/1858: 20170203_205803_kcor.fts.gz [eng] (sat)
2022-01-13 08:01:33 INFO: KCOR_QUALITY: 649/1858: 20170203_205818_kcor.fts.gz [eng] (ok)

Please remove the first 'ok' image after a 'device' sequence. The image will be marked either engineering or science. Then keep all the images after this one.

Please let me know if you have questions or if this isn't clear.

mgalloy commented 1 month ago

The above could be done early in the KCOR_EOD routine. Need to remove image from:

mgalloy commented 2 weeks ago

For the test date 201702023, before the removing of OK files:

q$ wc -l *.ls
    2 brt.ls
   31 cal.ls
    0 cld.ls
   51 dev.ls
    3 dim.ls
    7 nsy.ls
 1760 oka.ls
    4 sat.ls
 1858 total
mgalloy commented 2 weeks ago

For the test date, 20170203, from looking at the results from KCOR_QUALITY, the following files should be removed:

mgalloy commented 2 weeks ago

Here is the log messages that kcor_remove_moving_files wrote:

2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removing 20170203_181507_kcor.fts.gz...
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removed 1 files from oka.ls
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: added file to dev.ls
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removing 20170203_204051_kcor.fts.gz...
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removed 6 files from level1 dir
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removed 3 files from level2 dir
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removed 1 files from web archive
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removed 1 files from cropped archive
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removed 1 files from oka.ls
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: added file to dev.ls
2024-08-30 17:56:24 WARN: KCOR_REMOVE_OKFILE: removing 20170203_205818_kcor.fts.gz...
2024-08-30 17:56:25 WARN: KCOR_REMOVE_OKFILE: removed 6 files from level1 dir
2024-08-30 17:56:25 WARN: KCOR_REMOVE_OKFILE: removed 3 files from level2 dir
2024-08-30 17:56:25 WARN: KCOR_REMOVE_OKFILE: removed 1 files from web archive
2024-08-30 17:56:25 WARN: KCOR_REMOVE_OKFILE: removed 1 files from cropped archive
2024-08-30 17:56:25 WARN: KCOR_REMOVE_OKFILE: removed 1 files from oka.ls
2024-08-30 17:56:25 WARN: KCOR_REMOVE_OKFILE: added file to dev.ls
2024-08-30 17:56:25 WARN: KCOR_REMOVE_MOVING_FILES: removed 3 level 0 files
mgalloy commented 2 weeks ago

The logs are consistent with that:

q$ wc -l *.ls
    2 brt.ls
   31 cal.ls
    0 cld.ls
   54 dev.ls
    3 dim.ls
    7 nsy.ls
 1757 oka.ls
    4 sat.ls
 1858 total

and

q$ grep 20170203_181507_kcor.fts.gz *.ls
dev.ls:20170203_181507_kcor.fts.gz
q$ grep 20170203_204051_kcor.fts.gz *.ls
dev.ls:20170203_204051_kcor.fts.gz
q$ grep 20170203_205818_kcor.fts.gz *.ls
dev.ls:20170203_205818_kcor.fts.gz
mgalloy commented 2 weeks ago

The level1 and level2 directories also don't have the files anymore:

20170203$ ll level{1,2}/20170203_{181507,204051,205818}*
ls: cannot access 'level1/20170203_181507*': No such file or directory
ls: cannot access 'level1/20170203_204051*': No such file or directory
ls: cannot access 'level1/20170203_205818*': No such file or directory
ls: cannot access 'level2/20170203_181507*': No such file or directory
ls: cannot access 'level2/20170203_204051*': No such file or directory
ls: cannot access 'level2/20170203_205818*': No such file or directory
mgalloy commented 1 week ago

The result in raw.latest seems to be working.

mgalloy commented 1 week ago

For the 2.1.22 reprocessing of 2013, there were no errors from removing files. The following files were removed, excluding the first good science of the day:

20130930_191621_kcor_l1.fts.gz
20130930_200509_kcor_l1.fts.gz
20131001_193629_kcor_l1.fts.gz
20131004_190307_kcor_l1.fts.gz
20131004_203136_kcor_l1.fts.gz
20131004_203152_kcor_l1.fts.gz
20131004_223042_kcor_l1.fts.gz
20131004_223057_kcor_l1.fts.gz
20131004_233959_kcor_l1.fts.gz
20131004_234014_kcor_l1.fts.gz
20131005_002126_kcor_l1.fts.gz
20131005_002141_kcor_l1.fts.gz
20131005_013903_kcor_l1.fts.gz
20131005_013919_kcor_l1.fts.gz
20131005_180008_kcor_l1.fts.gz
20131005_213045_kcor_l1.fts.gz
20131005_213100_kcor_l1.fts.gz
20131009_184837_kcor_l1.fts.gz
20131009_192044_kcor_l1.fts.gz
20131009_194530_kcor_l1.fts.gz
20131009_200629_kcor_l1.fts.gz
20131009_211515_kcor_l1.fts.gz
20131016_183543_kcor_l1.fts.gz
20131018_195134_kcor_l1.fts.gz
20131019_174213_kcor_l1.fts.gz
20131019_191647_kcor_l1.fts.gz
20131019_191702_kcor_l1.fts.gz
20131019_193011_kcor_l1.fts.gz
20131019_193026_kcor_l1.fts.gz
20131023_190825_kcor_l1.fts.gz
20131023_205135_kcor_l1.fts.gz
20131023_212437_kcor_l1.fts.gz
20131023_230027_kcor_l1.fts.gz
20131024_172825_kcor_l1.fts.gz
20131024_185307_kcor_l1.fts.gz
20131024_185322_kcor_l1.fts.gz
20131024_220446_kcor_l1.fts.gz
20131024_220502_kcor_l1.fts.gz
20131025_182629_kcor_l1.fts.gz
20131025_194507_kcor_l1.fts.gz
20131025_203631_kcor_l1.fts.gz
20131028_192407_kcor_l1.fts.gz
20131028_192825_kcor_l1.fts.gz
20131028_201456_kcor_l1.fts.gz
20131029_172653_kcor_l1.fts.gz
20131029_185840_kcor_l1.fts.gz
20131029_185855_kcor_l1.fts.gz
20131031_010730_kcor_l1.fts.gz
20131031_213717_kcor_l1.fts.gz
20131031_214049_kcor_l1.fts.gz
20131031_214105_kcor_l1.fts.gz
20131031_215615_kcor_l1.fts.gz
20131031_215630_kcor_l1.fts.gz
20131101_173018_kcor_l1.fts.gz
20131101_180758_kcor_l1.fts.gz
20131101_193240_kcor_l1.fts.gz
20131101_214812_kcor_l1.fts.gz
20131102_000359_kcor_l1.fts.gz
20131102_175156_kcor_l1.fts.gz
20131102_221841_kcor_l1.fts.gz
20131102_221856_kcor_l1.fts.gz
20131105_202503_kcor_l1.fts.gz
20131111_184631_kcor_l1.fts.gz
20131111_193549_kcor_l1.fts.gz
20131113_193716_kcor_l1.fts.gz
20131116_194211_kcor_l1.fts.gz
20131116_202309_kcor_l1.fts.gz
20131116_211630_kcor_l1.fts.gz
20131117_193459_kcor_l1.fts.gz
20131117_195125_kcor_l1.fts.gz
20131117_203238_kcor_l1.fts.gz
20131117_210948_kcor_l1.fts.gz
20131117_224034_kcor_l1.fts.gz
20131122_224508_kcor_l1.fts.gz
20131122_230119_kcor_l1.fts.gz
20131123_194605_kcor_l1.fts.gz
20131124_012933_kcor_l1.fts.gz
20131124_021650_kcor_l1.fts.gz
20131124_192105_kcor_l1.fts.gz
20131125_190418_kcor_l1.fts.gz
20131126_210804_kcor_l1.fts.gz
20131126_223518_kcor_l1.fts.gz
20131128_014752_kcor_l1.fts.gz
20131128_201616_kcor_l1.fts.gz
20131128_211914_kcor_l1.fts.gz
20131204_005009_kcor_l1.fts.gz
20131204_192415_kcor_l1.fts.gz
20131204_203100_kcor_l1.fts.gz
20131205_191744_kcor_l1.fts.gz
20131209_192126_kcor_l1.fts.gz
20131209_195907_kcor_l1.fts.gz
20131209_202641_kcor_l1.fts.gz
20131210_200757_kcor_l1.fts.gz
20131210_211442_kcor_l1.fts.gz
20131211_193554_kcor_l1.fts.gz
20131211_213615_kcor_l1.fts.gz
20131211_233953_kcor_l1.fts.gz
20131212_203903_kcor_l1.fts.gz
20131215_194604_kcor_l1.fts.gz
20131216_191801_kcor_l1.fts.gz
20131216_214919_kcor_l1.fts.gz
20131216_221723_kcor_l1.fts.gz
20131216_223520_kcor_l1.fts.gz
20131216_225836_kcor_l1.fts.gz
20131216_230254_kcor_l1.fts.gz
20131216_233229_kcor_l1.fts.gz
20131216_235258_kcor_l1.fts.gz
20131217_222120_kcor_l1.fts.gz
20131218_195509_kcor_l1.fts.gz
20131219_200332_kcor_l1.fts.gz
20131221_210428_kcor_l1.fts.gz
20131222_201626_kcor_l1.fts.gz
20131224_203030_kcor_l1.fts.gz
20131225_000410_kcor_l1.fts.gz
20131225_225124_kcor_l1.fts.gz
20131227_202641_kcor_l1.fts.gz
20131227_214837_kcor_l1.fts.gz
mgalloy commented 6 days ago

For 2013104, there are files in the removed directory that are not listed in the eod log. These are the files in the removed directory:

There was also a file in the dev.ls file that doesn't seem to be a device file:

mgalloy commented 5 days ago

OK, I am not putting removed files in the dev.ls file and we are getting the correct results:

2013$ grep "KCOR_REMOVE_OKFILE: removing" 20131004.eod.log
2024-09-12 17:19:14 WARN: KCOR_REMOVE_OKFILE: removing 20131004_190251_kcor.fts.gz...
2024-09-12 17:19:14 WARN: KCOR_REMOVE_OKFILE: removing 20131004_203136_kcor.fts.gz...
2024-09-12 17:19:14 WARN: KCOR_REMOVE_OKFILE: removing 20131004_223042_kcor.fts.gz...
2024-09-12 17:19:14 WARN: KCOR_REMOVE_OKFILE: removing 20131004_233959_kcor.fts.gz...
2024-09-12 17:19:14 WARN: KCOR_REMOVE_OKFILE: removing 20131005_002126_kcor.fts.gz...
2024-09-12 17:19:14 WARN: KCOR_REMOVE_OKFILE: removing 20131005_013903_kcor.fts.gz...
2013$ cat ../../raw.latest/20131004/q/removed.ls
20131004_190251_kcor.fts.gz
20131004_203136_kcor.fts.gz
20131004_223042_kcor.fts.gz
20131004_233959_kcor.fts.gz
20131005_002126_kcor.fts.gz
20131005_013903_kcor.fts.gz
mgalloy commented 5 days ago

The file is still in dev.ls, though:

q$ grep 20131004_223027 *.ls
dev.ls:20131004_223027_kcor.fts.gz
mgalloy commented 5 days ago

20131004_223027_kcor.fts.gz was in an epoch that was marked process: no. These files are currently placed in the dev.ls file.