Problems with end-of-day averaging

[jburkepile]

I found 4 days in the K-Cor mission that have no average images or are missing some average images

• [x] 20160728 — no averages
• [x] 20140227 — no averages available after 22:54:12 but there is data until 23:54:39
• [x] 20140727 — no averages available after 21:06:54 but data run until 21:13:13
• [x] 20141104 — no averages

Tasks

• [x] reprocess 20160728 in production
• [x] reprocess 20141104 in production

[jburkepile]

Just found a few more:

• [ ] 20141104 — no averages

[jburkepile]

It appears that something may have happened in the end of day processing since the averages and the engineering plots (e.g. SGS plots, pB plots ) are also missing. Please reprocess these 2 days to see if that fixes the problem:

• 20160728 — no averages
• 20141104 — no averages

[mgalloy]

For 20160728, there were 373 files in the t1 log that are not present. I checked the Campaign Storage archive if the missing files were there, but they are not. There are odd error logs from the day, but don't seem to indicate a problem with transfer:

$ cat 20160728_190004_kcor_eod_err.log
--- kcor_eod 20160728 --- 2016-07-28 19:00:04
... L0 fits files exist in /hao/mlsodata1/Data/KCor/raw/20160728.
... L1 processing incomplete.
*** kcor_eod done ***

[mgalloy]

The t1 and t2 logs seems to be missing for 20141104, but the .gz version is there, so I am unzipping and reprocessing 20141104 again.

[mgalloy]

The database entries for kcor_sci have changed due to development work for new heights for #345, so I had to add back r13 and r18 back in temporarily. I am reprocessing 20141104 again.

[jburkepile]

Thanks for catching that problem and finding a fix so you could reprocess. I appreciate that!

[samaiyahfarid8]

Thanks Mike!

[jburkepile]

Mike, you entered this comment yesterday: : 'For 20160728, there were 373 files in the t1 log that are not present.' I just looked in the processing directory on for this day and checked the t1.log file against the list of images in level0 and I don't see any missing. Perhaps I misunderstood your comment. It looks like you reprocessed the day (thank you!) I checked the MLSO web page for that day and there are now averages available. Yeah! However, some of the engineering plots such as pB at a fixed height, SGS plots, and synoptic maps, are still missing.

Samaiyah: we need to show you what to do if Mike reports images missing from the t1 log files. We can talk about that when I get back from the workshop.

[jburkepile]

I have checked the web page for 20141104 and the average images are now available. Thanks so much! There is one engineering product missing: the plot of pB vs radius. The 28 day synoptic map is there but it only contains data for Nov 4. When you have time please rerun the synoptic maps so it picks up the surrounding days data to fill in the map. I realize that CoMP is the current priority so this can be done when you have time available. Same goes for the work remaining on 20160728 to create the missing engineering products.

[mgalloy]

@jburkepile I did this check:

20160728$ wc -l 20160728.kcor.t1.log
2079 20160728.kcor.t1.log
20160728$ cd level0
level0$ ls *.fts.gz | wc -l
1706

And 2079 - 1706 = 373. In particular, I needed to find 20160729_011818_kcor.fts.gz which is in the t1 log, but not in level0.

[mgalloy]

@jburkepile For 20141104, I am still having crashes because of the change in heights for the synoptic maps, so the current production pipeline is crashing. I have reprocessed 20141104, but it is not getting through the whole day yet.

[mgalloy]

I have finished reprocessing 20141104.

[samaiyahfarid8]

Regarding 20160728: The webpage shows averages, but the end of day instrument diagnostics are missing.

[mgalloy]

OK, for 20160728, there were missing files. There is logic in the end-of-day code to not do certain actions if there are missing files (or if any file is the wrong size).

2024-01-18 23:31:52 INFO: KCOR_EOD: t1.log: # L0 files: 1706
2024-01-18 23:31:52 WARN: KCOR_EOD: t1.log: # missing files: 373
2024-01-18 23:31:52 INFO: KCOR_EOD: skipping updating database

I think the idea was to find the missing files and then reprocess, but the missing files are not in the archive on Campaign Storage. I can make an updated t1.log with only the present files and reprocess. I will save away the original t1.log.

[mgalloy]

OK, I finally got all the plots showing for 20160728.

[samaiyahfarid8]

Awesome, thanks!

[mgalloy]

Please confirm 20141104 and 20160728 are both totally OK now (particularly averages, but anything else as well).

[jburkepile]

20141104

• averages are all there. The engineering data are all there.
• The synoptic map needs to be updated to pick up data from the previous 27 days.

20160728

• averages are all there. The engineering data are all there.
• The synoptic map needs to be updated to pick up data from previous 27 days.

Thanks for all the work Mike! I believe you update the synoptic maps in a separate processing so you don't have to reprocess the whole day again. Hope that's correct.

[samaiyahfarid8]

20140227: There are no averages because there is a data gap from 22: 55:28:28- 23:54:39. There is no action needed.

[samaiyahfarid8]

20140727: There are no averages after 21:06:54 because the last images are too far (> 2 mins) apart therefore the averaging criteria failed. Images are 21:08:40, 21:13:13 UT