Open jburkepile opened 9 months ago
detoma
commented
7 months ago The problem in 2012 seem to be related to the anomalous period in 2012 when the BKG was very high with no clear reason and all data were labelled bad in the GBU. We revised the GBU thresholds down. There may be other periods in 2011 when this happens. Once we reprocess the 2011 data, we can plot the BKG values and evaluate.
See issue #96
detoma
commented
5 months ago Some level 1 image in the high background period in 2012 are strange. See example below, which made me worry something had failed in the level 1 processing. It seemed the beams were not properly combined or the flat was off.
The raw data show that the occulter was not centered with the field mask, i.e. the annular region was wider on one side and narrower on the other.
The pipeline does not find the center of the field mask well but this can only explain the outer crescent moon artifact.
The solar disk seems to be well covered by the occulter, there is no bad spillage in the raw images, and the pipeline finds the center of the occulter. It is not clear to me why the image looks so bad. It may be helpful to have intermediate products for this day 20120517 and for day 20120531 when the background decreased with no apparent reason.
detoma
commented
5 months ago One final bit of information. I looked at two images on 2012 May 31: one with high background and one with low background. They are about 30m apart. They have the same flat applied. There is no data gap in between. They are plotted using the same min and max of 0 and 30. The difference is stunning.
The only things that is slightly different is the stray light correction but this is usually a small correction. The raw data show the same stunning difference, so the stray light correction is not the issue. The voltage and temperature in the two raw headers are identical and they should not be. I remembered there was a time when these values were not updated.
mgalloy
commented
4 months ago 575 1074 nm files 64 1079 nm files 0 1083 nm files
20184 images off detector 29 of 138 flats rejected for being below threshold 1 science images failed quality check
no error messages
See log /hao/acos/comp/logs/20110103.comp.log for details
575 files with post angle difference greater than tolerance (2.0 deg)
10 bad images because background > max 14.0 ppm 52 bad images because std dev of intensity - median intensity > 2.00 ppm 410 bad images because background contains more than 2000 pixels with value > 70.0
410 bad files out of 575 total 1074 nm files
64 files with post angle difference greater than tolerance (2.0 deg)
44 bad images because background contains more than 2000 pixels with value > 70.0
48 bad files out of 64 total 1079 nm files
mgalloy
commented
4 months ago No level 1 1074 nm or 1079 nm files are produced because the pipeline can't find a valid flat for them.
0 1074 nm files 0 1079 nm files 29 1083 nm files
2320 images off detector 72 of 102 flats rejected for being below threshold
2024-02-06 04:57:34 ERROR: COMP_APPLY_FLATS_DARKS: no valid flats found ...repeated 500 more times...
See log /hao/acos/comp/logs/20110203.comp.log for details
29 files with post angle difference greater than tolerance (2.0 deg)
no bad files out of 29 total 1083 nm files
mgalloy
commented
4 months ago 479 1074 nm files 0 1079 nm files 23 1083 nm files
16028 images off detector 136 of 282 flats rejected for being below threshold 29 science images failed quality check
no error messages
See log /hao/acos/comp/logs/20110309.comp.log for details
median background 10.2 exceeds limit 8.0 479 files with post angle difference greater than tolerance (2.0 deg)
90 bad images because background > max 14.0 ppm 12 bad images because std dev of intensity - median intensity > 2.00 ppm 210 bad images because background contains more than 2000 pixels with value > 70.0
210 bad files out of 479 total 1074 nm files
23 files with post angle difference greater than tolerance (2.0 deg)
no bad files out of 23 total 1083 nm files
mgalloy
commented
4 months ago 489 1074 nm files 51 1079 nm files 25 1083 nm files
18964 images off detector 24 of 132 flats rejected for being below threshold
no error messages
See log /hao/acos/comp/logs/20110401.comp.log for details
median background 12.6 exceeds limit 8.0 489 files with post angle difference greater than tolerance (2.0 deg)
149 bad images because background > max 14.0 ppm 6 bad images because background < min 2.2 ppm 11 bad images because std dev of intensity - median intensity > 2.00 ppm 21 bad images because background changes by more than 50.0% of median background 102 bad images because background contains more than 2000 pixels with value > 70.0 5 bad images because std dev of intensity - median intensity is NaN of Inf
194 bad files out of 489 total 1074 nm files
median background 12.5 exceeds limit 8.0 51 files with post angle difference greater than tolerance (2.0 deg)
15 bad images because background > max 14.0 ppm 4 bad images because background changes by more than 50.0% of median background 9 bad images because background contains more than 2000 pixels with value > 70.0
18 bad files out of 51 total 1079 nm files
25 files with post angle difference greater than tolerance (2.0 deg)
no bad files out of 25 total 1083 nm files
mgalloy
commented
4 months ago 540 1074 nm files 72 1079 nm files 0 1083 nm files
16560 images off detector 35 of 138 flats rejected for being below threshold
no error messages
See log /hao/acos/comp/logs/20110521.comp.log for details
median background 11.0 exceeds limit 8.0
3 bad images because background > max 14.0 ppm 1 bad images because background changes by more than 50.0% of median background 1 bad images because background contains more than 2000 pixels with value > 70.0
3 bad files out of 540 total 1074 nm files
median background 11.1 exceeds limit 8.0
no bad files out of 72 total 1079 nm files
mgalloy
commented
4 months ago 535 1074 nm files 0 1079 nm files 0 1083 nm files
16640 images off detector 70 of 198 flats rejected for being below threshold 17 science images failed quality check
no error messages
See log /hao/acos/comp/logs/20110622.comp.log for details
median background 13.9 exceeds limit 8.0
262 bad images because background > max 14.0 ppm 8 bad images because std dev of intensity - median intensity > 2.00 ppm 30 bad images because background contains more than 2000 pixels with value > 70.0
263 bad files out of 535 total 1074 nm files
mgalloy
commented
4 months ago 440 1074 nm files 56 1079 nm files 27 1083 nm files
21956 images off detector
no error messages
See log /hao/acos/comp/logs/20111016.comp.log for details
median background 11.6 exceeds limit 8.0
187 bad images because background contains more than 2000 pixels with value > 70.0
187 bad files out of 440 total 1074 nm files
median background 11.3 exceeds limit 8.0
8 bad images because background contains more than 2000 pixels with value > 70.0
8 bad files out of 56 total 1079 nm files
no warnings
no bad files out of 27 total 1083 nm files
mgalloy
commented
4 months ago 646 1074 nm files 67 1079 nm files 32 1083 nm files
24824 images off detector 13 of 168 flats rejected for being below threshold
no error messages
See log /hao/acos/comp/logs/20110701.comp.log for details
median background 11.8 exceeds limit 8.0
no bad files out of 646 total 1074 nm files
median background 11.7 exceeds limit 8.0
no bad files out of 67 total 1079 nm files
no warnings
no bad files out of 32 total 1083 nm files
mgalloy
commented
4 months ago 650 1074 nm files 60 1079 nm files 30 1083 nm files
24000 images off detector
no error messages
See log /hao/acos/comp/logs/20110801.comp.log for details
median background 13.0 exceeds limit 8.0
170 bad images because background > max 14.0 ppm 582 bad images because std dev of intensity - median intensity > 2.00 ppm 1 bad images because background contains more than 2000 pixels with value > 70.0
583 bad files out of 650 total 1074 nm files
median background 12.5 exceeds limit 8.0
19 bad images because background > max 14.0 ppm 33 bad images because std dev of intensity - median intensity > 2.00 ppm
51 bad files out of 60 total 1079 nm files
no warnings
no bad files out of 30 total 1083 nm files
mgalloy
commented
4 months ago 607 1074 nm files 76 1079 nm files 38 1083 nm files
30252 images off detector 1 of 138 flats rejected for being below threshold 2 science images failed quality check
no error messages
See log /hao/acos/comp/logs/20110904.comp.log for details
median background 15.1 exceeds limit 8.0
475 bad images because background > max 14.0 ppm 5 bad images because background < min 2.2 ppm 31 bad images because std dev of intensity - median intensity > 2.00 ppm 11 bad images because background changes by more than 50.0% of median background 113 bad images because background contains more than 2000 pixels with value > 70.0 5 bad images because std dev of intensity - median intensity is NaN of Inf
480 bad files out of 607 total 1074 nm files
median background 14.6 exceeds limit 8.0
56 bad images because background > max 14.0 ppm 3 bad images because std dev of intensity - median intensity > 2.00 ppm 14 bad images because background contains more than 2000 pixels with value > 70.0
56 bad files out of 76 total 1079 nm files
no warnings
no bad files out of 38 total 1083 nm files
mgalloy
commented
4 months ago 363 1074 nm files 70 1079 nm files 36 1083 nm files
20820 images off detector 6 of 138 flats rejected for being below threshold 32 science images failed quality check
no error messages
See log /hao/acos/comp/logs/20111107.comp.log for details
median background 10.9 exceeds limit 8.0 25 files with post angle difference greater than tolerance (2.0 deg)
92 bad images because background > max 14.0 ppm 4 bad images because background < min 2.2 ppm 27 bad images because std dev of intensity - median intensity > 2.00 ppm 63 bad images because background changes by more than 50.0% of median background 88 bad images because background contains more than 2000 pixels with value > 70.0 4 bad images because std dev of intensity - median intensity is NaN of Inf
120 bad files out of 363 total 1074 nm files
median background 10.5 exceeds limit 8.0
9 bad images because background > max 14.0 ppm 4 bad images because background changes by more than 50.0% of median background 10 bad images because background contains more than 2000 pixels with value > 70.0
15 bad files out of 70 total 1079 nm files
no warnings
no bad files out of 36 total 1083 nm files
mgalloy
commented
4 months ago 388 1074 nm files 28 1079 nm files 19 1083 nm files
17476 images off detector 4 of 92 flats rejected for being below threshold 85 science images failed quality check
no error messages
See log /hao/acos/comp/logs/20111201.comp.log for details
median background 9.5 exceeds limit 8.0
12 bad images because background > max 14.0 ppm 2 bad images because background changes by more than 50.0% of median background 185 bad images because background contains more than 2000 pixels with value > 70.0
186 bad files out of 388 total 1074 nm files
median background 9.4 exceeds limit 8.0
14 bad images because background contains more than 2000 pixels with value > 70.0
14 bad files out of 28 total 1079 nm files
no warnings
no bad files out of 19 total 1083 nm files
mgalloy
commented
4 months ago There was a bug in the rest wavelength calculation for #136 (I created a mask from velocity thresholds before converting velocity to km/s) that requires recomputing all the dynamics files (quick inverts were OK). That is currently underway.
mgalloy
commented
4 months ago Dynamics files have been recomputed.
The community was provided with quick invert files in 2011 and 2012. There are publications reporting results from these old quick invert files (e.g. Ula's 2013 paper on lagomorph/flux rope in July 27, 2011 CoMP). 2012 was reprocessed by Mike and in some cases the new pipeline did not produce level 2 data (it likely failed GBU). For some of those days, old quick inverts do exist on the web and have been used by the community. How do we deal with these old quick inverts?
Tasks
reprocess-check-2011specifies the dates