mgalloy
commented
4 months ago @bberkeyU Are the times listed in Julian date? Why is 12:00:00 so frequent? i.e. is the "2022-11-18 12:00:00.000" listed actually "2022-11-18T12:00:00Z"?
Open bberkeyU opened 4 months ago
mgalloy
commented
4 months ago @bberkeyU Are the times listed in Julian date? Why is 12:00:00 so frequent? i.e. is the "2022-11-18 12:00:00.000" listed actually "2022-11-18T12:00:00Z"?
mgalloy
commented
4 months ago I think there is a missing epoch for 706. I think that the next epoch after:
2021-10-08 00:00:00.000 2021-11-11 12:00:00.000is:
2022-02-23 19:12:00.000 2022-03-19 12:00:00.000So 2021-11-11 12:00:00.000 to 2022-02-23 19:12:00.000 is missing.
mgalloy
commented
4 months ago @bberkeyU Are there example files that you know will fail the quality check? I would like some good days to test on.
bberkeyU
commented
4 months ago @mgalloy I think they are in Julian Dates:
To get the epochs, I pulled the DATE-BEG from all of the header files and converted it to Julian data with the astropy.time import Time with the following command: Time(ext.header['DATE-BEG']).jd,fit[0]
I then plotted the flat medians vs this Julien data and tweaked my X and Y limits until I found good start/stop times and intensity limits. (Note: when doing the start/stop finding, I subtracted 2459000 from all of the Julien dates so I could think about smaller numbers.
I converted the values back from Julien date to date time strings with the following command flatRanges["date-beg"] = Time(flatRanges["jdstart"]+2459000,format="jd").iso
This process gave a 12:00 reference instead of a 00:00 reference time. In most but not all cases, the epochs change at the beginning or end of an observing day, so the exact timestamp isn't going to be important.
bberkeyU
commented
4 months ago @mgalloy I am looking into finding bad days that are both clearly and marginally bad.
bberkeyU
commented
3 months ago I created a workbook with Flat epoch plots. It is a fairly long document with the interesting part most of the way down.
In this example plot from the 637 section
we can see an epoch for 2022-03-19 to 2022-06-16 There are four plots in this images separated by camera and onband. Orange dots represent the median flat values in the "coronal" region. Most points fall within the yellow bars representing high and low limit values, but a handful is clearly below the low-intensity limit.
There is a similar plot for each of the six wave regions (530, 637, 706, 789, 1074,1079) for each epoch region I identified. The upper and lower limits were empirically selected; and others might pick different values.
Currently, we do not perform image data quality checks on the flats, which has allowed some marginal or deficient flats to be used in the image correction.
Between spot-checking good and bad flats and looking at metrics across the UCoMP mission, I find that for a given epoch, good flats have a range of meidan values vary by wave region, epoch, camera, and onband tuning.
I have computed a table of high and low flat values for each of the 637,706,1074, and 1079 regions. I also looked at 530, but all flats were equally good (or bad?), so the pipeline should pass all the 530 data. If the pipeline needs values for 530 and the other wavelengths, we should start with min=0 and max =4095 and allow everything to pass.
For the analysis, the median value of the raw L0 flat pixels within an annulus centered in the middle of the detector with an inner radius of 375 and an outer radius of 600 pixels was computed; the computation also excluded post values between x=550 ->700 in the raw detector coordinate systems. These median values were then divided by NUMSUM, and the mean was computed across the 4 modulation states.
The table below contains the high and low flat limits for the following states:
Date-beg and date-end define the start and stop of the epochs.
@mgalloy Please let me know if this table should be in a different format; I can easily transform and recreate it.
Possible epoch value definitions, which are all 2-element arrays of min/max values:
Tasks