Check keywords and make sure we are promoting the correct keywords between levels

bberkeyU commented 5 months ago

Are we doing the right thing by promoting keywords across levels and into averaging?

Questions are based on recent processing of the process.analytic-gaussian data. Tracking is looking at 20220202.213127 and the waves collections.

Copies of the header can (temporarily be found in the wiki)

Questions

[ ] Should 20220202.ucomp.1074.l1.waves.mean.fts contain the following keywords file specific keywords (note these are seen in all image extensions): RAWEXTS, RAWDARK1, DARKEXT1, RAWDARK2, DARKEXT2, FLTFILE1, FLTEXTS1, MFLTEXT1, FLTFILE2, FLTEXTS2, MFLTEXT2?
[ ] Should the 20220202.ucomp.1074.l1.waves.mean.fts and 20220202.ucomp.1074.l2.waves.mean.fts contain FLATDN and SKYTRANS keywords? They currently have different values. Should we expect them to be the same? How should this value be calculated?
[ ] 20220202.ucomp.1074.l2.waves.mean.fts contains multiple empty keywords in most extensions. Should we keep these? CAMCORR, CAMDIFF, RCAMMED, TCAMMED
[ ] For the L1 and L2 files, will we ever have a DATATYPE or OBJECT that isn't science or sun? The OBJECT keyword appears in all headers, while the DATATYPE appears in all extensions. If we define the L1 files as flat and dark-corrected coronal science data products, these keywords may not need to be in the primary headers.
[ ] For the L1/l2 mean how do we calculate DSUN_OBS, SECANT_Z, SID_TIME, and JUL_DATE? Should we add a comment to the header to identify how we computed these values?
[ ] For the L1/L2 Mean, how do we handle the quality metrics? VCROSSTK, MED_BKG, and number saturated or non-linear pixels for the L1/L2 mean files?
[ ] For the L1/L2 mean files, how do/should we handle NFRAM, NREPEAT and REMFRAME?
[ ] How do we handle the hardware position keywords: O1FOCUS, OCCLTR-X, OCCLTR-Y? O1FOCUS probably shouldn't change, but the occulter position will change when the observer realigns.
[ ] How do we handle CONTIN? Steve suggested we consider a program that does red, blue, and both to better measure the continuum. Will we allow the continuum to change during a program and combine REDS with BLUES in the averaging?
[ ] How do we handle the temperatures in the L1/L2 mean? We should change or edit the Temperature comments to reflect this collection action.
[ ] In the L1/L2 mean SGS and Weather sections, comments should be edited to add a notice that the values are averages across the collected files.
[ ] For the L1/L2 Mean, How are the occulter centering info keywords handled? XOFFSET0, YOFFSET0, RADUS0, FITCH0, XOFFSET1, YOFFSET1, RADUS1, FITCH1, POST_ANG, RADIUS, TCAMCC, RCAMCC
[ ] Should the LEVEL keywords in the mean files be changed from L1->L1 Mean and L2->L2 Mean

Todo

[ ] Evaluate what changes we need to make to the UCoMP manual as a consequence of these changes. We currently display an L1 and L2 header. Do we need to include example headers for the L1 and L2 Mean files (assuming we make changes for those files). Should we create a new section in the header files to discuss our averaging? Including criteria and how they affect the meta-data. In the current version, we have no details about the process, just that the result is some sort of average.

Level 1 files

[ ] add PRODUCT keyword after LEVEL with value “image”
[x] change BUNIT value to "millionths of solar disk center intensity" (no comment)
[x] set BSCALE to 1.0
[ ] move start of "Observing Info" section to after RCAMNUC (for L1 files)
[ ] drop FLC*, POLHOFF, REDHOFF in L1
[ ] add "mm" in OCCLTR-{X,Y} comments in L1
[ ] add "mm" in O1FOCUS comment in L1
[x] IMAGESCL – report to 4 digits in L1

Level 1 averages

[ ] move NUMFILES to after MJD-END
[ ] move NUMWAVE to after NUMFILES
[ ] update PRODUCT value to "image mean"
[ ] update DATE_DP value to current time
[ ] add comment saying values are reported for first image in averaged files for WCS and Ephemeris sections
[ ] change quality metrics section comment to "Mean standard deviations and Max values were calculated across allthe header values for the files included in the mean collection."
[ ] compute mean and standard deviation for VCROSSTK and MED_BKG, put mean in keyword (comments "Stokes V crosstalk metric mean" and "[ppm] median of background annulus mean") and standard deviation in VCRSSTKS and MED_BKGS (comments "Stokes V crosstalk metric std" and "[ppm] median of background annulus std")
[ ] drop NUM* keywords in quality section
[ ] set NFRAME comment to "total number of frames in level 0 file"
[ ] drop NFRAME, REMFRAME, NUMSUM, NREPEAT from averages
[ ] drop comment "NFRAME = …"
[ ] NUMWAVE moves to under NUMFILES in means
[ ] add OCCMOVED keyword in mean, T or F
[ ] add comment that temperatures are means in Temperature section:

Temperatures used in the Lyot filter calibrations are low-pass filtered and reported in keywords that start with T. The raw, unfiltered temperature values for recorded temperatures are recorded in keywords that begin with TU. Temperature values are calculated as the mean of the header values across all files include in the mean collection.
[ ] mean for WNDSPD
[ ] drop WNDDIR
[ ] drop COMMENT —--Weather info ——
[ ] add comment in SGS section that values are averages
[ ] Compute means and standard deviation of each (keep name for means, add S to end of keyword for standard deviation, dropping vowels as needed)
[ ] max for FITCHI{0,1} and {R,T}CAMECC

Level 1 average extensions

[ ] promote DATETYPE and OBJECT to primary (what is comment for OBJECT?)
[ ] drop RAWFILE…MFLATEXT2
[ ] FLATDN - compute mean of medians, change comment (verify correctness)
[ ] FLATDNS - standard deviation of FLATDN
[ ] drop CAMCORR…TCAMMED (why empty in L1?)
[ ] promote SKYTRANS to primary header.

Level 2 files

Related Issues:

226
222
147

bberkeyU commented 5 months ago

The pipeline computes the mean value for the mean files for the camera and cropico temps. This seems the correct way to handle the value.

I see two options to document how we manage the temperatures. Either change the meta-comment at the start of the temperature section or add Mean to each individual comment.

For the Meta comment change,we could add a couple lines to the end of the COMMENT --- Temperatures --- section like the following.

COMMENT --- Temperatures ---                                                    \
COMMENT Temperatures used in the Lyot filter calibrations are low-pass          \
COMMENT filtered and reported in keywords that start with T_. The raw,          \
COMMENT unfiltered temperature values for recorded temperatures are recorded    \
COMMENT in keywords that begin with TU_.   
COMMENT The values in this section are the mean temperature values values from all 
COMMENT  L1 files included in this Mean file.

If we change the individual comments we could do the following. current T_LCVR1 = 34.387 / [C] Lyot LCVR1 Temp

new T_LCVR1 = 34.387 / [C] Mean Lyot LCVR1 Temp

bberkeyU commented 5 months ago

No statistics are done to DSUN_OBS, SECANT_Z, SID_TIME, and JUL_DATE they are promoted from the first file.

The keywords in the --- Ephemeris info --- section use up most of the 31 characters available to them, so we would not be able to add information to the individual keyword without reworking the keyword language. Therefore, if we add language about how we compute header values for the mean file, it must be in a meta-comment for the whole section.

bberkeyU commented 5 months ago

The --- Quality metrics --- keywords are promoted from the first file.

It seems like we should either drop this section or change it to include the mean and standard deviation to give a sense of the statistics behind these values.

bberkeyU commented 5 months ago

Tracking the number of images used in an L1/L2 mean file. The averaging code computes NUMWAVE and NUMFILES by looking at all the wavelengths and files included in the average. However, the other image tracking numbers are prompted from the first file in the mean. Further, the pipeline doesn't check that all files in a mean have the same NUMSUM and NREPEAT but assumes that by policy, we will not try to average dissimilar files.

If we keep REMFRAME we should change it to the sum of all the REMFRAME values from the input files.

NREPEAT seems confusing in this context. We should either drop it. Or if we keep it, we should edit the comment to be something more like NREPEAT = 1 / number of repeat wavelength scans per L1 file

NFRAME comment should be edited. Perhaps NFRAME = 48 / total number of image frames in an L1 file

New comment for NUMFILES is addressed in #208

Sample from the header.

NFRAME  =                   48 / total number of image frames in file           \
REMFRAME=                    0 / number of bad frames removed                   \
NUMWAVE =                    3 / number of wavelengths                          \
NUMSUM  =                   14 / number of camera reads summed in an image frame\
NREPEAT =                    1 / number of repeats of wavelength scans          \
NUMBEAM =                    2 / number of beams                                \
NUMFILES=                   70 / number of level 1 files averaged               \
COMMENT Total camera reads in this file = NFRAME * NUMSUM where                 \
COMMENT   NFRAME = NUMWAVE * NREPEAT * NUMBEAM * 2(Cameras) * 4(Polarizations)  \

bberkeyU commented 5 months ago

We should expand the --- Occulter centering info --- section to record the mean and standard deviation of each of these values.

It may be hard to find reasonable keyword names for the new keywords.

bberkeyU commented 5 months ago

We should expand the --- Weather info --- section to include the mean and standard deviation for each of these values.

bberkeyU commented 5 months ago

We should drop the RAWEXTS, RAWDARK1, DARKEXT1, RAWDARK2, DARKEXT2, FLTFILE1, FLTEXTS1, MFLTEXT1, FLTFILE2, FLTEXTS2, MFLTEXT2 values from the L1 Mean files.

It will be complicated to keep track of these across the multiple L1 input files.

bberkeyU commented 5 months ago

FLATDN, CAMCORR, CAMDIFF, RCAMMED, TCAMMED, SKYTRAN Should not be prompted from the L1 file into L2 files.

If we want to keep them in the L2 and Mean files, we need to figure out how to average them across all the extensions.

RAWFILE should not be prompted to the Mean files.

mgalloy commented 5 months ago

My notes for changes to make are here.

mgalloy commented 4 months ago

From @bberkeyU, possible keywords and header comments:

COMMENT --- Occulter centering info ---                                         \
COMMENT X/YOFFSET define position w.r.t. distortion corrected L0 images         \
COMMENT Mean standard deviations and Max values were calculated across all      \
COMMENT the header values for the files included in the mean collection.        \
XOFFSET0=                1.154 / [pixels] RCAM occulter x-offset mean           \
XOFF0S=                  1.154 / [pixels] RCAM occulter x-offset std            \
YOFFSET0=               -5.666 / [pixels] RCAM occulter y-offset mean           \
YOFF0S=                  1.154 / [pixels] RCAM occulter y-offset std            \
RADIUS0 =              343.051 / [pixels] RCAM occulter radius mean             \
RADIUS0S=              343.051 / [pixels] RCAM occulter radius std              \
FITCHI0M=            0.022681 / [pixels] MAX chi-squared for RCAM center fit    \
XOFFSET1=                2.611 / [pixels] TCAM occulter x-offset mean           \
XOFF1S=                  1.154 / [pixels] TCAM occulter x-offset std            \
YOFFSET1=               -6.121 / [pixels] TCAM occulter y-offset mean           \
YOFF1S=                  1.154 / [pixels] TCAM occulter y-offset std            \
RADIUS1 =              343.418 / [pixels] TCAM occulter radius mean             \
RADIUS1S=              343.418 / [pixels] TCAM occulter radius std              \
FITCHI1M=             0.021198 / [pixels] MAX chi-squared for TCAM center fit   \
POST_ANG=              193.421 / [deg] post angle CCW from north mean           \
PST_ANGS=              193.421 / [deg] post angle CCW from north std            \
RADIUS  =              343.234 / [pixels] occulter average radius mean          \
RADIUSS  =             343.234 / [pixels] occulter average radius std           \
IMAGESCL=               2.9219 / [arcsec/pixels] image scale mean               \
IMGSCLS=                2.9219 / [arcsec/pixels] image scale std                \
RCAMECCM =              0.0178 / MAX occulter eccentricity in RCAM              \
TCAMECCM =              0.0313 / MAX occulter eccentricity in TCAM              \

bberkeyU commented 4 months ago

@mgalloy In the level1 flat/dark files (ie level1/YYYYMMDD.ucomp.dark.fts) should have the TCAMNUC /RCAMNUC keywords pushed into the extension headers. At least once during the mission we changed the NUC in the middle of an observing day.

NCAR / ucomp-pipeline