Create level 1 IQU intermediate product

mgalloy commented 3 months ago

Create a level 1 intermediate product that contains IQU.

Write cal_data_combined_center or cal_data_combined in a FITS file before the polarimetric transforms.

For the header,

OBJECT should be "Corona Stokes IQU" with comment "calibrated corona+sky in cartesian coordinates"
PRODUCT should be "Stokes IQU" with comment "calibrated corona+sky in cartesian coordinates"

The filename should YYYYMMDD_HHMMSS_kcor_l1_stokesIQU.fts.

Tasks

[x] add intermediate product that contains IQU (not qmk4 and umk4)
[x] produce results for 20240409
[x] move the writing of this product to near the final level 1 product, so that the header is complete, make sure to update OBJECT and PRODUCT

mgalloy commented 3 months ago

Test results are in latest for 20240409.

mgalloy commented 2 months ago

Move the writing of this product to near the final level 1 product, so that the header is complete.

mgalloy commented 1 month ago

Test run for 20191216 on raw.intermediate with the "stokesIQU" products. Here is an example header:

SIMPLE  =                    T / Written by IDL:  Tue Aug  6 22:58:30 2024
BITPIX  =                  -64 / IEEE double precision floating point
NAXIS   =                    3 / FITS Data Cube
NAXIS1  =                 1024 /Number of positions along axis 1
NAXIS2  =                 1024 /Number of positions along axis 2
NAXIS3  =                    3 /Number of positions along axis 3
OBSERVER=               'Berkey' / NAME OF OBSERVER
LOCATION=            'Mauna Loa' / Mauna Loa Solar Observatory, Hawaii
ORIGIN  =             'NCAR/HAO' /
BZERO   =                    0 / / Offset for unsigned integer data
BSCALE  =                    1.0 /
TELESCOP=            'MLSO Spar' / Mauna Loa Solar Observatory, Hawaii
OBJECT  = 'Corona Stokes IQU'  / calibrated corona+sky in cartesian coordinates
DATATYPE=          'engineering' / 'CORONAL'
DIFFUSER=                  'out' / Diffuser in or out of the beam
DIFFSRID=             'mk4-opal' / Unique ID of the current diffuser
CALPOL  =                  'out' / Calibration Polarizer in or out of the beam
CALPANG =                  -0.03 / [deg] Calibration polarizer angle
CALPOLID=          'AP38-006T-1' / Unique ID of the current calibration polarize
DARKSHUT=                  'out' / Dark shutter open or closed
EXPTIME =               2.500000 / [msec] Exposure time for each frame
DATE-OBS=  '2019-12-17T02:09:12' / UTC observation start
DATE-END=  '2019-12-17T02:09:27' / UTC observation end
SOCKETCA=        'socketcam.exe' / Socketcam Filename
TCAMID  =                '40256' / Unique ID of camera 1
RCAMID  =                '40257' / Unique ID of camera 0
TCAMLUT =       '40256_20190307' / Unique ID of the LUT for camera 1
RCAMLUT =       '40257_20190307' / Unique ID of the LUT for camera 0
MODLTRT =                   35.1 / [deg C] Modulator temperature
NUMSUM  =                    512 / Number of frames summed in each beam and stat
TCAMFOCS= 'NaN     '           / [mm] Camera 1 focus position
RCAMFOCS= 'NaN     '           / [mm] Camera 0 focus position
O1FOCS  =                133.441 / [mm] O1 focus position
O1ID    =                 'O1-2' / O1 ID number
LYOTSTOP=                 '35mm' / Specifies if the tempoary paper Lyot stop is
COMMENT       The COSMO K-coronagraph is a 20-cm aperture, internally occulted
COMMENT       coronagraph that observes the polarization brightness of the low
SGSDIMV =               8.346000 / [V] Mean SGS DIM Signal
SGSDIMS =               0.010100 / [V] SGS DIM signal standard deviation
SGSSUMV =               8.348000 / [V] Mean SGS sum signal
SGSRAV  =              -0.000060 / [V] Mean SGS RA error signal
SGSRAS  =               0.035170 / [V] SGS RA error signal standard deviation
SGSDECV =              -0.000083 / [V] Mean SGS Dec error signal
SGSDECS =               0.014280 / [V] SGS Dec error signal standard deviation
SGSSCINT=               2.323000 / [arcsec] SGS scintillation seeing estimate
SGSLOOP =                      1 / SGS loop closed fraction
SGSSUMS =               0.008711 / [V] SGS Sum signal standard deviation
SGSRAZR =              15.000000 / [Arcsec] SGS RA  zeropoint offset
SGSDECZR=             -55.000000 / [Arcsec] SGS DEC  zeropoint offset
INSTRUME=  'COSMO K-Coronagraph' /
LEVEL   =                   'L0' / L0
COVER   =                  'out' / Cover in or out of the beam
OCCLTRID=           'OC-1018.9"' / Unique ID of the current occulter
FILTERID=         'ANDV13425-01' / Unique ID of the current bandpass filter
MODLTRID=                    'A' / Unique ID of the current modulator
OBSSWID =      '20191125T093731' / Socktcam observing software date modified
COMMENT       corona in white light.  For detailed information, see the
COMMENT       COSMO K-coronagraph instrument paper (reference) and the
COMMENT       HAO/COSMO website at:
COMMENT       http://www.cosmo.ucar.edu/kcoronagraph.html
COMMENT       The Spar Guider System (SGS) provides average and standard
COMMENT       deviation of the sum, RA pointing error and Dec pointing error
COMMENT       signals, an estimate of the seeing based on scintillation, and
COMMENT       the fraction of time the pointing loop was closed. The SGS Disk
COMMENT       Intensity Monitor (DIM) provides the Sun intensity signal.
EXTEND  =                      F / No FITS extensions
O_BZERO =                32768 / Original BZERO Value

For some reason, PRODUCT is not present.

mgalloy commented 1 month ago

OK, the headers look better now:

level1$ fitscat -r 20191217_020943_kcor_l1_stokesIQU.fts
SIMPLE  =                    T / Written by IDL:  Thu Aug  8 11:06:49 2024
BITPIX  =                  -64 / IEEE double precision floating point
NAXIS   =                    3 / number of dimensions; FITS image
NAXIS1  =                 1024 /Number of positions along axis 1
NAXIS2  =                 1024 /Number of positions along axis 2
NAXIS3  =                    3 /Number of positions along axis 3
EXTEND  =                    F / no FITS extensions
DATE-OBS= '2019-12-17T02:09:43' / UTC observation start
DATE-END= '2019-12-17T02:09:58' / UTC observation end
MJD-OBS =      58834.090081019 / [days] modified Julian date
MJD-END =      58834.090254630 / [days] modified Julian date
TIMESYS = 'UTC     '           / date/time system: Coordinated Universal Time
DATE_HST= '2019-12-16T16:09:43' / MLSO observation date [HST]
LOCATION= 'MLSO    '           / Mauna Loa Solar Observatory, Hawaii
ORIGIN  = 'NCAR/HAO'           / Nat.Ctr.Atmos.Res. High Altitude Observatory
TELESCOP= 'COSMO K-Coronagraph' / COSMO: COronal Solar Magnetism Observatory
INSTRUME= 'COSMO K-Coronagraph' /
WAVELNTH=                  735 / [nm] center wavelength of bandpass filter
WAVEFWHM=                   30 / [nm] full width half max of bandpass filter
OBJECT  = 'Corona Stokes IQU'  / calibrated corona+sky in cartesian coordinates
PRODUCT = 'Stokes IQU'         / calibrated corona+sky in cartesian coordinates
DATATYPE= 'science '           / type of data acquired
OBSERVER= 'Berkey  '           / name of Mauna Loa observer
COMMENT                          HARDWARE MECHANISM KEYWORDS GROUPED BELOW
DARKSHUT= 'out     '           / dark shutter open (out) or closed (in)
COVER   = 'out     '           / cover in or out of the light beam
DIFFUSER= 'out     '           / diffuser in or out of the light beam
CALPOL  = 'out     '           / calibration polarizer in or out of beam
CALPANG =               -0.030 / calibration polarizer angle
EXPTIME =             0.002500 / [s] exposure time for each frame
NUMSUM  =                  512 / # frames summed per L0 img for each pol state
COMMENT                          SCALING KEYWORDS GROUPED BELOW
BUNIT   = 'Mean Solar Brightness' / [B/Bsun] mean solar disk brightness
BOPAL   =             1.38E-05 / [B/Bsun] opal transmission by Elmore at 775nm
BZERO   =                    0 / offset for unsigned integer data
BSCALE  =                1.000 / physical = data * BSCALE + BZERO
DATAMIN =          -3.3820E+02 / minimum value of data
DATAMAX =           8.2644E+03 / maximum value of data
DISPMIN =               -7E-09 / minimum value for display
DISPMAX =                7E-07 / maximum value for display
DISPEXP =                 0.70 / exponent value for display (d=b^DISPEXP)
DISPGAM =                 0.70 / gamma value for color table correction
COMMENT                          COORDINATE SYSTEM KEYWORDS GROUPED BELOW
WCSNAME = 'helioprojective-cartesian' / World Coordinate System (WCS) name
CTYPE1  = 'HPLN-TAN'           / [deg] helioprojective west angle: solar X
CRPIX1  =               512.50 / [pixel] solar X center (index origin=1)
CRVAL1  =                 0.00 / [arcsec] solar X sun center
CDELT1  =               5.6430 / [arcsec/pixel] solar X increment = platescale
CUNIT1  = 'arcsec  '           / unit of CRVAL1
CTYPE2  = 'HPLT-TAN'           / [deg] helioprojective north angle: solar Y
CRPIX2  =               512.50 / [pixel] solar Y center (index origin=1)
CRVAL2  =                 0.00 / [arcsec] solar Y sun center
CDELT2  =               5.6430 / [arcsec/pixel] solar Y increment = platescale
CUNIT2  = 'arcsec  '           / unit of CRVAL2
INST_ROT=                0.000 / [deg] rotation of the image wrt solar north
IMAGESCL=               5.6066 / [arcsec/pixel] image scale for this file
DSUN_OBS=       147223828712.8 / [m] distance to the Sun from observer
HGLN_OBS=                0.000 / [deg] Stonyhurst heliographic longitude
HGLT_OBS=               -1.105 / [deg] Stonyhurst heliographic latitude
PC1_1   =                1.000 / coord transform matrix element (1, 1) WCS std.
PC1_2   =                0.000 / coord transform matrix element (1, 2) WCS std.
PC2_1   =                0.000 / coord transform matrix element (2, 1) WCS std.
PC2_2   =                1.000 / coord transform matrix element (2, 2) WCS std.
COMMENT                          PROCESSING SOFTWARE KEYWORDS GROUPED BELOW
QUALITY = 'ok      '           / image quality
LEVEL   = 'L1      '           / level 1 inst.-corrected calibrated pB intensity
SOCKETCA= 'socketcam.exe'      / camera interface software filename
DATE_DP = '2024-08-08T17:06:44' / L1 processing date (UTC)
DPSWID  = '2.1.13-dev [49d15e6*]' / L1 data processing software (2024-08-06)
RCAMCORR= 'camera_calibration_MV-D1024E-CL-40257_02.5000_lut40257-20190307.ncdf'
TCAMCORR= 'camera_calibration_MV-D1024E-CL-40256_02.5000_lut40256-20190307.ncdf'
XCAMSH0 =                    2 / [px] image shift, + to right
XCAMSH1 =                    2 / [px] image shift, + to right
XCAMCSH0=                    2 / [px] cam corr shift to align w/ img, + to left
XCAMCSH1=                    2 / [px] cam corr shift to align w/ img, + to left
FIXCAMLC=                    1 / interp over bad pixels in camera lin correction
CALFILE = '20191217_190505_kcor_cal_v22_2.0.65_2.500ms.ncdf' / calibration file
DISTORT = 'dist_coeff_20191216_212805_kcor_darkcorrected_occulter_in_focus.sav'
CAMERAS = 'both    '           / cameras used in processing
ROLLCORR=                  0.7 / [deg] clockwise offset: spar polar axis align.
DMODSWID= '2016-05-26'         / date of demodulation software
OBSSWID = '20191125T093731'    / version of the LabVIEW observing software
COMMENT                          CAMERA OCCULTING KEYWORDS GROUPED BELOW
RCAMXCEN=               519.98 / [pixel] camera 0 raw X-coord occulting center
RCAMYCEN=               510.04 / [pixel] camera 0 raw Y-coord occulting center
RCAM_RAD=               180.22 / [pixel] camera 0 raw occulter radius
RCAM_DCX=               517.66 / [pixel] camera 0 dist cor occulter X center
RCAM_DCY=               515.75 / [pixel] camera 0 dist cor occulter Y center
RCAM_DCR=               182.64 / [pixel] camera 0 dist corrected occulter radius
TCAMXCEN=               516.34 / [pixel] camera 1 raw X-coord occulting center
TCAMYCEN=               516.06 / [pixel] camera 1 raw Y-coord occulting center
TCAM_RAD=               183.22 / [pixel] camera 1 raw occulter radius
TCAM_DCX=               517.92 / [pixel] camera 1 dist cor occulter X center
TCAM_DCY=               515.78 / [pixel] camera 1 dist cor occulter Y center
TCAM_DCR=               182.69 / [pixel] camera 1 dist corrected occulter radius
RCAMPOLS=                    0 / first state used in polarization demodulation
TCAMPOLS=                    0 / first state used in polarization demodulation
RSUN_REF=          6.96182E+08 / [m] solar radius doi:10.1051/0004-6361/20173215
COMMENT                          EPHEMERIS KEYWORDS GROUPED BELOW
RSUN_OBS=               975.10 / [arcsec] solar radius using ref radius 959.63"
RSUN    =               975.10 / [arcsec] solar radius (old standard keyword)
R_SUN   =               172.80 / [pixel] solar radius
SOLAR_P0=                9.406 / [deg] solar P angle applied (image has N up)
CRLT_OBS=               -1.105 / [deg] solar B angle: Carrington latitude
CRLN_OBS=              267.299 / [deg] solar L angle: Carrington longitude
CAR_ROT =                 2225 / Carrington rotation number
SOLAR_RA=              264.371 / [h] solar right ascension (hours)
SOLARDEC=              -23.336 / [deg] solar declination
COMMENT                          ENGINEERING KEYWORDS GROUPED BELOW
O1FOCS  =              133.441 / [mm] objective lens (01) focus position
RCAMFOCS=                0.000 / [mm] camera 0 focus position
TCAMFOCS=                0.000 / [mm] camera 1 focus position
MODLTRT =               35.100 / [deg C] modulator temperature
CALPOLID= 'AP38-006T-1'        / ID polarizer
DIFFSRID= 'mk4-opal'           / ID diffuser
FILTERID= 'ANDV13425-01'       / ID bandpass filter
O1ID    = 'Optimax '           / ID objective (O1) lens
LYOTSTOP= '35mm    '           / specifies if the 2nd Lyot stop is in the beam
OCCLTRID= 'OC-1018.9"'         / ID occulter, radius in arcsec
MODLTRID= 'A       '           / ID modulator
RCAMID  = 'MV-D1024E-CL-40257' / ID camera 0 (reflected)
TCAMID  = 'MV-D1024E-CL-40256' / ID camera 1 (transmitted)
RCAMLUT = '40257_20190307'     / bit-error only, RCAMCORR does non-linearity
TCAMLUT = '40256_20190307'     / bit-error only, TCAMCORR does non-linearity
COMMENT                          SPAR GUIDER SYSTEM KEYWORDS GROUPED BELOW
SGSDIMV =               8.3550 / [V] SGS DIM signal mean
SGSDIMS =            1.314E-02 / [V] SGS DIM signal standard deviation
SGSSUMV =               8.3640 / [V] mean SGS sum signal
SGSSUMS =            9.924E-03 / [V] SGS sum signal standard deviation
SGSRAV  =            6.110E-04 / [V] mean SGS RA error signal
SGSRAS  =            3.181E-02 / [V] mean SGS RA error standard deviation
SGSRAZR =              15.0000 / [arcsec] SGS RA zeropoint offset
SGSDECV =           -4.600E-05 / [V] mean SGS DEC error signal
SGSDECS =            1.457E-02 / [V] mean SGS DEC error standard deviation
SGSDECZR=             -55.0000 / [arcsec] SGS DEC zeropoint offset
SGSSCINT=               3.0200 / [arcsec] SGS scintillation seeing estimate
SGSLOOP =                    1 / SGS loop closed fraction
DATACITE= 'https://doi.org/10.5065/D69G5JV8' / URL for DOI
COMMENT The COSMO K-coronagraph is a 20-cm aperture, internally occulted
COMMENT coronagraph, which observes the polarization brightness of the corona
COMMENT with a field-of-view from ~1.05 to 3 solar radii in a wavelength range
COMMENT from 720 to 750 nm. Nominal time cadence is 15 seconds.
COMMENT
HISTORY Level 1 calibration and processing steps: dark current subtracted; gain
HISTORY correction; apply polarization demodulation matrix; apply distortion
HISTORY correction; align each camera to center, rotate to solar north and
HISTORY combine cameras; coordinate transformation from cartesian to tangential
HISTORY polarization.

mgalloy commented 1 month ago

See results in raw.intermediate for 20240409 for validation.

NCAR / kcor-pipeline

Create level 1 IQU intermediate product #378

Tasks