Apply sky transmission correction

[mgalloy]

Use SGSDIMV to compute a sky transmission from the ratio of the flat SGSDIMV to the science SGSDIMV:

corona *= flat_sgsdimv / sci_sgsdimv

The SGSDIMV value for the flat has to be averaged between extensions that are averaged together and then interpolated if flat interpolation is on. The SGSDIMV for the science image has to be averaged for the science file.

Because of the difference in wavelengths between the SGS and the observed line, the formula should actually be:

corona *= model(flat_sgsdimv, wave_region) / model(sci_sgsdimv, wave_region)

Tasks

• [x] do first order correction
• [ ] develop model based on wavelength and use that correction

[!NOTE] Time estimate: Once a scientists constructs the model for determining the wavelength specific sky transmission, this should be only another hour for me. The basic hooks to the flat SGSDIMV and science SGSDIMV values are already in place.

[mgalloy]

For 20220902.024545.ucomp.1074.l1.3.fts:

FLTFILE1= '20220902.011415.71.ucomp.1074.l0.fts' / name of raw flat file used
FLTEXTS1= '1,7,13,19'          / 20220902.011415.71.ucomp.1074.l0.fts ext used
MFLTEXT1= '31,34   '           / 20220901.ucomp.1074.flat.fts ext, wt 0.40
FLTFILE2= '20220902.034702.73.ucomp.1074.l0.fts' / name of raw flat file used
FLTEXTS2= '1,7,13,19'          / 20220902.034702.73.ucomp.1074.l0.fts ext used
MFLTEXT2= '37,40   '           / 20220901.ucomp.1074.flat.fts ext, wt 0.60
SKYTRANS=                0.958 / sky transmission correction normalized to gain

SGSDIMV for the relevent master flats extensions:

 31     8.313
 34     8.308
 37     7.051
 40     7.036

Should give an SGSDIMV for the flats:

0.4 * (8.313 + 8.308) + 0.6 * (7.051 + 7.036) = 7.5503

The SGSDIMV for the science file:

SGSDIMV =              8.01208 / [V] SGS Dim Mean

So SKYTRANS should be 0.94236453.

[mgalloy]

Results for 20220902.024545.ucomp.1074.l1.3.fts:

$ ucomp cat -k SKYTRANS 20220902.024545.ucomp.1074.l1.3.fts
  1     0.942
  2     0.942
  3     0.942
  4     7.552
  5     7.550
  6     7.549

I need to fix the background, but the onband images seem correct now.

[mgalloy]

OK, I think I have the sky transmission working now. Results for 20220902.024545.ucomp.1074.l1.3.fts:

level1$ ucomp cat -k SKYTRANS 20220902.024545.ucomp.1074.l1.3.fts
  1     0.942
  2     0.942
  3     0.942
  4     0.942
  5     0.942
  6     0.942

[jburkepile]

Mike: The sky transmission is a function of wavelength. Back in ~2012 I required that the SGS have a passband filter the same as K-Cor (720 to 750 nm) so we could correct sky transmission for K-Cor: We have to extrapolate the SGS wavelengths to both shorter and longer wavelengths. The sky is brighter at the shorter wavelengths and it is darker at the longer wavelengths. We need to work with Steve to model the sky transmission from the K-Cor wavelength to the various UCoMP wavelengths.

[bberkeyU]

I started poking around the 1074 data for 2022.

For some reason, we ran the waves program as flats instead of coronal images.on 2022-05-08 and 2022-05-09. This gives us fairly dense measurements of flats from about 50-80 degree elevations. It should be noted that the 2022-05-09 waves program started a little later due to some clouds and may have suffered some cirrus above 80-degree elevation.

We are zooming out to March-November 2022. The data has a lot of vertical spread, so we probably need to perform dark corrections to combine this data. However, there are 2 populations of data sparser blue/orange curves with values 325->340. Then a second denser collection of points ranging in value from 305-325 counts. I still need to dig into the difference between these two populations. But if we just look at one population, the flat values have a wide M shape with a sharp rise in the 20 degrees above the horizon and a slow dip down toward the zenith.

Note when looking west, the elevation is reported as 180-elevation (such that el=80deg gets reported as 100 deg and el=10 near sunset is reported at 170)

[bberkeyU]

Comparing the mean(flats)/sgsdimv vs Airmass I get the following relationship:

For this analysis, I looked at the L0 rcam flat values when the recipe wavelength=1074.7, onband=rcam, and contin=both. For each extension meeting this criteria I took the mean of the flat values across the 4x1280x1024 images. No darks were subtracted. I also saved the SGSDIMV value associated with that extension.

This gives raw flat values over the course of the mission of the following: Somewhere around x=1200 ~ mid-February 2022 we switched to gain and offset corrected NUC and the flat values stabilized a little under 350 counts. Applying the metric that we only want to look at flat values near the 300 counts, I threw out all the files that had mean flats above 500 or below 300 counts and get:

Looking at the SGSDIMV across the whole mission, we see: With some very low results early on in the mission before we fixed the guider pointing so we will adopt the metric that SGS should be above 7V to match the end of the mission. We get:

The SGS > 7V forces us to look at data taken after June 2022.

For the airmass. I am using sunpy/astropy's pointing transformations from MLSO at the timestamps given in the fits file names to produce altitude values. And then computing the airmass a 1/cos(90-altitude)

To me, it looks like at least for 1074 there is a fairly linear relationship between the airmass extinction in the SGS bandpass and the 1074 bandpass.

[StevenTomczyk]

@bberkeyU This looks like it can be useful for developing a mean sky transmission correction. I think you should subtract the dark from the flats.

[bberkeyU]

@StevenTomczyk Is it worth trying to implement dark interpolation, or can I just use the latest dark taken before the the flat?

[StevenTomczyk]

I think the last dark would be OK, but I think Mike interpolates darks for the flat files.

[mgalloy]

The pipeline has an option whether to interpolate. The default is to interpolate.

[detoma]

We expect the sky transmission correction changes with wavelength region. After discussing it we decided to:

• Not to apply the SGS DIMV correction from KCor to UCoMP. The flat interpolation should take care of it for most data, except for data for which there is only one flat and is not close in time (usually early morning and late afternoon)
• We will test how well the dark-corrected flat median value correlates with the SGS DIMV value from KCor (and as a by-product, we will check the stability of the dark-corrected flats over a day and from day to day.
• For data early in the morning and late afternoon when we do not have a good way to interpolate flats, we should use a model to account for the clear-sky transmission as a function of wavelength and zenith angle ad for the absorption due to water vapor and aerosols. AERONET can provide the water vapor and aerosol data we need. This requires an atmospheric model like MODTRAN and will be implemented at a later time