Create demodulation matrices for new wave regions and verify demodulation for old wave regions

[mgalloy]

Create a demodulation matrix for each of the new wave regions from data observed in late November 2022.

Basic steps:

1. Use UCoMP_Pol_Cal.pro to create a demodulation matrix for each day with polarimetric data after 20221026.
2. Then use UCoMP_Pol_Cal_Temperature.pro to combine days results to give a demodulation matrix with a temperature dependence.

Eventually, this should move into the pipeline infrastructure, but for now could just change the hard-coded paths from Steve's code.

Conclusions

My conclusions right now:

• I was able to reproduce the individual calibration file results that Steve got for 20221026 exactly.
• I have individual demodulation matrices for the new wave regions, but creating the temperature dependent fit, using only date from 20221118 and 20221119, failed. I needed somewhere between 35-48 calibrations (I just kept adding 1074 nm calibrations from the past until a 1074 nm temperature fit worked) to have enough data to make a fit, but we only have 2-4 calibrations for the new wave regions.
• Potential path forward: The demodulation matrices from the other wave regions could be used to tentatively demodulate any new wave region data we take until, sometime in the future, we have enough calibration data to create a temperature fit for the new wave regions. Then we could go back to reprocess the old data with the correct demodulation matrices. It would probably take about 3 months of taking calibration data at the old pace to get enough to make the fit.

Useful information

• Polarimetric calibration data is collected for the new wave regions on 20221118 and 20221119.
• The new wave regions are 670, 761, 802, and 991.

Tasks

• [x] create copy of all routines needed to produce calibration with modified paths
• [x] make Steve version of catalog file for 20221118 and 20221119 and point UCoMP_Read_Catalog_File.pro to it
• [x] run UCoMP_Pol_Cal on 20221118
• [x] run UCoMP_Pol_Cal on 20221119
• [x] modify UCoMP_Pol_Cal_Temperature to add new wave regions
• [x] combine results for the days to give a demodulation matrix with temperature dependence using UCoMP_Pol_Cal_Temperature
• [ ] collect enough data (about two months) to create reliable temperature dependent demodulation matrices for the new wave regions

Routines to find

• [x] openpost.pro, closepost.pro, and dependencies (doesn't seem to be the UCOMP subversion or in SolarSoft) found in Documents/idl/Plotting/ in the backup of Steve's data -- there are other versions, but I can tell this is the correct version
• [x] mueller_retarder.pro, mueller_polarizer.pro, and dependencies (doesn't seem to be the UCOMP subversion) found in Documents/idl/Plotting/ in the backup of Steve's data, but there are other versions that I can't tell if they are the correct ones

Summary

The 20220409 polarimetric calibration data, especially for 1074 nm, is not in line with the coefficient fits (see #265). Waiting for more polarimetric calibration data when we reopen to create these demodulation matrices.

[mgalloy]

I've gotten Steve's code running and produced D matrices for 20221118 and 20221119.

• 20221118 plots
• 20221119 plots

[mgalloy]

I ran 20221026 as a test date to compare Steve's results to mine to make sure I am getting the same results as Steve on a date where I had Steve's results. The demodulation matrices are all exactly the same, but the temperatures are wrong, mine are showing up as 0.0.

For example, for me:

Wavelength:      637.400
Unfiltered Temperature:      0.00000

and for Steve:

Wavelength:      637.400
Unfiltered Temperature:      32.7730

This will affect creating the temperature dependent fit in the next step.

[mgalloy]

I have fixed the issue with the temperatures. Mine now reads the same as Steve's:

Wavelength:      637.400
Unfiltered Temperature:      32.7730

[mgalloy]

The UCoMP_Pol_Cal_Temperature fit is failing with:

           2 sav files
           4 total calibrations at       637.400
           1 cals rejected due to bad chisq
       0 points identified as outliers
... large array of all zeros omitted ...
           1 cals rejected due to recurrent bad points
% Compiled module: POLY_FIT.
% POLY_FIT: Singular matrix detected.

I did check to see if the temperatures were different and they were:

      33.1440      33.1550      33.0770      33.0790

[mgalloy]

I don't think we have enough reading to fit the temperature. Using UCoMP_Pol_Cal_Temperature I had to include the August-November calibrations to get a 1074 nm temperature dependence, about 48 1074 nm calibrations — the 35 calibrations from September-November was not enough. We only have 2-4 calibrations for the new wave regions.

[mgalloy]

The DMatrix coefficients, it seems we have agreement to maximum of about 0.003 (1.40181 - 1.40454 = -0.00273).

For 20221118:

Dmatrix:
  -0.25433   0.66068   0.05861   0.53552
   0.33624  -0.80948   0.53646  -0.06347
  -0.28015   0.15453   1.15413  -1.02808
   1.40181  -0.73843  -0.25481  -0.40767

For 20221119:

Dmatrix:
  -0.25433   0.66139   0.05836   0.53522
   0.33627  -0.81005   0.53625  -0.06264
  -0.28197   0.15620   1.15623  -1.02889
   1.40454  -0.73967  -0.25390  -0.40889

IDL> d18 - d19
       0.0000000  -0.00071001053   0.00025000051   0.00029999018
  -3.0010939e-05   0.00056999922   0.00020998716  -0.00083000213
    0.0018199980   -0.0016700029   -0.0020999908   0.00081002712
   -0.0027298927    0.0012399554  -0.00091001391    0.0012200177

[bberkeyU]

Looking at the de-modulation coefficients across the wave regions. It seems like there is a fairly smooth quadratic relationship between the wavelength and coefficient. So we should be able to safely project to new wavelengths.

https://github.com/NCAR/ucomp-studies/blob/master/demodulation/de-modcheck.ipynb

[mgalloy]

@bberkeyU How much variation of temperature do you expect now? If there is no (or very little) variation, we can make a constant demodulation matrix even from the few readings we have.

[bberkeyU]

@mgalloy I pulled the tu_mod values from the database for Dec 2021->Nov 2022. The results were not exactly what I expected. Somewhere in April 2022, it looks like the temperature started slowly rising, moving toward 32.5C. We also have some big excursions over the course of a couple of days.

[detoma]

• We need to check (for the old lines) if the new demodulation matrix is consistent with the old demodulation matrices. After so much time closed, this is not something we can take for granted.

• We should check when was the last time Steve updated the demodulation matrix. I would update it again, especially if the last update was done a while back. If the new polarimetric calibration looks strange we will not use it for the update and need to understand what that means.

[detoma]

The 20220409 polarimetric calibration coefficient for the old wavelengths, especially for 1074 and 1079 nm are not in agreement with the old demodulation coefficients (see closed ticket https://github.com/NCAR/ucomp-pipeline/issues/265). In some cases they differ by 10% or more. Plots were produced by Ben to compare the new coefficients with the old ones. The plots can be found in this notebook:

https://github.com/NCAR/ucomp-studies/blob/master/demodulation/Plot%20UCoMP%20demodulation%20vs%20temp%20with%20new%202024%20data.ipynb

We do not have an explanation for why there is such difference and it is hard to draw conclusions with only one data point. We need to take more polarimetric data when the observatory reopens.