Test newly produced distortion coefficients

[mgalloy]

Test @bberkeyU's newly produced distortion coefficients in the pipeline. This distortion assumes rotation is in the cameras and not the grid, so we distort both cameras to a square, non-rotated grid. As far as we can tell the new grids look ok.

Tasks

• [x] create an IDL save file of the current distortion file format from Ben's netCDF files (it will not be exact because Ben's file has a different number of coefficients)
• [x] replace the current distortion file with the new one
• [x] update distortion applying method to recognize the number of coefficients and correctly apply in each case
• [x] run on test date

Questions

• [x] how do I apply the xCam and yCam arrays to science data?
• [x] xCam and yCam are 1310720 element arrays, are they row major or column major? i.e., can I just do reform(xCam, 1280, 1024)?

Summary

When using a good distortion and updating Steve's code to change the assumptions about the image tilts, we got a good fit. Steve assumed the tilt always came from the grid but we know it comes from the cameras. The new code version does not rotate the ideal grid array to the average of camera rotations.

[mgalloy]

I have preliminary results in process.latest for 20221118. I am applying the correction incorrectly in same way because there is a large offset in the images.

[bberkeyU]

Hi Mike,

I assumed we would say:

dx0_c = cam0_xMap
dy0_c = cam0_yMap
dx1_c = cam1_xMap
dy1_c = cam1_yMap

But I forgot that IDL and Python treat arrays in different orders. So we probally do have to make the swap.

BTW my distortion correction indices have RCAM flipped to the correct orientation. Which I think is what you are expecting in ucomp_l1_distortion.pro because you flip the input data before applying distortion correction.

data[*, *, p, 0, e] = reverse(ucomp_apply_distortion(reverse(data[*, *, p, 0, e], 1), $
                                                           dx0_c, dy0_c), 2)

[bberkeyU]

Attaching an L1 image to show the geometry problem.

Unfortunately, the distortion correction didn't work well either, with a fuzzy occulter disk and smeared coronal features.

[mgalloy]

It looks like x and y were swapped, not sure if it happened with the original netCDF creation or when I created the save file. But the new results in process.latest are better — than the ones above, and I think from the original distortion.

[bberkeyU]

Mike. That is correct I got my X and Y mixed up. It will be an easy fix to correct.

[mgalloy]

Here is the new distortion for the same date/time as above:

And here is the current production version:

[bberkeyU]

Mike, I produced 2 new sets of distortion coefficients. It is not clear which is better. Would it be possible to run one day with both data sets?

Files are copied to

/home/mgalloy/software/ucomp-pipeline/resource/distortion

The first set has the filenames distortMap_......, the second distortMap2......

distortMap_20240327_10006_1074.nc
distortMap_20240327_10009_1074.nc

distortMap2_20240327_10006_1074.nc
distortMap2_20240327_10009_1074.nc

[mgalloy]

Results are in process.distortmap and process.distortmap2.

[detoma]

For the eclipse we reverted back to Steve's IDL code using the new distortion grids taken on 2024 03 30. They were much better quality and dit show significant tilt like the old ones from 2022.

We need to test the latest distortion coefficients given by Ben using the IDL code by applying the distortion to the grid data and look at the difference/ratio. There is no code so this needs to be done manually. It would be good to have some code that does that under UCOMP analysis because it is something we want to do any time we take a new grid or test a new distortion method.

Mike is also doing a run with intermediate products to further check the distortion.

[mgalloy]

The distortion corrected flat files (which include the distortion grid files) are in the distortion directory in process.distortion/20240330.

[bberkeyU]

@mgalloy What other processing was done to these files? Did we do the X-axis camera flip?

[mgalloy]

The full distortion step was done:

• flip RCAM on the x-axis
• apply the distortion
• flip both cameras on the y-axis

[bberkeyU]

@mgalloy I think I have a better set of distortion coefficients and have copied the save file to:

/home/mgalloy/software/ucomp-pipeline/resource/distortion/dist_coeff_20240330_1074_degree4.sav

Can you reprocess the grid files (ala process.distortion) into process.distortion2 with this new set of coefficients?

[bberkeyU]

@mgalloy Based on time stamps It looks like the pipeline finished processing process.distortion2 but I dont see the process.distortion2/20240330/distortion directory. Could you reprocess this again with the make distortion directory option on.

[mgalloy]

I had to process the date, and then run the distort command on it. Running now...

[mgalloy]

OK, the distortion directory in process.distortion2 is there now.

[bberkeyU]

New versions of the IDL code checked into subversion, and new coefficients were copied to dawn.

The IDL code has been cleaned up a bit and I think it might be good to move this code to github. The two most obvious locations seem to be ucomp-pipeline or ucomp-studies but neither seems perfect.

[mgalloy]

The coefficients are in dist_coeff_20240330_1074_degree3.sav from yesterday?

I have a calibration directory in the ucomp-pipeline repo where I am collecting code that produces the artifacts that are used by the pipeline. Right now, there is only a polarimetric directory that has a modified version of Steve's code to product demodulation matrices. A parallel directory for distortion would probably be the best place for this code.

[bberkeyU]

@mgalloy That is the best distortion as far as I know.

Next steps: Re-run the eclipse data dist_coeff_20240330_1074_degree3.sav with intermediate products so we can check the quality of the data with this distortion compared to the previous one.

[mgalloy]

@bberkeyU Should I close this issue, or move it to the 1.1 milestone?

[bberkeyU]

My personal should close issue 251. I am sure we will do a lot more work on distortion throughout the mission, but I dont see any more work on in the short term.

[detoma]

Testing the velocity I run into two files processed with the old and new version of the distortion. They are very very different.

Ben: Can you grab a AIA file on 2024 04 09 and check which orientation looks more consistent with AIA?

Can Don make a composite for 2024 04 09 with AIA and KCor?

[jburkepile]

I will ask Don to make composites. I suggest a 1074 image with an AIA 193 image. We assume the Sun is centered under the occulter. The corona moves in the 1074 images between 18:07 and 19:05. There is no observer log. Which do we think is has better pointing?

I will also ask him to make a K-Cor-AIA composite.

Don can't use fits images - Adobe illustrator does not accept fits format. He can use 'quicklooks' such as gif, png, etc. I ask him to scale AIA to the pixel size of K-Cor or UCoMP based on the platescale of the instruments.

[bberkeyU]

@detoma I have never worked with AIA, but I can try it. Can you elaborate on the differences you see between the old and new distortions and why there is a concern? I was expecting changes as we found better coefficients. Is the concern mostly rotation or something else?

@jburkepile We didn't have the obs fully backup on April 9th. Lisa made notes in an email she sent out to the group. But the notes dont have (or observatory logs in general) dont tell us anything about how well we are pointing.

[detoma]

Ben: do not worry about AIA. We will do it.

The rotation between the two distortion is very obvious, we need to find out which one gives a better orientation. The eclipse data will be used a lot.

If there is a motion over the day, this complicates things.

[detoma]

I tried a AIA/UCOMP composite using an image at 1905UT from the process.eclipse directory (not sure which distortion has). I did not have code to do composites, so I wrote something quickly. Don can do better. It does not look aligned in the NE, it looks better in then SW.

It seem I need a ~2deg rotation in the NE but it matches with no rotation in the SW. This point to a distortion issue and not a spar rotation.

[bberkeyU]

@detoma process. eclispe an old version of the distortion. I think this was the discropy( python version), but it might be another version of Steves's code I will try to check.

What does the process.hot-pixels-20221125_test version look like?

[bberkeyU]

We processed the eclipse data with 3 (or up to 5) versions of the distortion corrections. The distortion files with filenames that end in .sav were produced with some version of the IDL code. I think the 1.0.5 releases used the python code (which produced .nc files), but we didnt record the filename in the header.

process : 1.0.5 [b1417f0] No DISTORTF in the header
process.latest : 1.0.10-dev [bab404f4*] dist_coeff_20240330_1074_degree3.sav
process.4-term : 1.0.10-dev [733f97f4*] dist_coeff_20240330_1074_degree4.sav
process.3-term : 1.0.10-dev [733f97f4*] dist_coeff_20240330_1074_degree4.sav
process.hot-pixels-20240409_test : 1.0.10-dev [b6afa292*] dist_coeff_20240330_1074_degree3.sav
process.intermediate : 1.0.10-dev [c90a89d6*] No DISTORTF in the header
process.eclipse : 1.0.5-dev [e29c733b*] No DISTORTF in the header
process.analytic-gaussian : 1.0.10-dev [b6afa292*] dist_coeff_20240330_1074_degree3.sav

[mgalloy]

By the way, this is the current correct distortion files to use for the various time periods? Just the one switch when the instrument was updated in November 2022?

[20210526]
distortion_basename          : dist_coeff_20210708_1074.sav

[20221111]
distortion_basename          : dist_coeff_20240330_1074_degree3.sav

[detoma]

Process.latest has what Ben thinks is the best distortion. I used the file in that directory to do the AIA alignment and it looks better. We ca give the green light to that distortion file.

[bberkeyU]

For posterity, the process. latest plot uses dist_coeff_20240330_1074_degree3.sav, which uses the IDL code with a 3-degree fit.

When commuting the distortion coefficients, we assume the tilt in the image grids comes from a camera rotation, not a misalignment in the targets. Therefore, we commented out the section of the code that rotated the ideal-rectilinear target array to the average rotation of the two cameras.