NCAR / ucomp-pipeline

Data processing pipeline for UCoMP
Other
6 stars 3 forks source link

Correct the distortion at all wavelengths #185

Open StevenTomczyk opened 1 year ago

StevenTomczyk commented 1 year ago

Description: We calibrate the distortion introduced by the UCoMP optical system (from the occulter to the cameras) by measuring the deviations in an image of a distortion mask (or grid) with equally spaced dots. The grid is placed at the location of the occulter and is imaged by both beams, so this also allows us to register the two beams. We assume that any subsequent small drifts in the x-y location of the cameras can be corrected with a small offset.

Issue: Currently, we use distortion coefficients measured at 1074 nm to correct the distortion at all the wavelengths. This is because it is difficult to measure at other wavelengths.

Questions: Is the distortion a function of wavelength? We could check this by comparing distortion coefficients at different wavelengths? Can the distortion at other wavelengths be computed by scaling from the 1074 coefficients or some other simple means?

Tasks

jburkepile commented 1 year ago

I took a quick look at the eccentricity of the occulters for different UCoMP wavelengths over the mission (these plots are produced by Mike). The plots show that 1074/1079 have the lowest eccentricity over the mission - as expected. The eccentricity is larger for the 637 nm line (also not surprising). The 789 nm line had larger eccentricity than the 637 nm line ... which was surprising. Also, all wavelengths show larger eccentricities in TCAM, especially starting in summer 2022. Plots are below.

20221025 ucomp 1074 mission eccentricity 20221025 ucomp 1079 mission eccentricity 20221025 ucomp 789 mission eccentricity 20221025 ucomp 637 mission eccentricity

detoma commented 1 year ago

The difference between 789 and 637 is not very large and mostly in the second half of 2022 in RCAM, more striking is the difference between TCAM and RCAM.

It would be useful to have median values over a 2-3 months period to compare.

mgalloy commented 10 months ago

For our test date, 20220901, the RCAM and TCAM radius differ by almost a 3/4 pixel in size by the end of the day (filename, RCAM radius, TCAM radius, RCAM radius - TCAM radius):

20220901.182014.ucomp.1074.l1.p3.fts   333.13   333.26   -0.13
20220901.182246.ucomp.1074.l1.p3.fts   333.16   333.28   -0.12
20220901.185824.ucomp.1074.l1.p3.fts   333.19   333.32   -0.13
20220901.190056.ucomp.1074.l1.p3.fts   333.17   333.34   -0.17
20220901.193635.ucomp.1074.l1.p3.fts   333.24   333.41   -0.17
20220901.193908.ucomp.1074.l1.p3.fts   333.49   333.42    0.07
20220901.200958.ucomp.1074.l1.p3.fts   332.72   332.87   -0.15
20220901.201032.ucomp.1074.l1.p3.fts   332.81   332.97   -0.16
20220901.201106.ucomp.1074.l1.p3.fts   332.90   333.06   -0.16
20220901.201139.ucomp.1074.l1.p3.fts   332.96   333.07   -0.11
20220901.201213.ucomp.1074.l1.p3.fts   333.04   333.19   -0.15
20220901.201246.ucomp.1074.l1.p3.fts   333.04   333.19   -0.15
20220901.201320.ucomp.1074.l1.p3.fts   333.06   333.24   -0.18
20220901.201354.ucomp.1074.l1.p3.fts   333.06   333.24   -0.18
20220901.201427.ucomp.1074.l1.p3.fts   333.07   333.22   -0.15
20220901.201501.ucomp.1074.l1.p3.fts   333.13   333.29   -0.16
20220901.201534.ucomp.1074.l1.p3.fts   333.15   333.33   -0.18
20220901.201608.ucomp.1074.l1.p3.fts   333.18   333.29   -0.11
20220901.201642.ucomp.1074.l1.p3.fts   333.21   333.37   -0.16
20220901.201715.ucomp.1074.l1.p3.fts   333.26   333.42   -0.16
20220901.201749.ucomp.1074.l1.p3.fts   333.24   333.43   -0.19
20220901.201822.ucomp.1074.l1.p3.fts   333.27   333.34   -0.07
20220901.201856.ucomp.1074.l1.p3.fts   333.25   333.41   -0.16
20220901.201930.ucomp.1074.l1.p3.fts   333.26   333.40   -0.14
20220901.202003.ucomp.1074.l1.p3.fts   333.24   333.40   -0.16
20220901.202037.ucomp.1074.l1.p3.fts   333.27   333.45   -0.18
20220901.202110.ucomp.1074.l1.p3.fts   333.26   333.40   -0.14
20220901.202144.ucomp.1074.l1.p3.fts   333.34   333.42   -0.08
20220901.202218.ucomp.1074.l1.p3.fts   333.27   333.48   -0.21
20220901.202251.ucomp.1074.l1.p3.fts   333.24   333.46   -0.22
20220901.202325.ucomp.1074.l1.p3.fts   333.31   333.47   -0.16
20220901.202358.ucomp.1074.l1.p3.fts   333.31   333.45   -0.14
20220901.202432.ucomp.1074.l1.p3.fts   333.29   333.52   -0.23
20220901.202505.ucomp.1074.l1.p3.fts   333.26   333.42   -0.16
20220901.202539.ucomp.1074.l1.p3.fts   333.26   333.40   -0.14
20220901.202613.ucomp.1074.l1.p3.fts   333.30   333.41   -0.11
20220901.202646.ucomp.1074.l1.p3.fts   333.29   333.50   -0.21
20220901.202720.ucomp.1074.l1.p3.fts   333.29   333.48   -0.19
20220901.202753.ucomp.1074.l1.p3.fts   333.24   333.40   -0.16
20220901.202827.ucomp.1074.l1.p3.fts   333.29   333.41   -0.12
20220901.202901.ucomp.1074.l1.p3.fts   333.28   333.43   -0.15
20220901.202934.ucomp.1074.l1.p3.fts   333.31   333.48   -0.17
20220901.203008.ucomp.1074.l1.p3.fts   333.31   333.47   -0.16
20220901.203041.ucomp.1074.l1.p3.fts   333.29   333.44   -0.15
20220901.203115.ucomp.1074.l1.p3.fts   333.29   333.49   -0.20
20220901.203149.ucomp.1074.l1.p3.fts   333.25   333.37   -0.12
20220901.203222.ucomp.1074.l1.p3.fts   333.36   333.41   -0.05
20220901.203256.ucomp.1074.l1.p3.fts   333.28   333.55   -0.27
20220901.203329.ucomp.1074.l1.p3.fts   333.27   333.51   -0.24
20220901.203403.ucomp.1074.l1.p3.fts   333.32   333.51   -0.19
20220901.203436.ucomp.1074.l1.p3.fts   333.39   333.50   -0.11
20220901.203510.ucomp.1074.l1.p3.fts   333.27   333.49   -0.22
20220901.203544.ucomp.1074.l1.p3.fts   333.29   333.49   -0.20
20220901.203617.ucomp.1074.l1.p3.fts   333.25   333.46   -0.21
20220901.203651.ucomp.1074.l1.p3.fts   333.35   333.49   -0.14
20220901.203724.ucomp.1074.l1.p3.fts   333.21   333.41   -0.20
20220901.203758.ucomp.1074.l1.p3.fts   333.29   333.45   -0.16
20220901.203832.ucomp.1074.l1.p3.fts   333.28   333.47   -0.19
20220901.203905.ucomp.1074.l1.p3.fts   333.38   333.52   -0.14
20220901.203939.ucomp.1074.l1.p3.fts   333.39   333.53   -0.14
20220901.204012.ucomp.1074.l1.p3.fts   333.32   333.57   -0.25
20220901.204046.ucomp.1074.l1.p3.fts   333.31   333.50   -0.19
20220901.204120.ucomp.1074.l1.p3.fts   333.28   333.56   -0.28
20220901.204153.ucomp.1074.l1.p3.fts   333.22   333.42   -0.20
20220901.204227.ucomp.1074.l1.p3.fts   333.25   333.45   -0.20
20220901.204300.ucomp.1074.l1.p3.fts   333.33   333.50   -0.17
20220901.204334.ucomp.1074.l1.p3.fts   333.26   333.45   -0.19
20220901.204408.ucomp.1074.l1.p3.fts   333.22   333.44   -0.22
20220901.204441.ucomp.1074.l1.p3.fts   333.23   333.42   -0.19
20220901.204515.ucomp.1074.l1.p3.fts   333.25   333.52   -0.27
20220901.204548.ucomp.1074.l1.p3.fts   333.26   333.47   -0.21
20220901.204622.ucomp.1074.l1.p3.fts   333.30   333.47   -0.17
20220901.204656.ucomp.1074.l1.p3.fts   333.34   333.49   -0.15
20220901.204729.ucomp.1074.l1.p3.fts   333.33   333.52   -0.19
20220901.204803.ucomp.1074.l1.p3.fts   333.37   333.46   -0.09
20220901.204836.ucomp.1074.l1.p3.fts   333.33   333.54   -0.21
20220901.204910.ucomp.1074.l1.p3.fts   333.37   333.49   -0.12
20220901.204944.ucomp.1074.l1.p3.fts   333.24   333.49   -0.25
20220901.205017.ucomp.1074.l1.p3.fts   333.29   333.53   -0.24
20220901.205051.ucomp.1074.l1.p3.fts   333.24   333.41   -0.17
20220901.205124.ucomp.1074.l1.p3.fts   333.37   333.55   -0.18
20220901.205158.ucomp.1074.l1.p3.fts   333.36   333.52   -0.16
20220901.205232.ucomp.1074.l1.p3.fts   333.36   333.54   -0.18
20220901.205305.ucomp.1074.l1.p3.fts   333.28   333.48   -0.20
20220901.205339.ucomp.1074.l1.p3.fts   333.27   333.49   -0.22
20220901.205412.ucomp.1074.l1.p3.fts   333.22   333.45   -0.23
20220901.205446.ucomp.1074.l1.p3.fts   333.36   333.43   -0.07
20220901.205520.ucomp.1074.l1.p3.fts   333.30   333.56   -0.26
20220901.205553.ucomp.1074.l1.p3.fts   333.35   333.55   -0.20
20220901.205627.ucomp.1074.l1.p3.fts   333.28   333.56   -0.28
20220901.205700.ucomp.1074.l1.p3.fts   333.21   333.40   -0.19
20220901.205734.ucomp.1074.l1.p3.fts   333.26   333.41   -0.15
20220901.205808.ucomp.1074.l1.p3.fts   333.36   333.46   -0.10
20220901.205841.ucomp.1074.l1.p3.fts   333.29   333.53   -0.24
20220901.205915.ucomp.1074.l1.p3.fts   333.26   333.44   -0.18
20220901.205948.ucomp.1074.l1.p3.fts   333.27   333.41   -0.14
20220901.210022.ucomp.1074.l1.p3.fts   333.37   333.50   -0.13
20220901.210056.ucomp.1074.l1.p3.fts   333.31   333.57   -0.26
20220901.210129.ucomp.1074.l1.p3.fts   333.34   333.53   -0.19
20220901.210203.ucomp.1074.l1.p3.fts   333.28   333.52   -0.24
20220901.210236.ucomp.1074.l1.p3.fts   333.34   333.43   -0.09
20220901.210310.ucomp.1074.l1.p3.fts   333.19   333.47   -0.28
20220901.210344.ucomp.1074.l1.p3.fts   333.16   333.36   -0.20
20220901.210417.ucomp.1074.l1.p3.fts   333.20   333.39   -0.19
20220901.210451.ucomp.1074.l1.p3.fts   333.33   333.51   -0.18
20220901.210524.ucomp.1074.l1.p3.fts   333.34   333.50   -0.16
20220901.210558.ucomp.1074.l1.p3.fts   333.32   333.58   -0.26
20220901.210631.ucomp.1074.l1.p3.fts   333.29   333.54   -0.25
20220901.210705.ucomp.1074.l1.p3.fts   333.29   333.61   -0.32
20220901.210739.ucomp.1074.l1.p3.fts   333.29   333.48   -0.19
20220901.210812.ucomp.1074.l1.p3.fts   333.29   333.54   -0.25
20220901.210846.ucomp.1074.l1.p3.fts   333.28   333.56   -0.28
20220901.210919.ucomp.1074.l1.p3.fts   333.27   333.48   -0.21
20220901.210953.ucomp.1074.l1.p3.fts   333.35   333.44   -0.09
20220901.211027.ucomp.1074.l1.p3.fts   333.34   333.53   -0.19
20220901.211100.ucomp.1074.l1.p3.fts   333.37   333.59   -0.22
20220901.211134.ucomp.1074.l1.p3.fts   333.27   333.55   -0.28
20220901.211207.ucomp.1074.l1.p3.fts   333.30   333.55   -0.25
20220901.211241.ucomp.1074.l1.p3.fts   333.25   333.44   -0.19
20220901.211315.ucomp.1074.l1.p3.fts   333.25   333.42   -0.17
20220901.211348.ucomp.1074.l1.p3.fts   333.35   333.53   -0.18
20220901.211422.ucomp.1074.l1.p3.fts   333.33   333.52   -0.19
20220901.211455.ucomp.1074.l1.p3.fts   333.39   333.59   -0.20
20220901.211529.ucomp.1074.l1.p3.fts   333.25   333.49   -0.24
20220901.211602.ucomp.1074.l1.p3.fts   333.30   333.49   -0.19
20220901.211636.ucomp.1074.l1.p3.fts   333.26   333.49   -0.23
20220901.222919.ucomp.1074.l1.p3.fts   333.31   333.58   -0.27
20220901.223152.ucomp.1074.l1.p3.fts   333.32   333.62   -0.30
20220901.232814.ucomp.1074.l1.p3.fts   333.25   333.57   -0.32
20220901.233046.ucomp.1074.l1.p3.fts   333.26   333.60   -0.34
20220902.004837.ucomp.1074.l1.p3.fts   332.67   333.19   -0.52
20220902.005109.ucomp.1074.l1.p3.fts   332.72   333.21   -0.49
20220902.020509.ucomp.1074.l1.p3.fts   332.53   333.21   -0.68
20220902.020742.ucomp.1074.l1.p3.fts   332.48   333.20   -0.72
20220902.024313.ucomp.1074.l1.p3.fts   332.48   333.22   -0.74
20220902.024545.ucomp.1074.l1.p3.fts   332.55   333.24   -0.69
20220902.032124.ucomp.1074.l1.p3.fts   332.74   333.48   -0.74
20220902.032356.ucomp.1074.l1.p3.fts   332.68   333.46   -0.78
mgalloy commented 10 months ago

Here is the difference between the camera radii graphed for 1074 nm:

ucomp camera-radius-difference

bberkeyU commented 10 months ago

The big jumps in daily scattered seem to be related to occulter size changes:

jburkepile commented 10 months ago

Good info Ben! Can you think of why there is no big change after the 4/30/22 occulter change? There should have been 2 more occulter changes after that. Could it be due to the fact that you purposely removed much of the occulter station hardware to reduce scattered light?

detoma commented 8 months ago

Mike made a special run without the distortion correction (and no hot pixels correction). The level 1 images look a lot better than the ones with the distortion correction from the wrong camera applied. We are doing significant damage to the images by applying the wrong distortion correction. This does not mean of course that we can process data without distortion. Distortion is a critical step, but the late November images were so bad looking because we used the old pipeline to process them. Examples are below.

Level 1 image with old distortion correction applied:

Screenshot 2024-02-15 at 9 43 12 AM

Level1 image without no distortion correction:

Screenshot 2024-02-15 at 9 44 23 AM

Note: images are flipped

jburkepile commented 8 months ago

Yes! Dramatic improvement - you are absolutely correct - using the wrong distortion is much worse than no distortion at all.

To your comment below, I didn't mean to imply we don't need a proper distortion. I totally agree that we do need one.

detoma commented 8 months ago

We still need to apply the proper distortion correction. Distortion is critically important for polarization data.

This no-distortion correction run should be our baseline to compare with, not the old pipeline run.

bberkeyU commented 8 months ago

Interestingly, the bad distortion seems to show some wide oops near the pole that the no distortion seems to wash out.

I have called out the loops in the image below.

[image: image.png]

mgalloy commented 8 months ago

@bberkeyU Can you report above comment via the web interface? Replying via email lost your image (and also limits formatting abilities).

bberkeyU commented 8 months ago

image

jburkepile commented 8 months ago

Ben - these two images are not oriented the same way. There is a vertical flip between the two images. I asked Mike to process only 1 camera and there is no distortion applied. The vertical flip is done when the distortion is applied ... it was not done for the latest reprocessing

bberkeyU commented 8 months ago

Correct the post shows that the image was not flipped. I was just pointing out that the bad distortion showed some coronal structures that the undistorted didn't. I didn't investigate this very carefully to see if this was because the structures were processing artifacts in one version and not the other. Or if the wrong distortion acted like a sharpness mask and pulled out a structure washed out in the no-distortion version.

mgalloy commented 8 months ago

OK, I fixed the distortion step to flip the images the same way, but just not do the distortion. New images below:

20221119 185404 ucomp 1074 l1 p3 intensity

Only TCAM:

20221119 185404 ucomp 1074 l1 p3 intensity

mgalloy commented 5 months ago

This issue requires non-1074 nm distortion data to be taken, so it can't be done until we re-open.