Revise masking for level 2 data

detoma commented 6 months ago

Giuliana to review and finalize masking criteria.

Tasks

[x] update int_min_1074_thresh for epochs
[x] update intensity and enhanced intensity masking
[x] determine occulter radius for intensity images that are "masked with the geometry l2 mask with the occulter mask fixed over the day" use occulter ID to lookup size of occulter radius (convert to pixels using plate scale)
[x] update velocity and line width masking
[x] update polarization data (Q, U, V, L and azimuth) masking

detoma commented 6 months ago

The revised mask for the level 2 data in the December test run will be the following:

INTENSITY & ENHANCED INTENSITY

Intensity & enhanced intensity are only masked with the geometry l2 mask with the occulter mask fixed over the day, i.e., we do not use the occulter radius found for that image when fitting a circle but the nominal occulter radius. This was decided to fix the size of the occulter over a day.

VELOCITY AND LINE WIDTH

The velocity and line width will have the same geometry mask as the intensity plus the other conditions below:

good = where(geo_mask gt 0 and velocity ne 0 and abs(velocity) lt 100 $
                                    and intensity[*,*,0] gt 0.1 and intensity[*,*,1] gt int_thr and intensity[*,*,2] gt 0.1 $
                                    and intensity[*,*,0] lt 60  and intensity[*,*,1] lt 60 and intensity[*,*,2] lt 60 $
                                    and line_fwhm gt 36.0 and line_fwhm lt 170.0,  ngood, complement=bad)

Please note that in the condition above line_fwhm is the FWHM in km/s. If the e-folding W is used the min and max would be: 22 and 102.

The minimum correspond to the instrument line width with no thermal and velocity effect taken into account and the maximum includes line broadening and a velocity 100 km/s. The maximum must be tested on CME events to make sure it does not remove good pixels.

the intensity threshold int_thr in the equation above changes over time:

year 2013 int_thr = 0.45
year 2014 and 2015 int_thr = 0.4
year 2016 Jun-Jul-Aug-Sep int_thr = 0.35
year 2016 Oct-Nov-Dec int_thr = 0.3
year 2017 int_thr = 0.25
year 2018 int_thr = 0.2

POLARIZATION DATA (Q, U, V, L and azimuth)

We will use less masking. We will use the conditions:

good = where(geo_mask gt 0 $
                       and intensity[*,*,0] gt 0.05 and intensity[*,*,1] gt 0.25 and intensity[*,*,2] gt 0.05 $
                       and intensity[*,*,0] lt 60  and intensity[*,*,1] lt 60 and intensity[*,*,2] lt 60 )

This may let too much noise in the images. We will evaluate the test run results and adjust as needed.

detoma commented 5 months ago

The values to use for the radius are the not the occulter radius from the fit, but a pre-determined value based on the occulter used. This was done to avoid. in movies the radius to change (as seeing degrades the fitted radius usually increase or fluctuates) which was confusing to users.

I think this was implemented in January 2023 and the values included in the epoch file. Mike, pleased double check. The values are the followings:

after 2012 Dec 9

id = 27 radius = 226.8 id = 31 radius = 229.6 id = 35 radius = 232.8

before 2012 Dec 9

id = 27 radius = 221.5 id = 31 radius = 224.0 id = 35 radius = 227.0

These values were based on mean/median values of the occulter see Feb 2-6 2023 emails.

detoma commented 5 months ago

The occulter-offset to add to the values above is +3 pixels for the level2 masking. Other places in the code may use a different value.

This should be sufficient to give a good mask, i.e. no spoilage, but it needs to be verified.

detoma commented 5 months ago

These are the plots and mean values we base the radius value on See February 6 2022 email.

Epoch 1 is Jan 1, 2012 to Dec 7, 2012 and Epoch 2 is May 1, 2013 to April 5, 2018.

Epoch 1: Radius 1: Occulter: 27, radius mean: 221.47, median: 221.47, stdev: 0.96 Epoch 1: Radius 2: Occulter: 27, radius mean: 221.50, median: 221.51, stdev: 0.84

Epoch 2: Radius 1: Occulter: 27, radius mean: 226.87, median: 226.79, stdev: 0.81 Epoch 2: Radius 2: Occulter: 27, radius mean: 226.74, median: 226.65, stdev: 0.85

Epoch 1: Radius 1: Occulter: 31, radius mean: 223.94, median: 223.82, stdev: 0.80 Epoch 1: Radius 2: Occulter: 31, radius mean: 224.02, median: 223.96, stdev: 0.88

Epoch 2: Radius 1: Occulter: 31, radius mean: 229.71, median: 229.64, stdev: 0.52 Epoch 2: Radius 2: Occulter: 31, radius mean: 229.57, median: 229.51, stdev: 0.58

Epoch 1: Radius 1: Occulter: 35, radius mean: 225.26, median: 225.41, stdev: 0.78 Epoch 1: Radius 2: Occulter: 35, radius mean: 225.33, median: 225.45, stdev: 0.92

Epoch 2: Radius 1: Occulter: 35, radius mean: 232.89, median: 232.89, stdev: 0.57 Epoch 2: Radius 2: Occulter: 35, radius mean: 232.80, median: 232.82, stdev: 0.68

detoma commented 3 months ago

Level 2 data before 2013 need to be masked more in the outer fov.

Need to check values in epoch file and adjust for 2011 and 2012

detoma commented 3 months ago

The movies and .png do not have the post masked.

This was a bug in comp_l2_mask.pro. Changed: if (keyword_set(no_post)) then begin pmask = comp_post_mask(post_angle, post_width) into: if (not keyword_set(no_post)) then begin pmask = comp_post_mask(post_angle, post_width) This should fix the issue.

The data are already masked in the level 2 files and there should not be extra masking needed when png and mp4 are created.

NCAR / comp-pipeline