search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
29 stars 6 forks source link

Excessive AOflagging of gain calibrator at low elevation #1261

Open paoloserra opened 4 years ago

paoloserra commented 4 years ago

@edeblok and @gigjozsa found that low-elevation scans of the secondary calibrator are heavily flagged (sometimes 100%) when using AOflagger in the flag worker with our default strategy firstpass_Q.rfis.

I confirm that.

I'm finding that this depends on whether the low-elevation scans of the secondary calibrator are flagged on their own or together with the high-elevation scans of the secondary calibrator. I believe that this is because of AOflagger not respecting scan boundaries, possibly coupled with a time-window size larger than a single-scan duration in firstpass_Q.rfis.

The issue of AOflagger not respecting scan boundaries is known and is discussed at https://github.com/caracal-pipeline/caracal/issues/914.

And here's some actual numbers.

I am working on an MS with 6 gain calibrator scans. The approximate elevation of the scans is:

scan 2: 25 deg
scan 4: 35 deg
scan 6: 45 deg
scan 8: 55 deg
scan 11: 65 deg
scan 13: 75 deg

I created MS files that include the first N scans (N=1,...,6) and flagged in the usual way: MW, shadowing, autocorr, RFI with the firstpass_Q.rfis strategy.

Below I list the flagged fraction of the various scans. When you only see 1 scan it means that's all the MS had in it. When you see 6 scans it means that all available scans were included in the MS.

scan 2 flagged: 5.79607e+07 total: 5.55408e+08 (10.4%)
scan 2 flagged: 5.69639e+07 total: 5.55408e+08 (10.3%)
scan 4 flagged: 5.64e+07 total: 5.55408e+08 (10.2%)
scan 2 flagged: 7.60167e+07 total: 5.55408e+08 (13.7%)
scan 4 flagged: 5.79525e+07 total: 5.55408e+08 (10.4%)
scan 6 flagged: 7.57057e+07 total: 5.9508e+08 (12.7%)
scan 2 flagged: 1.7554e+08 total: 5.55408e+08 (31.6%)
scan 4 flagged: 6.56206e+07 total: 5.55408e+08 (11.8%)
scan 6 flagged: 8.09561e+07 total: 5.9508e+08 (13.6%)
scan 8 flagged: 7.95852e+07 total: 5.55408e+08 (14.3%)
scan 2 flagged: 3.25448e+08 total: 5.55408e+08 (58.6%)
scan 4 flagged: 1.08453e+08 total: 5.55408e+08 (19.5%)
scan 6 flagged: 9.3907e+07 total: 5.9508e+08 (15.8%)
scan 8 flagged: 8.81374e+07 total: 5.55408e+08 (15.9%)
scan 11 flagged: 1.14796e+08 total: 5.55408e+08 (20.7%)
scan 2 flagged: 3.65215e+08 total: 5.55408e+08 (65.8%)
scan 4 flagged: 1.42009e+08 total: 5.55408e+08 (25.6%)
scan 6 flagged: 1.03511e+08 total: 5.9508e+08 (17.4%)
scan 8 flagged: 9.078e+07 total: 5.55408e+08 (16.3%)
scan 11 flagged: 1.08707e+08 total: 5.55408e+08 (19.6%)
scan 13 flagged: 9.87305e+07 total: 5.55408e+08 (17.8%)

For those who want to continue using AOflagger it might be worth checking whether a smaller time-window size helps.

Tricolour respects scan boundaries.

edeblok commented 4 years ago

Repeating that test on my data I see similar behaviour. With more testing with rfigui I found that in my data the most effective way to restore the expected flagging behaviour is to increase the TimeSelectionAction parameter in the CombineFlagResults section from 2.1 to 3.5. This parameter is quite sensitive: there is only a range of a few tenths around it where it has the desired behaviour, both for the calibrator and the target. 

                    <children>
                      <action type="FrequencySelectionAction">
                        <threshold>1.7</threshold>
                      </action>
                      <action type="TimeSelectionAction">
                        <threshold>3.5</threshold>
                      </action>
                    </children>

It would be interesting to know how sensitive this parameter is to calibrator scan length and interval. Is just increasing the value enough to fix this for all MeerKAT data forever or does it need tweaking every time an observation with a different scan length is processed? Maybe @gigjozsa can test on his data to see if increasing to this particular value works for him?

paoloserra commented 4 years ago

Thanks @edeblok, we will test that solution too. (We've got 2 min secondary scans. You?)

Also, in the meantime, @molnard89 is trying to find a good Tricolour calibrator flagging strategy for the zoom band.

edeblok commented 4 years ago

Secondary cal scans are 2 mins here as well.

AstroRipples commented 3 years ago

I've got a feeling I'm suffering from this issue as well... Based on @KshitijT's suggestion, I'm running the flag step including

  flag_rfi:
    enable: true
    flagger: tricolour
    tricolour:
      strategy: mk_rfi_flagging_calibrator_fields_firstpass.yaml

When I run the pipeline, up to and including the calibration step, the resulting flag statistics are higher than I would have expected for MeerKAT data:

field J0408-6545 flagged: 1.34067e+09 total: 2.21872e+09 (60.4%)
field J1331+3030 flagged: 1.22959e+09 total: 2.18874e+09 (56.2%)
field J1445+0958 flagged: 2.91635e+09 total: 6.44628e+09 (45.2%)
field J1939-6342 flagged: 2.51833e+09 total: 4.43744e+09 (56.8%)
...
Total Flagged: 8.00495e+09 Total Counts: 1.52912e+10 (52.4%)

Maybe it's just me though - is this typical for MeerKAT?

However, once CARACal gets past the flagging step, it fails to find a good solution for J0408-6545, and is currently failing to find bandpass solutions for J1939-6342 -- I'm getting Insufficient unflagged antennas to proceed with this solve warnings for every channel in my dataset...

The target for this run is in the northern hemisphere, so I get that life is going to be a bit more difficult, but this seems a bit extreme!

KshitijT commented 3 years ago

Maybe it's just me though - is this typical for MeerKAT?

This is typical for MeerKAT.

Insufficient unflagged antennas to proceed with this solve warnings for every channel in my dataset...

Wow, that's concerning; do you know the antenna flagging stats ?

AstroRipples commented 3 years ago

This is typical for MeerKAT.

Ah okay, thanks for clarifying!

Wow, that's concerning; do you know the antenna flagging stats ?

Yeah - they seem kinda high, but I would not have thought they were prohibitively so:

# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m000 flagged: 2.94808e+08 total: 5.0135e+08 (58.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m001 flagged: 2.96309e+08 total: 5.0135e+08 (59.1%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m002 flagged: 2.89156e+08 total: 5.0135e+08 (57.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m003 flagged: 2.92939e+08 total: 5.0135e+08 (58.4%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m004 flagged: 2.89913e+08 total: 5.0135e+08 (57.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m005 flagged: 2.91078e+08 total: 5.0135e+08 (58.1%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m006 flagged: 2.89975e+08 total: 5.0135e+08 (57.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m007 flagged: 2.93539e+08 total: 5.0135e+08 (58.5%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m008 flagged: 2.9949e+08 total: 5.0135e+08 (59.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m009 flagged: 2.91645e+08 total: 5.0135e+08 (58.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m010 flagged: 2.85189e+08 total: 5.0135e+08 (56.9%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m011 flagged: 2.97106e+08 total: 5.0135e+08 (59.3%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m012 flagged: 2.89664e+08 total: 5.0135e+08 (57.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m013 flagged: 2.93337e+08 total: 5.0135e+08 (58.5%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m014 flagged: 2.88815e+08 total: 5.0135e+08 (57.6%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m015 flagged: 2.89673e+08 total: 5.0135e+08 (57.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m016 flagged: 2.88414e+08 total: 5.0135e+08 (57.5%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m017 flagged: 3.1236e+08 total: 5.0135e+08 (62.3%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m018 flagged: 2.97316e+08 total: 5.0135e+08 (59.3%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m019 flagged: 2.91573e+08 total: 5.0135e+08 (58.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m021 flagged: 2.92602e+08 total: 5.0135e+08 (58.4%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m022 flagged: 2.7934e+08 total: 5.0135e+08 (55.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m023 flagged: 2.86935e+08 total: 5.0135e+08 (57.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m024 flagged: 2.76713e+08 total: 5.0135e+08 (55.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m026 flagged: 2.94821e+08 total: 5.0135e+08 (58.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m027 flagged: 2.94586e+08 total: 5.0135e+08 (58.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m028 flagged: 2.91114e+08 total: 5.0135e+08 (58.1%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m029 flagged: 2.91613e+08 total: 5.0135e+08 (58.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m030 flagged: 2.81637e+08 total: 5.0135e+08 (56.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m031 flagged: 2.84402e+08 total: 5.0135e+08 (56.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m032 flagged: 2.66438e+08 total: 5.0135e+08 (53.1%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m033 flagged: 2.18914e+08 total: 5.0135e+08 (43.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m035 flagged: 2.85223e+08 total: 5.0135e+08 (56.9%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m036 flagged: 2.80485e+08 total: 5.0135e+08 (55.9%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m037 flagged: 2.85705e+08 total: 5.0135e+08 (57%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m038 flagged: 3.14427e+08 total: 5.0135e+08 (62.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m039 flagged: 2.85204e+08 total: 5.0135e+08 (56.9%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m040 flagged: 2.9433e+08 total: 5.0135e+08 (58.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m041 flagged: 2.89367e+08 total: 5.0135e+08 (57.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m042 flagged: 2.90033e+08 total: 5.0135e+08 (57.9%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m043 flagged: 2.60244e+08 total: 5.0135e+08 (51.9%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m044 flagged: 2.21227e+08 total: 5.0135e+08 (44.1%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m045 flagged: 2.03958e+08 total: 5.0135e+08 (40.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m046 flagged: 1.90804e+08 total: 5.0135e+08 (38.1%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m047 flagged: 2.62861e+08 total: 5.0135e+08 (52.4%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m048 flagged: 1.81403e+08 total: 5.0135e+08 (36.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m049 flagged: 1.89606e+08 total: 5.0135e+08 (37.8%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m050 flagged: 2.12571e+08 total: 5.0135e+08 (42.4%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m051 flagged: 2.27739e+08 total: 5.0135e+08 (45.4%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m052 flagged: 2.23935e+08 total: 5.0135e+08 (44.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m053 flagged: 2.268e+08 total: 5.0135e+08 (45.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m054 flagged: 2.36496e+08 total: 5.0135e+08 (47.2%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m055 flagged: 2.16874e+08 total: 5.0135e+08 (43.3%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m056 flagged: 2.1189e+08 total: 5.0135e+08 (42.3%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m057 flagged: 1.95942e+08 total: 5.0135e+08 (39.1%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m058 flagged: 2.23811e+08 total: 5.0135e+08 (44.6%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m059 flagged: 1.86954e+08 total: 5.0135e+08 (37.3%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m060 flagged: 1.89015e+08 total: 5.0135e+08 (37.7%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m061 flagged: 2.03108e+08 total: 5.0135e+08 (40.5%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m062 flagged: 2.02572e+08 total: 5.0135e+08 (40.4%)
# 2021-05-18 21:08:13   INFO    Summary::getResult   antenna m063 flagged: 1.95894e+08 total: 5.0135e+08 (39.1%)
bennahugo commented 3 years ago

Are you sure it is all channels. Some bandpass channels will be completely gone. Could you make waterfall plots? Typically 50 or so percent is not actually high enough. I do a few rounds of flagging and calibration which pushes the percentages to about 70% on calibrators, but it is really clean so I get good solutions for the most part.

On Thu, 20 May 2021, 13:04 AstroRipples, @.***> wrote:

This is typical for MeerKAT.

Ah okay, thanks for clarifying!

Wow, that's concerning; do you know the antenna flagging stats ?

Yeah - they seem kinda high, but I would not have thought they were prohibitively so:

2021-05-18 21:08:13 INFO Summary::getResult antenna m000 flagged: 2.94808e+08 total: 5.0135e+08 (58.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m001 flagged: 2.96309e+08 total: 5.0135e+08 (59.1%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m002 flagged: 2.89156e+08 total: 5.0135e+08 (57.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m003 flagged: 2.92939e+08 total: 5.0135e+08 (58.4%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m004 flagged: 2.89913e+08 total: 5.0135e+08 (57.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m005 flagged: 2.91078e+08 total: 5.0135e+08 (58.1%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m006 flagged: 2.89975e+08 total: 5.0135e+08 (57.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m007 flagged: 2.93539e+08 total: 5.0135e+08 (58.5%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m008 flagged: 2.9949e+08 total: 5.0135e+08 (59.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m009 flagged: 2.91645e+08 total: 5.0135e+08 (58.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m010 flagged: 2.85189e+08 total: 5.0135e+08 (56.9%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m011 flagged: 2.97106e+08 total: 5.0135e+08 (59.3%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m012 flagged: 2.89664e+08 total: 5.0135e+08 (57.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m013 flagged: 2.93337e+08 total: 5.0135e+08 (58.5%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m014 flagged: 2.88815e+08 total: 5.0135e+08 (57.6%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m015 flagged: 2.89673e+08 total: 5.0135e+08 (57.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m016 flagged: 2.88414e+08 total: 5.0135e+08 (57.5%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m017 flagged: 3.1236e+08 total: 5.0135e+08 (62.3%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m018 flagged: 2.97316e+08 total: 5.0135e+08 (59.3%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m019 flagged: 2.91573e+08 total: 5.0135e+08 (58.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m021 flagged: 2.92602e+08 total: 5.0135e+08 (58.4%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m022 flagged: 2.7934e+08 total: 5.0135e+08 (55.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m023 flagged: 2.86935e+08 total: 5.0135e+08 (57.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m024 flagged: 2.76713e+08 total: 5.0135e+08 (55.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m026 flagged: 2.94821e+08 total: 5.0135e+08 (58.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m027 flagged: 2.94586e+08 total: 5.0135e+08 (58.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m028 flagged: 2.91114e+08 total: 5.0135e+08 (58.1%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m029 flagged: 2.91613e+08 total: 5.0135e+08 (58.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m030 flagged: 2.81637e+08 total: 5.0135e+08 (56.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m031 flagged: 2.84402e+08 total: 5.0135e+08 (56.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m032 flagged: 2.66438e+08 total: 5.0135e+08 (53.1%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m033 flagged: 2.18914e+08 total: 5.0135e+08 (43.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m035 flagged: 2.85223e+08 total: 5.0135e+08 (56.9%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m036 flagged: 2.80485e+08 total: 5.0135e+08 (55.9%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m037 flagged: 2.85705e+08 total: 5.0135e+08 (57%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m038 flagged: 3.14427e+08 total: 5.0135e+08 (62.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m039 flagged: 2.85204e+08 total: 5.0135e+08 (56.9%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m040 flagged: 2.9433e+08 total: 5.0135e+08 (58.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m041 flagged: 2.89367e+08 total: 5.0135e+08 (57.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m042 flagged: 2.90033e+08 total: 5.0135e+08 (57.9%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m043 flagged: 2.60244e+08 total: 5.0135e+08 (51.9%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m044 flagged: 2.21227e+08 total: 5.0135e+08 (44.1%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m045 flagged: 2.03958e+08 total: 5.0135e+08 (40.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m046 flagged: 1.90804e+08 total: 5.0135e+08 (38.1%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m047 flagged: 2.62861e+08 total: 5.0135e+08 (52.4%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m048 flagged: 1.81403e+08 total: 5.0135e+08 (36.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m049 flagged: 1.89606e+08 total: 5.0135e+08 (37.8%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m050 flagged: 2.12571e+08 total: 5.0135e+08 (42.4%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m051 flagged: 2.27739e+08 total: 5.0135e+08 (45.4%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m052 flagged: 2.23935e+08 total: 5.0135e+08 (44.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m053 flagged: 2.268e+08 total: 5.0135e+08 (45.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m054 flagged: 2.36496e+08 total: 5.0135e+08 (47.2%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m055 flagged: 2.16874e+08 total: 5.0135e+08 (43.3%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m056 flagged: 2.1189e+08 total: 5.0135e+08 (42.3%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m057 flagged: 1.95942e+08 total: 5.0135e+08 (39.1%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m058 flagged: 2.23811e+08 total: 5.0135e+08 (44.6%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m059 flagged: 1.86954e+08 total: 5.0135e+08 (37.3%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m060 flagged: 1.89015e+08 total: 5.0135e+08 (37.7%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m061 flagged: 2.03108e+08 total: 5.0135e+08 (40.5%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m062 flagged: 2.02572e+08 total: 5.0135e+08 (40.4%)

2021-05-18 21:08:13 INFO Summary::getResult antenna m063 flagged: 1.95894e+08 total: 5.0135e+08 (39.1%)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1261#issuecomment-844985027, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6WFQ3UJUCAXZ3L7MFTTOTUDFANCNFSM4SWAJWGQ .

KshitijT commented 3 years ago

Yeah - they seem kinda high, but I would not have thought they were prohibitively so:

Yeah, I thought maybe somehow all your longer baselines are totally flagged or something and that with the baseline cutoff you don't get any data for calibration - but that's clearly not the case.

AstroRipples commented 3 years ago

Are you sure it is all channels. Some bandpass channels will be completely gone. Could you make waterfall plots?

I've got a feeling that perhaps one scan is particularly highly-flagged. Of our two scans on 1934-638, one was only shortly after it had risen, so I wouldn't be surprised if it was failing -- that seems to be what happened for J0408-6545. In and of itself, that strikes me as a real issue, if CARACal can't handle calibrating sources below ~20deg elevation.

As for waterfall plots, sure, I'll try! Are there any tools for this included in the CARACal framework? I've dug around the online documentation (on the readthedocs page) and I can't see a lot of descriptive documentation, but perhaps I'm overlooking something?

Typically 50 or so percent is not actually high enough. I do a few rounds of flagging and calibration which pushes the percentages to about 70% on calibrators, but it is really clean so I get good solutions for the most part.

A flagging fraction of ~70% seems pretty bananas to me, and I deal with ATCA data regularly! 😅

bennahugo commented 3 years ago

I suggest using Ian Heywood's Surfvis. I don't think it is elevation. MeerKAT scheduling will not schedule your observation if the calibrator partially sets during observation. I thnk you could maybe just have been really unlucky with getting GPS satellites close to the beam. This happens.

On Thu, May 20, 2021 at 1:17 PM AstroRipples @.***> wrote:

Are you sure it is all channels. Some bandpass channels will be completely gone. Could you make waterfall plots?

I've got a feeling that perhaps one scan is particularly highly-flagged. Of our two scans on 1934-638, one was only shortly after it had risen, so I wouldn't be surprised if it was failing -- that seems to be what happened for J0408-6545. In and of itself, that strikes me as a real issue, if CARACal can't handle calibrating sources below ~20deg elevation.

As for waterfall plots, sure, I'll try! Are there any tools for this included in the CARACal framework? I've dug around the online documentation (on the readthedocs page) and I can't see a lot of descriptive documentation, but perhaps I'm overlooking something?

Typically 50 or so percent is not actually high enough. I do a few rounds of flagging and calibration which pushes the percentages to about 70% on calibrators, but it is really clean so I get good solutions for the most part.

A flagging fraction of ~70% seems pretty bananas to me, and I deal with ATCA data regularly! 😅

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1261#issuecomment-844995148, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6XZSYH6GTTESOAFCDDTOTVUTANCNFSM4SWAJWGQ .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

AstroRipples commented 3 years ago

Hmm, okay, I haven't had the opportunity to look into whether I can run that on our cluster or not, or whether it's included with CARACal or anything (this is literally the first time I've heard about this!) but... based on the plots for my observing run available through the SARAO Web Interface, the run looks extremely clean. Aside from the known RFI bands, we only see the occasional narrow-band RFI blips - certainly nothing I can see that would cause calibration on J0408-6545 to fail:

Flags_s_v_chan

KshitijT commented 3 years ago

@AstroRipples , could you please share the logfile for crosscalibration?

AstroRipples commented 3 years ago

Hey @KshitijT! Sure thing, it's attached - thanks for all the assistance!

log-crosscal-gain_cal-1gc1-0-0-primary-20210518-213800.txt

AstroRipples commented 3 years ago

It's been a few days since I last updated y'all (see post ), and I've been able to study my logs, tweak a few settings, and re-run my pipeline. I think I'm largely happy with how the first pass of calibration is looking for 1934-638 and my gain calibrator (SOURCE_NAME). Plots of amp. vs. channel (averaging 8chan in freq, 10s in time) are below, and these look pretty reasonable.

1934-638:

Screenshot 2021-05-25 at 11 44 32

Gain calibrator (J1445+0958):

Screenshot 2021-05-25 at 11 36 50

So I'm counting this as a success 🎉 there's some residual bad data that could be excised, but when I tried to run a second pass of flagging, CARACal threw an error, so for now I'm ignoring it. @bennahugo : if you have an example yml file that contains the second pass of flagging and calibration that you mentioned, I'd appreciate a look-in.

Now looking at my target field (J1454+2233), when I run the pipeline with the following flag inputs:

flag__2:
  enable: true
  field: target
  label_in: corr
  flag_autocorr:
    enable: true
  flag_shadow:
    enable: true
  flag_spw:
    chans: '*:856~880MHz , *:1658~1800MHz, *:1419.8~1421.3MHz'
    enable: true
  flag_rfi:
    enable: true
    flagger: tricolour
    tricolour:
      strategy: mk_rfi_flagging_target_fields_firstpass.yaml

The data does not look so clean:

Screenshot 2021-05-25 at 12 46 18

I'm wondering if there's a better MeerKAT flagging file that can be used on the target field. I'm hesitant to go mucking about with it myself, as I can't find any documentation about the contents of the tricolour config. file on the CARACal wiki, so advice would be appreciated.

bennahugo commented 3 years ago

Hi

The parameters for tricolour's flagging file is discussed in the code: https://github.com/ska-sa/tricolour/blob/master/tricolour/flagging.py

Looks good. The flagging strategies I discussed are in my pipeline: https://github.com/ska-sa/vermeerkat/tree/master/vermeerkat/data/input

On Tue, May 25, 2021 at 12:48 PM AstroRipples @.***> wrote:

It's been a few days since I last updated y'all (see post ), and I've been able to study my logs, tweak a few settings, and re-run my pipeline. I think I'm largely happy with how the first pass of calibration is looking for 1934-638 and my gain calibrator (SOURCE_NAME). Plots of amp. vs. channel (averaging 8chan in freq, 10s in time) are below, and these look pretty reasonable.

1934-638:

[image: Screenshot 2021-05-25 at 11 44 32] https://user-images.githubusercontent.com/42791701/119483709-8a97b500-bd55-11eb-9123-e02764aaebe7.png

Gain calibrator (J1445+0958):

[image: Screenshot 2021-05-25 at 11 36 50] https://user-images.githubusercontent.com/42791701/119483731-8f5c6900-bd55-11eb-81cb-f7429bb01abd.png

So I'm counting this as a success 🎉 there's some residual bad data that could be excised, but when I tried to run a second pass of flagging, CARACal threw an error, so for now I'm ignoring it. @bennahugo https://github.com/bennahugo : if you have an example yml file that contains the second pass of flagging and calibration that you mentioned, I'd appreciate a look-in.

Now looking at my target field (J1454+2233), when I run the pipeline with the following flag inputs:

flag__2:

enable: true

field: target

label_in: corr

flag_autocorr:

enable: true

flag_shadow:

enable: true

flag_spw:

chans: '*:856~880MHz , *:1658~1800MHz, *:1419.8~1421.3MHz'

enable: true

flag_rfi:

enable: true

flagger: tricolour

tricolour:

  strategy: mk_rfi_flagging_target_fields_firstpass.yaml

The data does not look so clean:

[image: Screenshot 2021-05-25 at 12 46 18] https://user-images.githubusercontent.com/42791701/119485337-3a215700-bd57-11eb-90b5-30de255df5ba.png

I'm wondering if there's a better MeerKAT flagging file that can be used on the target field. I'm hesitant to go mucking about with it myself, as I can't find any documentation about the contents of the tricolour config. file on the CARACal wiki, so advice would be appreciated.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1261#issuecomment-847763480, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6WDP64EFSKUIECAP5LTPN6ABANCNFSM4SWAJWGQ .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

o-smirnov commented 3 years ago

I've added some new tricolour strategies in this branch: https://github.com/caracal-pipeline/caracal/tree/add_polcal/caracal/data/meerkat_files

You can try stalin.yaml if you feel your target is underflagged. Though I prefer to use Gorbachev, then image and let madmax do the flagging in selfcal.

bennahugo commented 3 years ago

Yes I typically unflag, repredict from a image model and then reflag the residual data. Note there is a -smc option to subtraction a model data column to form residuals to flag on

AstroRipples commented 3 years ago

Thanks for the info @o-smirnov, @bennahugo, much appreciated!

Since yesterday I've tried running stalin.yaml and gorbachev.yaml on my target data... Stalin seems give me marginally better results than the default config file (mk_rfi_flagging_target_fields_firstpass.yaml):

Screenshot 2021-05-26 at 08 54 01 Screenshot 2021-05-26 at 08 54 01

You can see the outlier amplitudes have dropped a little bit, but there's still a lot of what clearly looks like bad data that's completely untouched...

Conversely, Gorbachev does a terrible job:

Screenshot 2021-05-26 at 12 38 20

So it looks like I'm going to need to start digging into the guts of tricolour config. files... Anyone have any suggestions where to start? Is there a user manual for Tricolour anywhere?

Edit: the key point is that I'm trying to excise as much of the RFI as possible before proceeding to the self-cal, as I'm going to do all imaging & self-cal outside of CARACal using DDFacet and killMS.

bennahugo commented 3 years ago

Hi

As I mentioned do one round of masked imaging to build a model of the brightest AGN then reset flags and do -smc MODEL_DATA. This will flag better than just trying to remove outliers because the amplitude of a target is much more complicated than a calibrator field.

You can adjust the sigma downwards for the second pass without running the risk of flagging strong fringes.

On Wed, 26 May 2021, 12:47 AstroRipples, @.***> wrote:

Thanks for the info @o-smirnov https://github.com/o-smirnov, @bennahugo https://github.com/bennahugo, much appreciated!

Since yesterday I've tried running stalin.yaml and gorbachev.yaml on my target data... Stalin seems give me marginally better results than the default config file (mk_rfi_flagging_target_fields_firstpass.yaml):

[image: Screenshot 2021-05-26 at 08 54 01] https://user-images.githubusercontent.com/42791701/119646903-f6deeb00-be1f-11eb-9a87-3c474820cd84.png

[image: Screenshot 2021-05-26 at 08 54 01] https://user-images.githubusercontent.com/42791701/119646912-f9d9db80-be1f-11eb-804a-c85ed90a01d5.png

You can see the outlier amplitudes have dropped a little bit, but there's still a lot of what clearly looks like bad data that's completely untouched...

Conversely, Gorbachev does a terrible job:

[image: Screenshot 2021-05-26 at 12 38 20] https://user-images.githubusercontent.com/42791701/119647043-19710400-be20-11eb-96b5-c84d0673937b.png

So it looks like I'm going to need to start digging into the guts of tricolour config. files... Anyone have any suggestions where to start? Is there a user manual for Tricolour anywhere?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1261#issuecomment-848667176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6REUSJBWY44VU2PIP3TPTGTVANCNFSM4SWAJWGQ .

AstroRipples commented 3 years ago

Hi As I mentioned do one round of masked imaging to build a model of the brightest AGN then reset flags and do -smc MODEL_DATA. This will flag better than just trying to remove outliers because the amplitude of a target is much more complicated than a calibrator field. You can adjust the sigma downwards for the second pass without running the risk of flagging strong fringes.

Ahhh, apologies, I misunderstood your meaning - I had thought your comment about flagging on the residual data was in reference to the calibrator fields! Of course the target field amplitude is going to be much more complicated than a strong point source at the phase centre, but then again I'd have expected the auto-flagging routine to have clipped the obviously large amplitudes on those intermediate channels...

bennahugo commented 3 years ago

The Sigma's are explicitly large because some fields have strong sources off centre. Flagging them with low Sigma's will clip real fringes, so there is a tradeoff. Therefore flag residuals after one round of imaging - the data is good enough for a shallow MFS - the SDP flags are orders of magnitude worse than this and they still make images.

On Thu, 27 May 2021, 09:39 AstroRipples, @.***> wrote:

Hi As I mentioned do one round of masked imaging to build a model of the brightest AGN then reset flags and do -smc MODEL_DATA. This will flag better than just trying to remove outliers because the amplitude of a target is much more complicated than a calibrator field. You can adjust the sigma downwards for the second pass without running the risk of flagging strong fringes.

Ahhh, apologies, I misunderstood your meaning - I had thought your comment about flagging on the residual data was in reference to the calibrator fields! Of course the target field amplitude is going to be much more complicated than a strong point source at the phase centre, but then again I'd have expected the auto-flagging routine to have clipped the obviously large amplitudes on those intermediate channels...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1261#issuecomment-849411346, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6SYALJ52NEGV3LF5RLTPXZM3ANCNFSM4SWAJWGQ .

AstroRipples commented 3 years ago

Okay, right, I'm with you now. Can you point me to an example CARACal config. where you've implemented this? I'd like to take a look at a working example so I can be sure I'm on the right track.

bennahugo commented 3 years ago

I don't know if this is implemented in caracal. This is just what I typically do in my stop start approach in VermeerKAT. It is easy enough to call flagmanager and tricolour outside though after the first model image visibilities are written (ie. before your first round of selfcalibration)?

On Thu, May 27, 2021 at 11:58 AM AstroRipples @.***> wrote:

Okay, right, I'm with you now. Can you point me to an example CARACal config. where you've implemented this? I'd like to take a look at a working example so I can be sure I'm on the right track.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1261#issuecomment-849502806, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6U2WHLAIEKP3SGWNDLTPYJU5ANCNFSM4SWAJWGQ .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

AstroRipples commented 3 years ago

It's been a while so I thought I'd update this thread (even though I kinda hijacked it... sorry...). @bennahugo : thanks for the suggestions and assistance in getting tricolour installed - I've now flagged my MeerKAT data to a suitable level and I'm proceeding with self-calibration 🎉