A number of cases have come up in operations where the inf_EB solint succeeds because the SNR goes up, but because of flagging the RMS also goes up, in some cases by a considerable amount, due to flagging in the inf_EB solutions. In many of these cases, the SNR of all of the solutions, not just the ones that were flagged, tends to be quite low, which leads me to believe that in these cases the data are just overall fairly low SNR (at least as far as selfcal is concerned) rather than the flagged data truly being bad.
To remedy this, this PR implements a new fallback mode wherein if the inf_EB solint passes with combinespw but RMS increases by more than 2%, it will redo inf_EB and check for/pass through failed antennas if the solutions are all weak (mean SNR < 5). I implemented this as a repeat of the solint rather than simply looking for overall low SNR inf_EB solutions and passing through the flagged ones at the outset as it seemed to me that it would be better to let inf_EB pass with the flagged solutions if it doesn't dramatically harm the RMS.
A few other points that might be worth some thought:
I've hard-coded the trigger criteria at a 2% increase in RMS and the mean of all (good) gain solutions to be < 5, but it might be worth considering how we tune those values. Could also make sense to make them free parameters?
In that vein, what do we think of the second trigger criterion, of mean(SNR) < 5? I was also thinking about incorporating the std somehow... (e.g. mean(SNR) - 2*std(SNR)), but was worried about not triggering enough (or triggering too much) in cases where the SNR got relatively large.
I've tested this on two datasets that show this problem and both look good. I want to do testing on a broader range of data to see how it impacts the overall inf_EB solint before we merge.
A number of cases have come up in operations where the inf_EB solint succeeds because the SNR goes up, but because of flagging the RMS also goes up, in some cases by a considerable amount, due to flagging in the inf_EB solutions. In many of these cases, the SNR of all of the solutions, not just the ones that were flagged, tends to be quite low, which leads me to believe that in these cases the data are just overall fairly low SNR (at least as far as selfcal is concerned) rather than the flagged data truly being bad.
To remedy this, this PR implements a new fallback mode wherein if the inf_EB solint passes with combinespw but RMS increases by more than 2%, it will redo inf_EB and check for/pass through failed antennas if the solutions are all weak (mean SNR < 5). I implemented this as a repeat of the solint rather than simply looking for overall low SNR inf_EB solutions and passing through the flagged ones at the outset as it seemed to me that it would be better to let inf_EB pass with the flagged solutions if it doesn't dramatically harm the RMS.
A few other points that might be worth some thought:
I've tested this on two datasets that show this problem and both look good. I want to do testing on a broader range of data to see how it impacts the overall inf_EB solint before we merge.