Quack should be improved

[varenius]

Currently 5 minutes is quacked for the bright calibrators. This is often insufficient, as slewing from far away may take longer (up to 23 minutes from experience in scheduling, but should be double checked). In multiple data sets I have processed, at least 10 minutes should be quacked for at least one of the bright cals. I propose some heuristic is introduced to figure out how long quack is needed on the bright cals. It should be fairly easy to e.g. define a running median filter with 30 seconds in time or so to find the edge when the scan begins. Given that these sources are all strong, it should be able to find the correct quack scan.

Perhaps such a routine could also be used to identify Lovell dropout scans?

[jmoldon]

I completely agree, but not time yet to implement it. There are two things to do that are different:

• Identification of drop scans. This can be done using visstat, which is relatively fast. It can give you amplitude average, std, etc. Then a robust decision criteria is needed: which spw to use, baselines, if it should match most of baselines to one antenna, etc. Checking all baselines x spw can take some time. Finally this can be applied to Lo dropout and also for first phasecal scan after long slew (first scan of observation, or after visiting the flux scale cal or bandpass).

• A different think is defining a function to find end of slewing. That is a bit slower because you need to read many more visibilities, and each baseline can be different. A good idea is probably select a few random scans of the phase calibrator and refine the slewing time until it converges. In any case the script needs some work and optimization.

Let me know if you want to start writing either of those two. I know it has to be done eventually. I use some similar things to identify the best reference antenna, and also for the monitoring function. So that could be a start.

[varenius]

Right. The two things are similar, but not identical so I'm not certain if there should be one common solution or two different ones. I may be able to contribute here, but first need to think a bit and perhaps discuss some more before starting the work.

[daniellefenech]

I think what you're after is already what we coded into SERPent as the zero-level dropout and Lovell-dropout sections, essentially testing against running scan-averages and in-scan averages (actually medians because it works better). I've got a working version of SERPent for running on CASA data now, so I'll get this over to you and actually update the GitHub page as well. Even if you didn't want to run the full-flagging at this stage of the calibration, you can isolate for just those sections to run.

[jmoldon]

Great Danielle, thanks. Yes, please, share the version you have to see the structure. It should be easy to plug it in the pipeline.

The best approach would be to call all the small subtasks (like finding Lo dropouts) in the a-priori flagging step flag_1_apriori, and have a separate step for the main flagging flag_0_serpent. That way we can choose to conduct the main flagging with the current aoflagger or the new SERPent, and compare the results.

[varenius]

Great Danielle! Yes, if the SERPent code already contains solutions to these problems then I think we should definitely use those rather than spending time on reinventing things again.

As Javi suggests, I also think that separating the dropouts and Lovell-dropouts from the actual Flagging part (SumThreshold based algorithm) is preferred. I would be happy to help here, if Danielle can share the SERPent code (e.g. on that Github account)

Comparing SERPent with AOFlagger would be interesting. I am still somewhat confused regarding the differences and merits with respective softwares, so I would be very happy to discuss this at some point. Given what I currently know, I suspect that the "optimal" solution would be to use the wisdom accumulated when building SERPent to design "strategies" for AOflagger. I think this because my current limited knowledge suggests that AOflagger's native c++ implementation should be faster than any python-version, no matter how efficiently numpy etc. is used. But, AOFlagger needs tailored strategies to be useful, and my experience with SERPent in general shows that it does capture the RFI very well. But since my understanding is that SERPent is essentially a re-implementation and/or wrapper around the same algorithm as in AOFlagger, it seems to me that the pure c++ code should be faster. But I may be mistaken, perhaps there are additional steps in the SERPent implementation which cannot be carried over to strategies and/or separate a-priori functions such as the dropouts. Again - would be nice to have a chat about this to enlighten me!

[daniellefenech]

The dropouts and Lovell-dropouts are already separate from the SumThreshold algorithm and can be run individually.

I'd love to see some like-for-like comparisons of SERPent and AOFlagger. You're right the SumThreshold algorithm is the same in SERPent and actually there's still an extra bit of code that the AOFlagger has that I need to add to SERPent. Obviously SERPent makes use of as much of the built-in vectorised functions in numpy as possible i.e. those that wrap C-functions, to make it is efficient as possible. I'd expect the pure C++ version (AOFlagger) to be quicker than the python version (SERPent) but I've never tested it properly and especially when running both through ms datasets I'd be curious to see what the relative performance actually is. Generally, it isn't aways the case that C/C++ code will out perform python, but as a rule it mostly does. The major advantage to SERPent of course, is that no compiling is necessary i.e. armed with casa (therefore python and numpy) it runs out of the box. That and it enabled, as you said, the addition of things like the dropouts.

[daniellefenech]

What's the current ethos in terms of saving flagging information and writing to log files? Do you intend to store a copy of the previous flag info at each stage and are you writing separate logfiles for things like flagging or will there be a single logfile for each full calibration run? I'll make sure SERPent is complient.

[varenius]

Just regarding compiling: I agree that AOFlagger has been a pain to install in the past. But since the Ubuntu 17.04 release this year, AOflagger is in the repo. So it is possible to get it all, with rfigui, just by running "sudo apt get install aoflagger". So given that a reasonably current Linux distro is available, AOflagger is no longer that annoying to install.

[jmoldon]

Currently the pipeline has its own log (using the standard python logging) that keeps track of what has been run. This includes what aoflagger strategy was used for each source. We can do the same for SERPENT. It also produces a file with the list of flagging steps that have been conducted. Of course, if you change the flags externally or manually, the pipeline will not know what happened. The pipeline also saves the casa log so the individual CASA commands are stored.

So at the moment all information is just informative, but there is not a flag table/steps manager and it is the user responsibility to manage that.

[varenius]

I think Danielle raise an important point regarding how we expect the pipeline to behave it run partially or more than once. Perhaps this should be discussed on thursday, but here's a very quick summary of my ideas on this:

I do not believe in unsupervised pipelines. All pipelines break at some point. They can however be immensely helpful if supervised properly. To me, pipelines are a set of tools that an expert user can run and that will save this user a lot of time and effort. Any pipeline run at some point need a human intervention, usually to assess some step and re-run this step. Hence, being able to re-run steps is, I think, a feature which should be very highly ranked. This includes re-setting flags before a particular step. (The ALMA reduction scripts have had intentions to do this for ages, but last time I checked it actually did not do this properly.) CASA has the "flagmanager" task which can switch between different flag table versions, as long as one has the identifier for a particular table. One can also use the "unflag-mode" of flagdata to reset all flags. So one way to enable flagtable-resets is for the pipeline to keep some sort of dictionary as to what flag table ID corresponds to what step and what run (since unfortunately it's not possible to name flagtables in CASA yet, they are just called incremental numbers). Then, one could in principle restore any flag-table state using the flag manager.

How does the CASA implementation of SERPent write flags? Directly to the flag table? Through some CASA task?

[daniellefenech]

Currently it writes flags directly to the ms file i.e. changes them in situ. I don't see any reason that it can't effectively use/re-create the flag manager step to store a copy of the previous versions which can then be used as for any other situation to restore flags.

[daniellefenech]

That's good news about AOFlagger getting into the repo for Ubuntu, that makes life much easier. I guess a couple of comments on the situation then. Testing relative efficiency and performance between SERPent and AOFlagger is going to be very useful and it may be that SERPent can be used to guide AOFlagger if that's much more efficient. I don't know if Javier has mentioned but the current plan is to combine SERPent with the flagging software that Bob Watson wrote (Bobsflagger) which has a nice optional gui interface and some base-lining options that are very useful. Initially this will be for the current AIPS version of the pipeline but we'll be writing a CASA version as well. I think that will be a very useful tool for developing flagging strategies generally, especially for standard practices, but also as a tool for the user if they need to make any tweaks/changes from a purely pipeline output. I'll be up in Manchester beginning of next week to get that moving along with Bob and hopefully input from everyone else as well.

[varenius]

This issue thread is getting a bit off topic... but: I would very much like to be present for flagger discussions, that would be interesting. I have heard Bobsflagger mentioned but never seen or tested it. Would be very interesting to see what it can do with respect to other things.

AOFlagger for LOFAR has demonstrated to me that flagging CAN be solved as a black box (while the rest of calibration and imaging rarely can be solve in that way). I believe that it should be possible to tailor AOFlagger strategies for eMERLIN data, just as done for LOFAR data. LOFAR has terrible RFI but everything is very nicely removed without any need for manual intervention (no need to set any parameter values at all for any of my dozen datasets there on different sources). Personally, again with limited knowledge, I hence believe that investment in proper design of AOFlagger strategies for eMERLIN should be a wise investment. This also means leaving all code maintenance and future portability in hands of other actors, such as ASTRON, which (I think) is a good thing here.

Personally, I think the RFI inspection GUIs are great tools. AOFlagger provides this as well, as "rfigui", comes with it when installing on e.g. ubuntu. Not sure that investment in GUI options to actually do the flagging is a wise investment, but they are definitely very useful for inspection. Of course, my view on flagging is strongly biased towards my experience, so I would very much like to talk to others and learn more options.

I also believe that with eMERLIN, it is important not only to try to flag all problems, but also to feed back the problems to the staff and engineers. For example, it is possible to modify the correlator code to only record when telescopes are on source. This is known, but the information is not used. Fixing this would essentially remove the need for "dropout checks" and initial quacking. While it is unfortunately not as simple as just chosing either one or the other, since different people and time allocations are involved, we should not forget to feed back the error identified (such as dropouts) back the source. Again, and again :).

[daniellefenech]

Sorry, yes that's my fault things have started swerving away from the original topic. Though we should definitely keep this conversation going (perhaps not here though). Unless something has changed drastically, my experience with e-MERLIN data is that C-band should be fine for getting black-box flagging going but that L-band is far from being in the same boat for a few reasons. The RFI environment for e-MERLIN seems to be very complicated with many sources that are unclear and that appear at times with no consistent time or frequency pattern between datasets and as far as I'm aware is much worse as a result than for LOFAR. Especially when higher sensitivities are really important, I've ended up needing multiple iterations with different strategies to deal with bright higher level RFI (broad and narrow) and then lower-level RFI that only becomes apparent as you work down. Again this hasn't so far proved consistent in terms of dealing with one dataset to the next, even for the same source at the same time of day. I'd love for this to be a problem that gets solved though! Not to sound too sceptical, good luck with the feedback, it should definitely be done, that's the only way it'll ever change but to give an example, before we even started writing the dropouts code about 5 years ago, we tried to get them to make use of that information and have been since....

[jmoldon]

Regarding quack. For data before Jan 25, 2019 an a-priori approximation is used. It can be manually set by the user. For data after Jan 25, 2019 either the observatory slewing database can be used, or the a-priori approximation.