phase jitter calibration requires to use all channels in spw

[jmoldon]

In all gaincal calls now I use the 90% central channels of each spw, for example: spw='*:13~115'.

There is a step in which I calibrate the phase jitter by solving on integration times but combining all spw. When I use combine='spw' and a selection of channels spw='*:13~115' like in other tasks, the solver crashes because there is a mismatch of frequencies. I changed the jitter calibration to use all channels.

Caught exception: Mismatch between Solution frequencies and existing CalTable frequencies for 
spw=0
Related to a 
Current CalTable nchan= 1
Current CalTable freq = [1510649500]
Current Solution nchan= 1
Current Solution freq = [1457316166.67]
Diff = [-53333333.3333]

It looks like a CASA bug, but I don't have time now to check it properly and report it.

[jmoldon]

I'm tracking the origin of the problem. I think gaincal is not the problem, but the inconsistency between frequencies appears when there are channels missing that are not there after averaging. The error seems to happen when the data are splitted using keepflags=False. In that case some frequency channels are completely flagged and gaincal gets confused when combine='spw' is used, in particular when selecting a range of channels.

I don't know exhaustively what combination of combine='spw', spw range selection, and flagging causes the problem, but it seems to work when keepflags=True, so all channels are present in the averaged data set.

[jmoldon]

The problems seems to be more fundamental, and related to how averaging works (or doesn't). The solver crashes because when averaging from 1 sec integration times to 2 sec, some spw get additional interleaved data at wrong times. For example I plot a few seconds of data vs frequency:

Unaveraged data set

Data set averaged to 2 sec integration time

You can see also that the averaged data sets do not match an integer amount of seconds.

Apparently CASA 4.7.2 does not have problem calibrating these data with solint='int', but CASA 5.1 has, and maybe that is why I had not realised before.

Any ideas or suggestions or similar problems?

[jmoldon]

Here it is more clear, only one baseline.

• Unaveraged flagged data set (1 sec integration time):
• Averaged data to 2 sec:

This is the situation:

• The jump of the timestamp after averaging is normal, it is caused because all visibilities in a particular spw are flagged for one integration time, so they are not considered when averaging. That happens for example in columns 1 and 19.
• gaincal crashes when using combine='spw' because there are some visibilities in some particular shifted timestamps that do not have all spw, so the central frequency is inconsistent with previous calibrations (for example a delay calibration with long solution interval and combine='spw').
• The problem is that gaincal in CASA 4.7.2 didn't have a problem, and was correctly correcting the shifted timestamps. But the gaincal in CASA 5.1 cannot deal with the small shifted timestamps and it crashes.
• Also, there is a another problem (I don't think it is relevant here), that the averaging is not symmetric. In the plot above, for times (0,1) the top spw is flagged for timestamp 1, so the averaged visibilities keep the timestamp of the unflagged visibilities (0), good. But that does not happen when the flagged visibilities correspond to "the first timestamp", for example in the case of (2,3), where timestamp 2 is flagged, or (4, 5), where 4 is flagged. That seems inconsistent.

I have opened a NRAO ticket and we are discussing it. If we find a solution I will post it here.

More context: split produces the same results in CASA 4.7 and in 5.1. Note that the CASA 5.1 documentation says that now split uses mstransform underneath. I see exactly the same behaviour in split(4.7), split(5.1) and mstransform, so I don't think the averaging step is the problem, but how gaincal deals with data with small time shifts.

[jmoldon]

This is important:

Apparently all this comes from a more general CASA bug in gaincal. From NRAO I got:

It turns out that this is a bug that was discovered previously by a different means. The cause is a new "invisible" data iteration and retrieval mechanism, "VisibilityIterator2 or VI2", was implemented in CASA 5.0 (see https://science.nrao.edu/enews/casa_005/ )

An easy workaround for you may be to simply not average the data in time, or use the version of CASA that works for the gaincal. If you want to use tclean in CASA 5.1.1, the ms should be fine to jump between 5.1.1 and 4.7.2. Hopefully we'll have this fixed in the next version of CASA....

So there is nothing we can do. The solution for the moment is:

• Our average step will only average in frequency, and will not average in time. Then the whole pipeline can be run in CASA 5.
• For self calibration, use CASA 4.7.

[daniellefenech]

With regards to the gaincal issues, do you only hit the problems in casa 5.0 if you average the data first?

[jmoldon]

As far as I know the problem happens when running gaincal on time-averaged data that has been flagged, and using any version of CASA 5.1.x (I suspect 5.0 also). So the solution for us is not average data in time.

But it can happen in other situations. You can see the CASA bug if you search for bug 11195 in: https://casa.nrao.edu/hdtickets

NRAO 11195 | Problem with gaincal using append on multi-EB dataset

but I don't have access to see more details.

[daniellefenech]

Thanks for that, I don't have access to see more either, maybe Anita could help?

[jmoldon]

I have tried the new CASA version 5.1.2

https://casa.nrao.edu/casadocs/casa-5.1.2/introduction/release-notes

And still has problems related to combine spw to form solutions:

For spw=1:
Current CalTable nchan= 1
Current CalTable freq = [1542649500]
Current Solution nchan= 1
Current Solution freq = [1521316166.67]
Diff = [-21333333.3333]
2018-02-05 17:21:31 SEVERE  Calibrater::solve   Caught exception: Mismatch between Solution frequencies and existing CalTable frequencies for spw=1
2018-02-05 17:21:31 SEVERE      Exception Reported: Error in Calibrater::solve.
*** Error *** Error in Calibrater::solve.
2018-02-05 17:21:31 SEVERE  gaincal:::: Error in gaincal: Error in Calibrater::solve.
2018-02-05 17:21:31 SEVERE  gaincal:::: An error occurred running task gaincal.

The same test was successful when using CASA v4.7

[jmoldon]

Resolved by #100

[amsrichards]

Hi Javier, I just ran into the problem with ALMA data in gaincal where I need to average spw but there are some missing data in some spw. Thank-you so much for the fix of using 4.7.2! But I cannot find the CASA ticket you mention, either on their JIRA or the NRAO web site. Please do you know if it was ever solved? Thanks. PS you can email me a.m.s.richards@manchester.ac.uk if you prefer as this is not really an e-M issue any more.