Support observations where calibrators and targets live in different MSs

[bennahugo]

Reporting this on behalf of Sinah who will be implementing a kludge to make this possible. I suggest doing this in a fork as it may not be needed generally, however this has uses for telescopes other than MeerKAT - e.g. WSRT archival observations which is split by default (at least on all the observations I worked on).

The MeerKAT data in question had to be transferred field by field (which is now supported in KATdal) in order not to fail midway - the dataset is 10 TiB in size and took many days to transfer to Rhodes - more than the allowable time on the token at which point it fails.

Before you say just virtualconcat or concat this:

• virtualconcat fails at the last step because MeerKAT data do not have a POINTING subtable, which I believe is optional in almost all cases except for e.g. raster scan observations using single antennae
• concat in casa 5.x has a serious bug. I plot just the autocorrelations here, but it gives you the picture. concat introduces a massive gap in some of the scans. Note the UTC on the x-axis is different in the concatenated database because there are many earlier scans of the calibrators for other targets than the one we are dealing with. In the non-concatenated data the calibrator scans have no complete dropouts - only an instance where a GNSS satellite passes almost directly overhead which will make the system non-linear and will be flagged by hand later:

whereas the concatenated dataset has the following large gap in that particular scan! This is target data and we do not have many scans available per target so we cannot just throw away perfectly sound data

From our email conversation I highlight the following that needs changing - it looks like a very quick workaround even if we just hardcode the ms names in for now and rerun the relevant steps: Flagging worker:

As I see it the measurement set name is formed here:
https://github.com/caracal-pipeline/caracal/blob/195fa62776aaeb3d3051f044c4bf3724e84c7945/caracal/workers/worker_administrator.py#L216
using
https://github.com/caracal-pipeline/caracal/blob/master/caracal/sample_configurations/meerkat-continuum-defaults.yml#L32
as the label

if https://github.com/caracal-pipeline/caracal/blob/master/caracal/sample_configurations/meerkat-continuum-defaults.yml#L31 is not set to 'target' then
the target argument is left to None, so the it should look something like
'myms-cal.ms'

If you label your calibrator databases appropriately, e.g.
'myms-bpcal.ms'
and
'myms-gaincal.ms'
and
'myms-polcal.ms'
and set
and repeat the 'flag' task suffixed as 'flag', 'flag__2', 'flag__3' setting 'label_in' to 'bpcal', 'gaincal', 'polcal' things should just work.

You will need to adjust this step to have a unique suffix
https://github.com/caracal-pipeline/caracal/blob/master/caracal/sample_configurations/meerkat-continuum-defaults.yml#L152

Transfer worker

Specifically, you need to modify
https://github.com/caracal-pipeline/caracal/blob/master/caracal/workers/crosscal_worker.py#L607
and
https://github.com/caracal-pipeline/caracal/blob/master/caracal/workers/crosscal_worker.py#L610
to accept different input ms during calibration

and application to apply to different calibrator measurement sets:
https://github.com/caracal-pipeline/caracal/blob/master/caracal/workers/crosscal_worker.py#L618
and
https://github.com/caracal-pipeline/caracal/blob/master/caracal/workers/crosscal_worker.py#L626
and
https://github.com/caracal-pipeline/caracal/blob/master/caracal/workers/crosscal_worker.py#L629

The transform worker then needs modification to split out corrected target data from the target measurement set

https://github.com/caracal-pipeline/caracal/blob/master/caracal/workers/transform_worker.py#L89

[o-smirnov]

A bug in CASA? Forsooth! (clutches pearls and faints.)

(Actually, comparing the two plots, it looks like it's not introducing a gap per se, but rather adding extra data after the scan. The contiguous section of data is ~20m in both cases -- and the shape of the curves on both plots match -- it's just that the second plot has an extra ~5m chunk of data tacked on a bit later. Only god knows where the beast got this extra data from, but it can hardly be reputable source!...)

So if I understand correctly, what you really need to work around this problem is support for MSs where each MS contains some subset of targets and calibrators (at the limit, one field per MS). This seems like a very reasonable feature request.

[bennahugo]

Yup. It is very hard to tell where this gap comes from but if it is including a portion of the next scan it is going to cause absolute havoc on the calibration - esp. the phases.

So the way I see it there is just no alternative here than to have support for cases such as this.

On Mon, Jun 21, 2021 at 8:56 PM Oleg Smirnov @.***> wrote:

A bug in CASA? Forsooth! (clutches pearls and faints.)

(Actually, comparing the two plots, it looks like it's not introducing a gap per se, but rather adding extra data after the scan. The contiguous section of data is ~20m in both cases -- and the shape of the curves on both plots match -- it's just that the second plot has an extra ~5m chunk of data tacked on a bit later. Only god knows where the beast got this extra data from, but it can hardly be reputable source!...)

So if I understand correctly, what you really need to work around this problem is support for MSs where each MS contains some subset of targets and calibrators (at the limit, one field per MS). This seems like a very reasonable feature request.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-865268163, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6V4YHSJPUQC4F5PJM3TT6DN3ANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[Sinah-astro]

I'm getting another error when running the crosscal worker and attache the log file indicating the error log-caracal.txt

[KshitijT]

It's a side-effect of skipping the transform step (since you are trying out a custom setup). I very kludgy solution might be to run the pipeline separately for your 'cal' dataset upto only the obsconf step, take the json file and rename it for use in your current run.

[Sinah-astro]

All the calibrators (primary and secondary) are in seperate datasets. So, I'll have two json files to rename and use for my run?

[Sinah-astro]

I've run the pipeline till the prep worker and this file PSZ2G254.08-58.45-cal-obsinfo.txt was produced. Is it okay that even though it's a cal file it doesn't list any calibrators?

[KshitijT]

I've run the pipeline till the prep worker and this file PSZ2G254.08-58.45-cal-obsinfo.txt was produced. Is it okay that even though it's a cal file it doesn't list any calibrators?

Could you actually share the file? And the pipeline should also produce a .json file, can you find it?

[Sinah-astro]

PSZ2G254.08-58.45-cal-obsinfo.txt

This is the file. Yes, I've found the json file but can't seem to upload it here. This is the first bit though: {"FIELD": {"INTENTS": ["UNKNOWN", "TARGET"], "FIELD_ID": [0], "STATE_ID": [1], "PERIOD": [7194.956067085266], "DELAY_DIR": [[[0.8040731863937877, -0.7684859229456232]]], "PHASE_DIR": [[[0.8040731863937877, -0.7684859229456232]]], "REFERENCE_DIR": [[[0.8040731863937877, -0.7684859229456232]]], "CODE": ["T"], "FLAG_ROW": [false], "NAME": ["PSZ2G254.08-58.45"], "NUM_POLY": [0], "SOURCE_ID": [0], "TIME": [5066687879.284202]},

[KshitijT]

It's very surprising that the "cal" dataset(s) has only target scans? Maybe it was mislabelled?

[Sinah-astro]

I'm really not sure because even the one that has to have all the fields only has the target's scans only: PSZ2G254.08-58.45-obsinfo.txt

[KshitijT]

I'm really not sure because even the one that has to have all the fields only has the target's scans only: PSZ2G254.08-58.45-obsinfo.txt

Very weird. @Sinah-astro ,you mentioned two separate dataset for calibrators, do both of them contain only target scans?

[Sinah-astro]

No, the primary and secondary calibrators are each in their own individual datasets, and the target is also in its own dataset

[bennahugo]

On another note - I just realized we are going to hit another small snag because of casa itself. For fluxscale the solutions have to be in one caltable from both fields in order to transfer. However it is not possible at the time of casa 6.0 to append solutions derived from two separate databases (the two calibrator databases). So what we need to do is to replace fluxcal with gaincal on the primary (calmode "a") to solve only the amplitude offset from the normalized bandpass and solve for calmode "p" on the secondary.

On the json. Rather than depending on json can't we just by hand fill the relevant keys of the json and ignore it being missing here. I'm sure this is stemming from the fact that everything is in 3 different databases.

edit: Of course this does not correct amplitude drifts over time but for meerkat at L-band it is really not necessary

[bennahugo]

Alternatively one can deeply image the secondary with a mask and use that model instead of assuming unity. Then fluxscale is not necessary either. Up to you to decide which approach you want to use. I typically use the latter.

[o-smirnov]

This is turning into quite the mission. Reviewing how we got here -- could we not give virtconcat one more try? Perhaps trying to fake a POINTING suitable?

[bennahugo]

you can try casa 6.x. I've tried it with a small portion yesterday (for the ARIWS tutorial), but the timestamps seems very wrong for some of the scans edit: it does not crash unlike casa 5.x but on the other hand the result I get in the listr looks very wrong. You can try compiling the latest casa 6.x wheels and running it. I must say I'm marvelously impressed - CASA 6.x actually compiles in a few minutes and is easier to install than most of our software.

edit: by wrong timestamps I mean the primary and target scan appear to be observed at exactly the same times UTC in the listobs after virtualconcat in casa 6.0! As the gain interpolation requires the time to be accurate this is a no-no

[bennahugo]

Have a look at what I've done here - this runs inside Google Colab https://github.com/bennahugo/ARIWS-Cookbook/blob/main/3-Imaging_and_spectral_lines/TriA%20calibration%20and%20imaging.ipynb

It also applies the former of the two strategies.

[KshitijT]

On the json. Rather than depending on json can't we just by hand fill the relevant keys of the json and ignore it being missing here. I'm sure this is stemming from the fact that everything is in 3 different databases.

I like this idea - maybe there can be a completely "manual" mode (it cuts out a lot of automation but removes the reliance on the json file).

BTW, @bennahugo , have you tried a data transfer through Globus? Would that help fix some of these issues temporarily ?

[bennahugo]

No I haven't. The data is about 10TiB in total and the token is only valid for a week. It takes longer than a week to transfer to Rhodes, so I had to chunk it. I also don't have 10TiB laying around at SARAO to stage it locally at higher speed.

We also no longer have the data staged in the archive for reconversion, so I think it is easier to add manual processing steps - than to restage data from tape, which is a very labour intensive process for Sean and Chris, given the current pressure on SDP.

On Mon, Jun 28, 2021 at 4:00 PM Kshitij Thorat @.***> wrote:

On the json. Rather than depending on json can't we just by hand fill the relevant keys of the json and ignore it being missing here. I'm sure this is stemming from the fact that everything is in 3 different databases.

I like this idea - maybe there can be a completely "manual" mode (it cuts out a lot of automation but removes the reliance on the json file).

BTW, @bennahugo https://github.com/bennahugo , have you tried a data transfer through Globus? Would that help fix some of these issues temporarily ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-869709757, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6V4TDRIHTLMKICNADDTVB56TANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[Sinah-astro]

@bennahugo in the notebook you attached, I see that you're manually doing the self-calibration. Are you suggesting that I do the same instead?

This is turning into quite the mission. Reviewing how we got here -- could we not give virtconcat one more try? Perhaps trying to fake a POINTING suitable?

@o-smirnov can I still use virtualconcat like you're suggesting

[bennahugo]

Hi SInah

This is just an illustrative example of the transfer and selfcal process I put together for ARIWS because the AGN in the field is so pretty - feel free to work through it. We can try to adjust the caracal strategy based on this (approach I mentioned above). The data here is just a single scan and 100MHz subband, and highly averaged to keep things small. We can also try to run the calibration steps manually, replacing the flagging with a combination of hand and automated flagging. Selfcal can be left to run through the pipeline as is.

But this we should perhaps discuss at the meeting - I can't make this week though.

On Mon, Jun 28, 2021 at 4:58 PM Sinah-astro @.***> wrote:

@bennahugo https://github.com/bennahugo in the notebook you attached, I see that you're manually doing the self-calibration. Are you suggesting that I do the same instead?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-869756863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6RALRPGX6YUMVPRD4LTVCEZTANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[o-smirnov]

I'll allocate myself a Caracal day and try to fix this properly...

[o-smirnov]

@bennahugo why do you say the POINTING table is the issue? I get this:

2021-06-29 14:00:58 INFO    virtualconcat::::   Concatenating the POINTING tables ...
2021-06-29 14:00:58 INFO    virtualconcat::::      1559937657.J1939-6342.ms/POINTING
2021-06-29 14:00:58 INFO    virtualconcat::::      1559937657.J1331+3030.ms/POINTING
2021-06-29 14:00:58 INFO    virtualconcat::::      J0203-4349.ms/POINTING
2021-06-29 14:00:58 INFO    virtualconcat::::      PSZ2G254.08-58.45.ms/POINTING
2021-06-29 14:00:58 INFO    ms::createmultims   Copying subtables from 1559937657.J1939-6342.ms to the other MMS members.
2021-06-29 14:00:58 SEVERE  ms::createmultims   Exception Reported: All tables in ConCatTable must have same description
2021-06-29 14:00:58 SEVERE  ms::detached    ms is not attached to a file - cannot perform operation.
2021-06-29 14:00:58 SEVERE  ms::detached+   Call ms.open('filename') to reattach.

I read that as a main tabledesc problem not a POINTING suitable problem. Looking at the table descs, some of the MSs have tiled DATA columns, while others don't. Maybe that's what's tripping the beast over.

Also, I notice SCAN_NUMBER is renumbered from 1 in each per-field MS. Do you reckon concat is being "cunning" and not renumbering scans? Maybe that's where our extra data comes from? I'll do some digging.

[bennahugo]

Likely. I just read it as stemming from somewhere inside POINTING. I did not do further digging. That is not a problem in CASA 6.0 - so if you pull the wheel for python 3.6 then you will see it pass through.

Concat must renumber things - katdal will incrementally increase the scan selection based on dumped number of scans, not the unselected data, but the bug probably lies in how concat and virtualconcat adjusts the scan ID's and the ddids in the main table though (and could be the reason for some of the assumptions in data selection for listobs, plotting and cal breaking after the virtualconcat task in casa 6). Either way the katdal data is conformant to memo 229, so these are solidly casa bugs. What we could of course do is manually relabel the ddids, scan numbers and field numbers before we get started (Ie take care of the bookkeeping casa should have done), but that honestly sounds like more work than just rewriting the some of the calibration procedure to handle the split up case?

The use case is general enough though and is used by other observatories - wsrt measurement set products are stored in different measurement sets. It is quite common and can even be exploited to preaverage calibrator fields which really only need to be used to derive a single solution for interpolation - this can substantially cut down some of that 25% calibrator dataset overhead :)

On Tue, 29 Jun 2021, 16:16 Oleg Smirnov, @.***> wrote:

@bennahugo https://github.com/bennahugo why do you say the POINTING table is the issue? I get this:

2021-06-29 14:00:58 INFO virtualconcat:::: Concatenating the POINTING tables ... 2021-06-29 14:00:58 INFO virtualconcat:::: 1559937657.J1939-6342.ms/POINTING 2021-06-29 http://1559937657.J1939-6342.ms/POINTING2021-06-29 14:00:58 INFO virtualconcat:::: 1559937657.J1331+3030.ms/POINTING 2021-06-29 http://3030.ms/POINTING2021-06-29 14:00:58 INFO virtualconcat:::: J0203-4349.ms/POINTING 2021-06-29 14:00:58 INFO virtualconcat:::: PSZ2G254.08-58.45.ms/POINTING 2021-06-29 http://PSZ2G254.08-58.45.ms/POINTING2021-06-29 14:00:58 INFO ms::createmultims Copying subtables from 1559937657.J1939-6342.ms to the other MMS members. 2021-06-29 14:00:58 SEVERE ms::createmultims Exception Reported: All tables in ConCatTable must have same description 2021-06-29 14:00:58 SEVERE ms::detached ms is not attached to a file - cannot perform operation. 2021-06-29 14:00:58 SEVERE ms::detached+ Call ms.open('filename') to reattach.

I read that as a main tabledesc problem not a POINTING suitable problem. Looking at the table descs, some of the MSs have tiled DATA columns, while others don't. Maybe that's what's tripping the beast over.

Also, I notice SCAN_NUMBER is renumbered from 1 in each per-field MS. Do you reckon concat is being "cunning" and not renumbering scans? Maybe that's where our extra data comes from? I'll do some digging.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-870641185, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6SAALCNXSPL6UWLTELTVHIUDANCNFSM47CDTVKA .

[o-smirnov]

but that honestly sounds like more work than just rewriting the some of the calibration procedure to handle the split up case?

That was my starting-off point... but you've assured me that CASA will refuse to do fluxscale transfer when the calibrators are in different MSs? So we're back at square zero, unless we say "IF calibrators are in different MSs THEN refuse to do fluxscale and insist on imaging", which is the ugliest IF-statement I have yet to write.

QuartiCal will come and save us all, right @JSKenyon?

[bennahugo]

Hi yup - at least that is from my investigation into this. I tried the append to cal table prior to fluxscale and (casa 6.0) refuses to comply. We can try a newer version of the release though...

It is ugly but so is casa 🤣

On Tue, 29 Jun 2021, 16:46 Oleg Smirnov, @.***> wrote:

but that honestly sounds like more work than just rewriting the some of the calibration procedure to handle the split up case?

That was my starting-off point... but you've assured me that CASA will refuse to do fluxscale transfer when the calibrators are in different MSs? So we're back at square zero, unless we say "IF calibrators are in different MSs THEN refuse to do fluxscale and insist on imaging", which is the ugliest IF-statement I have yet to write.

QuartiCal will come and save us all, right @JSKenyon https://github.com/JSKenyon?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-870665251, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6VTYFEUT6DU5SRFJDTTVHMC5ANCNFSM47CDTVKA .

[bennahugo]

I will reproduc my listobs for your enjoyment... give me a few mins

On Tue, 29 Jun 2021, 16:52 Benna Hugo, @.***> wrote:

Hi yup - at least that is from my investigation into this. I tried the append to cal table prior to fluxscale and (casa 6.0) refuses to comply. We can try a newer version of the release though...

It is ugly but so is casa 🤣

On Tue, 29 Jun 2021, 16:46 Oleg Smirnov, @.***> wrote:

but that honestly sounds like more work than just rewriting the some of the calibration procedure to handle the split up case?

That was my starting-off point... but you've assured me that CASA will refuse to do fluxscale transfer when the calibrators are in different MSs? So we're back at square zero, unless we say "IF calibrators are in different MSs THEN refuse to do fluxscale and insist on imaging", which is the ugliest IF-statement I have yet to write.

QuartiCal will come and save us all, right @JSKenyon https://github.com/JSKenyon?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-870665251, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6VTYFEUT6DU5SRFJDTTVHMC5ANCNFSM47CDTVKA .

[bennahugo]

Also you can always do my second stop gap - gaincal mode a on primary calibrator in place of flux scale. It doesn't capture amplitude drifts in time but if the antennas are all behaving it is not necessary, the atmospheric gain isn't too much of a concern.

On Tue, 29 Jun 2021, 16:53 Benna Hugo, @.***> wrote:

I will reproduc my listobs for your enjoyment... give me a few mins

On Tue, 29 Jun 2021, 16:52 Benna Hugo, @.***> wrote:

Hi yup - at least that is from my investigation into this. I tried the append to cal table prior to fluxscale and (casa 6.0) refuses to comply. We can try a newer version of the release though...

It is ugly but so is casa 🤣

On Tue, 29 Jun 2021, 16:46 Oleg Smirnov, @.***> wrote:

but that honestly sounds like more work than just rewriting the some of the calibration procedure to handle the split up case?

That was my starting-off point... but you've assured me that CASA will refuse to do fluxscale transfer when the calibrators are in different MSs? So we're back at square zero, unless we say "IF calibrators are in different MSs THEN refuse to do fluxscale and insist on imaging", which is the ugliest IF-statement I have yet to write.

QuartiCal will come and save us all, right @JSKenyon https://github.com/JSKenyon?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-870665251, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6VTYFEUT6DU5SRFJDTTVHMC5ANCNFSM47CDTVKA .

[bennahugo]

For your enjoyment as promised - I selected 12 antennae (note the rest is not removed from the antennas table by splitting) from the core and intermediate spacings. I purposely cull the long spacings because I want to make calibration easy on my students and limit the sampling rates to make things as small as possible

! rm -rf /content/ARIWS
! mkdir -p /content/ARIWS
! wget ftp://elwood.ru.ac.za/pub/hugo/ARIWS.INTROFIMAGING.TriA.1559937657.tar.gz -P /content/ARIWS
! tar -zxf /content/ARIWS/ARIWS.INTROFIMAGING.TriA.1559937657.tar.gz -C /content/ARIWS

PRIMDB = "/content/ARIWS/1559937657.PRIMARY.1.0.1.1.ghz.avg832khz32s.12ants.scan2.3.ms"
SECDB = "/content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms"
TARGETDB = "/content/ARIWS/1559937657.RXCJ1638.2-6420.1.0.1.1.ghz.avg832khz12s.12ants.scan2.ms"

CONCATDB = "/content/ARIWS/1559937657.allscans.ms"
virtualconcat(vis=[PRIMDB, SECDB, TARGETDB], concatvis=CONCATDB)
listobs(CONCATDB)
pass

yields

2021-06-29 15:19:12 INFO    virtualconcat::::casa   ##########################################
2021-06-29 15:19:12 INFO    virtualconcat::::casa   ##### Begin Task: virtualconcat      #####
2021-06-29 15:19:12 INFO    virtualconcat::::casa   virtualconcat( vis=['/content/ARIWS/1559937657.PRIMARY.1.0.1.1.ghz.avg832khz32s.12ants.scan2.3.ms', '/content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms', '/content/ARIWS/1559937657.RXCJ1638.2-6420.1.0.1.1.ghz.avg832khz12s.12ants.scan2.ms'], concatvis='/content/ARIWS/1559937657.allscans.ms', freqtol='', dirtol='', respectname=True, visweightscale=[], keepcopy=False, copypointing=True )
2021-06-29 15:19:12 INFO    virtualconcat::::casa   Concatenating ...
2021-06-29 15:19:12 INFO    virtualconcat::::casa   adding /content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms to multi-MS /content/ARIWS/1559937657.allscans.ms
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 1 rows and matched 1 from the state subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 1 rows to the source subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 0 rows and matched 1 from the data description subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 0 rows and matched 1 from the spectral window subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 0 rows and matched 62 from the antenna subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat +   Added 0 rows to the feed subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat +   Added 1 rows and matched 0 from the field subtable
2021-06-29 15:19:12 INFO    MSConcat::copyObservation   Added 0 rows and matched 1 rows in the observation subtable.
2021-06-29 15:19:12 INFO    MSConcat::copyProcessor Added 0 rows and matched 0 rows in the processor subtable.
2021-06-29 15:19:12 INFO    MSConcat::copySysCal    No valid syscal tables present. Result won't have one either.
2021-06-29 15:19:12 INFO    MSConcat::copyWeather   No valid weather tables present. Result won't have one either.
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Will create auxiliary file concat_aux_1624979951.782518
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Working on appended Main table ...
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Writing to concat_aux_1624979951.782518
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Writing the scalar columns ...
2021-06-29 15:19:12 INFO    virtualconcat::::casa   adding /content/ARIWS/1559937657.RXCJ1638.2-6420.1.0.1.1.ghz.avg832khz12s.12ants.scan2.ms to multi-MS /content/ARIWS/1559937657.allscans.ms
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 1 rows and matched 1 from the state subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 1 rows to the source subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 0 rows and matched 1 from the data description subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 0 rows and matched 1 from the spectral window subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Added 0 rows and matched 62 from the antenna subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat +   Added 0 rows to the feed subtable
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat +   Added 1 rows and matched 0 from the field subtable
2021-06-29 15:19:12 INFO    MSConcat::copyObservation   Added 0 rows and matched 1 rows in the observation subtable.
2021-06-29 15:19:12 INFO    MSConcat::copyProcessor Added 0 rows and matched 0 rows in the processor subtable.
2021-06-29 15:19:12 INFO    MSConcat::copySysCal    No valid syscal tables present. Result won't have one either.
2021-06-29 15:19:12 INFO    MSConcat::copyWeather   No valid weather tables present. Result won't have one either.
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Reading from concat_aux_1624979951.782518
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Working on appended Main table ...
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Writing to concat_aux_1624979951.782518
2021-06-29 15:19:12 INFO    MSConcat::virtualconcat     Writing the scalar columns ...
2021-06-29 15:19:12 INFO    virtualconcat::::casa   Concatenating the POINTING tables ...
2021-06-29 15:19:12 INFO    virtualconcat::::casa      /content/ARIWS/1559937657.PRIMARY.1.0.1.1.ghz.avg832khz32s.12ants.scan2.3.ms/POINTING
2021-06-29 15:19:12 INFO    virtualconcat::::casa      /content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms/POINTING
2021-06-29 15:19:12 INFO    virtualconcat::::casa      /content/ARIWS/1559937657.RXCJ1638.2-6420.1.0.1.1.ghz.avg832khz12s.12ants.scan2.ms/POINTING
2021-06-29 15:19:12 INFO    ms::createmultims   Copying subtables from /content/ARIWS/1559937657.PRIMARY.1.0.1.1.ghz.avg832khz32s.12ants.scan2.3.ms to the other MMS members.
2021-06-29 15:19:13 INFO    virtualconcat::::casa   Task virtualconcat complete. Start time: 2021-06-29 15:19:11.731029 End time: 2021-06-29 15:19:12.547123
2021-06-29 15:19:13 INFO    virtualconcat::::casa   ##### End Task: virtualconcat        #####
2021-06-29 15:19:13 INFO    virtualconcat::::casa   ##########################################

2021-06-29 15:19:13 INFO    listobs::::casa ##########################################
2021-06-29 15:19:13 INFO    listobs::::casa ##### Begin Task: listobs            #####
2021-06-29 15:19:13 INFO    listobs::::casa listobs( vis='/content/ARIWS/1559937657.allscans.ms', selectdata=True, spw='', field='', antenna='', uvrange='', timerange='', correlation='', scan='', intent='', feed='', array='', observation='', verbose=True, listfile='', listunfl=False, cachesize=50.0, overwrite=False )
2021-06-29 15:19:13 INFO    listobs::ms::summary    ================================================================================
2021-06-29 15:19:13 INFO    listobs::ms::summary+              MeasurementSet Name:  /content/ARIWS/1559937657.allscans.ms      MS Version 2
2021-06-29 15:19:13 INFO    listobs::ms::summary+   ================================================================================
2021-06-29 15:19:13 INFO    listobs::ms::summary+      Observer: Marisa Geyer     Project: 20190607-0020  
2021-06-29 15:19:13 INFO    listobs::ms::summary+   Observation: MeerKAT
2021-06-29 15:19:13 INFO    listobs::MSMetaData::_computeScanAndSubScanProperties   Computing scan and subscan properties...
2021-06-29 15:19:13 INFO    listobs::ms::summary    Data records: 90576       Total elapsed time = 7892.66 seconds
2021-06-29 15:19:13 INFO    listobs::ms::summary+      Observed from   07-Jun-2019/22:27:22.7   to   08-Jun-2019/00:38:55.4 (UTC)
2021-06-29 15:19:13 INFO    listobs::ms::summary    
2021-06-29 15:19:13 INFO    listobs::ms::summary+      ObservationID = 0         ArrayID = 0
2021-06-29 15:19:13 INFO    listobs::ms::summary+     Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
2021-06-29 15:19:13 INFO    listobs::ms::summary+     07-Jun-2019/22:27:22.7 - 23:21:03.3     2      0 J1939-6342                6660  [0]  [29.6] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
2021-06-29 15:19:13 INFO    listobs::ms::summary+                 22:27:22.7 - 23:21:03.3     2      2 RXCJ1638.2-6420          67266  [0]  [11.9] [TARGET]
2021-06-29 15:19:13 INFO    listobs::ms::summary+     08-Jun-2019/00:33:59.5 - 00:38:55.4     3      0 J1939-6342                6660  [0]  [29.5] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
2021-06-29 15:19:13 INFO    listobs::ms::summary+     07-Jun-2019/22:58:05.9 - 23:00:35.9    10      1 J1859-6615                3330  [0]  [29.8] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
2021-06-29 15:19:13 INFO    listobs::ms::summary+                 23:21:23.3 - 23:23:49.3    11      1 J1859-6615                3330  [0]  [29.2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
2021-06-29 15:19:13 INFO    listobs::ms::summary+                 23:44:38.7 - 23:47:06.7    12      1 J1859-6615                3330  [0]  [29] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
2021-06-29 15:19:13 INFO    listobs::ms::summary               (nRows = Total number of rows per scan) 
2021-06-29 15:19:13 INFO    listobs::ms::summary    Fields: 3
2021-06-29 15:19:13 INFO    listobs::ms::summary+     ID   Code Name                RA               Decl           Epoch   SrcId      nRows
2021-06-29 15:19:13 INFO    listobs::ms::summary+     0    T    J1939-6342          19:39:25.050000 -63.42.43.60000 J2000   0          13320
2021-06-29 15:19:13 INFO    listobs::ms::summary+     1    T    J1859-6615          18:59:57.800000 -66.15.02.80000 J2000   1           9990
2021-06-29 15:19:13 INFO    listobs::ms::summary+     2    T    RXCJ1638.2-6420     16:38:18.300000 -64.21.07.00000 J2000   2          67266
2021-06-29 15:19:13 INFO    listobs::ms::summary    Spectral Windows:  (1 unique spectral windows and 1 unique polarization setups)
2021-06-29 15:19:13 INFO    listobs::ms::summary+     SpwID  Name   #Chans   Frame   Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) CtrFreq(MHz)  Corrs          
2021-06-29 15:19:13 INFO    listobs::ms::summary+     0      none     119   TOPO    1000.513       835.938     99476.6   1049.8330   XX  XY  YX  YY
2021-06-29 15:19:13 INFO    listobs::ms::summary    Sources: 3
2021-06-29 15:19:13 INFO    listobs::ms::summary+     ID   Name                SpwId RestFreq(MHz)  
2021-06-29 15:19:13 INFO    listobs::ms::summary+     0    J1939-6342          0     -              
2021-06-29 15:19:13 INFO    listobs::ms::summary+     1    J1859-6615          0     -              
2021-06-29 15:19:13 INFO    listobs::ms::summary+     2    RXCJ1638.2-6420     0     -              
2021-06-29 15:19:13 INFO    listobs::ms::summary+     NB: No systemic velocity information found in SOURCE table.
2021-06-29 15:19:13 INFO    listobs::ms::summary    Antennas: 62:
2021-06-29 15:19:13 INFO    listobs::ms::summary+     ID   Name  Station   Diam.    Long.         Lat.                Offset from array center (m)                ITRF Geocentric coordinates (m)        
2021-06-29 15:19:13 INFO    listobs::ms::summary+                                                                        East         North     Elevation               x               y               z
2021-06-29 15:19:13 INFO    listobs::ms::summary+     0    m000  m000      13.5 m   +021.26.37.7  -30.32.39.3         -9.3284       30.3604       45.2895  5109271.497354  2006808.893028 -3239130.736141
2021-06-29 15:19:13 INFO    listobs::ms::summary+     1    m001  m001      13.5 m   +021.26.38.0  -30.32.38.1          0.0560       65.8884       45.2670  5109284.854078  2006824.221724 -3239100.126460
2021-06-29 15:19:13 INFO    listobs::ms::summary+     2    m002  m002      13.5 m   +021.26.36.8  -30.32.39.8        -33.1772       13.4145       45.2882  5109272.199343  2006783.546050 -3239145.330042
2021-06-29 15:19:13 INFO    listobs::ms::summary+     3    m003  m003      13.5 m   +021.26.35.5  -30.32.39.1        -67.5818       35.3754       44.9929  5109294.928585  2006755.509854 -3239126.266250
2021-06-29 15:19:13 INFO    listobs::ms::summary+     4    m004  m004      13.5 m   +021.26.33.4  -30.32.40.7       -124.6878      -15.2941       45.0744  5109291.902149  2006692.968027 -3239169.946413
2021-06-29 15:19:13 INFO    listobs::ms::summary+     5    m005  m005      13.5 m   +021.26.34.2  -30.32.41.7       -103.1522      -45.4691       45.3485  5109269.975114  2006707.493216 -3239196.073499
2021-06-29 15:19:13 INFO    listobs::ms::summary+     6    m006  m006      13.5 m   +021.26.37.3  -30.32.42.1        -19.2960      -57.7783       45.6250  5109233.717826  2006783.345193 -3239206.815200
2021-06-29 15:19:13 INFO    listobs::ms::summary+     7    m007  m007      13.5 m   +021.26.34.6  -30.32.45.6        -90.6572     -165.0824       45.8961  5109209.263489  2006697.072283 -3239299.366683
2021-06-29 15:19:13 INFO    listobs::ms::summary+     8    m008  m008      13.5 m   +021.26.34.5  -30.32.49.9        -94.5934     -297.3768       46.1924  5109148.356361  2006668.921549 -3239413.452183
2021-06-29 15:19:13 INFO    listobs::ms::summary+     9    m009  m009      13.5 m   +021.26.39.2  -30.32.44.6         31.2922     -133.4077       46.3583  5109180.035241  2006816.610113 -3239272.322421
2021-06-29 15:19:13 INFO    listobs::ms::summary+     10   m010  m010      13.5 m   +021.26.41.3  -30.32.49.1         87.0321     -274.2252       46.9996  5109093.556556  2006842.526947 -3239393.923881
2021-06-29 15:19:13 INFO    listobs::ms::summary+     11   m011  m011      13.5 m   +021.26.41.2  -30.32.43.9         82.9473     -114.4301       46.4246  5109170.180559  2006868.236095 -3239256.012063
2021-06-29 15:19:13 INFO    listobs::ms::summary+     12   m012  m012      13.5 m   +021.26.43.3  -30.32.44.5        138.9552     -130.6192       46.6763  5109142.247324  2006917.437396 -3239270.082552
2021-06-29 15:19:13 INFO    listobs::ms::summary+     13   m013  m013      13.5 m   +021.26.46.9  -30.32.45.3        235.7287     -155.8122       47.2820  5109095.433207  2007003.019458 -3239292.087312
2021-06-29 15:19:13 INFO    listobs::ms::summary+     14   m014  m014      13.5 m   +021.26.48.5  -30.32.41.8        279.6037      -48.1427       47.0176  5109130.110480  2007063.780755 -3239199.224411
2021-06-29 15:19:13 INFO    listobs::ms::summary+     15   m015  m015      13.5 m   +021.26.45.9  -30.32.39.6        209.5771       18.5078       46.3999  5109186.746785  2007010.792885 -3239141.508486
2021-06-29 15:19:13 INFO    listobs::ms::summary+     16   m016  m016      13.5 m   +021.26.48.8  -30.32.38.6        287.0910       51.7775       46.5569  5109174.267883  2007089.171932 -3239112.934879
2021-06-29 15:19:13 INFO    listobs::ms::summary+     17   m017  m017      13.5 m   +021.26.45.5  -30.32.36.2        198.5552      125.3871       45.9280  5109240.953886  2007020.244747 -3239049.219156
2021-06-29 15:19:13 INFO    listobs::ms::summary+     18   m018  m018      13.5 m   +021.26.42.0  -30.32.40.5        104.6613       -8.2209       46.1106  5109212.230825  2006908.082578 -3239164.381246
2021-06-29 15:19:13 INFO    listobs::ms::summary+     19   m019  m019      13.5 m   +021.26.44.4  -30.32.41.8        169.7215      -47.5746       46.5393  5109170.171973  2006961.461379 -3239198.492114
2021-06-29 15:19:13 INFO    listobs::ms::summary+     20   m020  m020      13.5 m   +021.26.41.6  -30.32.42.2         95.9514      -61.9898       46.3020  5109190.134224  2006890.045163 -3239210.786387
2021-06-29 15:19:13 INFO    listobs::ms::summary+     21   m021  m021      13.5 m   +021.26.26.9  
...
2021-06-29 15:19:13 INFO    listobs::::casa ##### End Task: listobs              #####
2021-06-29 15:19:13 INFO    listobs::::casa ##########################################

Low and behold our 12 antennas have pulled off the ultimate stunt of boss-eyed observing of calibrator and target fields that astronomers secretly lank for all these years!!!

2021-06-29 15:19:13 INFO    listobs::ms::summary+     07-Jun-2019/22:27:22.7 - 23:21:03.3     2      0 J1939-6342                6660  [0]  [29.6] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
2021-06-29 15:19:13 INFO    listobs::ms::summary+                 22:27:22.7 - 23:21:03.3     2      2 RXCJ1638.2-6420          67266  [0]  [11.9] [TARGET]

[bennahugo]

@o-smirnov simplest recipe (already solved for bandpasses at this point - the gaincal usually needed before fluxscale solving for solutions on both primary and secondary in order to scale from the unity model assumption of the secondary needs to of course be split into two steps with two different datasets. As I said casa do not allow appending to the existing gain table because I guess the subtables, e.g. FIELD are different and it does not have the machinery to append the new rows to that.

gaincal(vis=SECDB, field="J1859-6615", caltable="sec.K0", gaintype="K", refant="m000", 
        interp=["linear"], gaintable=["prim.B0"], 
        gainfield=["J1939-6342"], combine="", solint="inf")
gaincal(vis=SECDB, field="J1859-6615", caltable="sec.G0", gaintype="G", refant="m000", 
        calmode="ap",
        interp=["linear","nearest"], gaintable=["prim.B0","sec.K0"], 
        gainfield=["J1939-6342","J1859-6615"], combine="", solint="inf")
gaincal(vis=PRIMDB, field="J1939-6342", caltable="sec.G0", # we write these into to the same table for fluxscale
        append=True,
        gaintype="G", refant="m000", 
        calmode="ap",
        interp=["nearest","linear",], 
        gaintable=["prim.K0","prim.B0"], 
        gainfield=["J1939-6342","J1939-6342"], combine="", solint="inf")

produces

2021-06-29 16:31:01 INFO    gaincal::::casa ##########################################
2021-06-29 16:31:01 INFO    gaincal::::casa ##### Begin Task: gaincal            #####
2021-06-29 16:31:01 INFO    gaincal::::casa gaincal( vis='/content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms', caltable='sec.K0', field='J1859-6615', spw='', intent='', selectdata=True, timerange='', uvrange='', antenna='', scan='', observation='', msselect='', solint='inf', combine='', preavg=-1.0, refant='m000', refantmode='flex', minblperant=4, minsnr=3.0, solnorm=False, normtype='mean', gaintype='K', smodel=[], calmode='ap', solmode='', rmsthresh=[], corrdepflags=False, append=False, splinetime=3600.0, npointaver=3, phasewrap=180.0, docallib=False, callib='', gaintable=['prim.B0'], gainfield=['J1939-6342'], interp=['linear'], spwmap=[], parang=False )
2021-06-29 16:31:01 INFO    gaincal::calibrater::open   ****Using NEW VI2-driven calibrater tool****
2021-06-29 16:31:01 INFO    gaincal::calibrater::open   Opening MS: /content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms for calibration.
2021-06-29 16:31:01 INFO    gaincal::Calibrater::   Initializing nominal selection to the whole MS.
2021-06-29 16:31:01 INFO    gaincal::::casa NB: gaincal automatically excludes auto-correlations.
2021-06-29 16:31:01 INFO    calibrater::setdata Beginning selectvis--(MSSelection version)-------
2021-06-29 16:31:01 INFO    calibrater::reset   Reseting solve/apply state
2021-06-29 16:31:01 INFO    Calibrater::selectvis   Performing selection on MeasurementSet
2021-06-29 16:31:01 INFO    Calibrater::selectvis+   Selecting on field: 'J1859-6615'
2021-06-29 16:31:01 INFO    Calibrater::selectvis+   Selecting with TaQL: 'ANTENNA1!=ANTENNA2'
2021-06-29 16:31:01 INFO    Calibrater::selectvis   Selection did not drop any rows
2021-06-29 16:31:01 INFO    Calibrater::selectvis   Frequency selection: Selecting all channels in all spws.
2021-06-29 16:31:01 INFO    calibrater::setdata chanmode=none nchan=1 start=0 step=1 mStart='0km/s' mStep='0km/s' msSelect='ANTENNA1!=ANTENNA2'
2021-06-29 16:31:01 INFO    calibrater::setapply    Beginning setapply--(MSSelection version)-------
2021-06-29 16:31:01 INFO    Calibrater::setapply(type, applypar)    Arranging to APPLY:
2021-06-29 16:31:01 INFO    Calibrater::setapply(type, applypar)    .   B Jones: table=prim.B0 select= interp=linear,linear spwmap=[-1] calWt=true
2021-06-29 16:31:01 INFO    calibrater::setsolve    Beginning setsolve--(MSSelection version)-------
2021-06-29 16:31:01 INFO    Calibrater::setsolve    Arranging to SOLVE:
2021-06-29 16:31:01 INFO    Calibrater::setsolve    .   K Jones: table=sec.K0 append=false solint=inf refantmode='flex' refant='m000' minsnr=3 apmode=AP solnorm=false
2021-06-29 16:31:01 INFO    calibrater::solve   Beginning solve-----------------------------
2021-06-29 16:31:01 INFO    Calibrater::solve   The following calibration terms are arranged for apply:
2021-06-29 16:31:01 INFO    Calibrater::solve   .   B Jones: table=prim.B0 select= interp=linear,linear spwmap=[-1] calWt=true
2021-06-29 16:31:01 INFO    Calibrater::solve   The following calibration term is arranged for solve:
2021-06-29 16:31:01 INFO    Calibrater::solve   .   K Jones: table=sec.K0 append=false solint=inf refantmode='flex' refant='m000' minsnr=3 apmode=AP solnorm=false
2021-06-29 16:31:01 INFO    Calibrater::solve   For solint = inf, found 3 solution intervals.
2021-06-29 16:31:02 INFO    Calibrater::solve     Found good K Jones solutions in 3 solution intervals.
2021-06-29 16:31:02 INFO        Writing solutions to table: sec.K0
2021-06-29 16:31:02 INFO    calibrater::solve   Finished solving.
2021-06-29 16:31:02 INFO    gaincal::::casa Calibration solve statistics per spw:  (expected/attempted/succeeded):
2021-06-29 16:31:02 INFO    gaincal::::casa   Spw 0: 3/3/3
2021-06-29 16:31:02 INFO    gaincal::::casa Task gaincal complete. Start time: 2021-06-29 16:31:00.546801 End time: 2021-06-29 16:31:02.111249
2021-06-29 16:31:02 INFO    gaincal::::casa ##### End Task: gaincal              #####
2021-06-29 16:31:02 INFO    gaincal::::casa ##########################################

2021-06-29 16:31:02 INFO    gaincal::::casa ##########################################
2021-06-29 16:31:02 INFO    gaincal::::casa ##### Begin Task: gaincal            #####
2021-06-29 16:31:02 INFO    gaincal::::casa gaincal( vis='/content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms', caltable='sec.G0', field='J1859-6615', spw='', intent='', selectdata=True, timerange='', uvrange='', antenna='', scan='', observation='', msselect='', solint='inf', combine='', preavg=-1.0, refant='m000', refantmode='flex', minblperant=4, minsnr=3.0, solnorm=False, normtype='mean', gaintype='G', smodel=[], calmode='ap', solmode='', rmsthresh=[], corrdepflags=False, append=False, splinetime=3600.0, npointaver=3, phasewrap=180.0, docallib=False, callib='', gaintable=['prim.B0', 'sec.K0'], gainfield=['J1939-6342', 'J1859-6615'], interp=['linear', 'nearest'], spwmap=[], parang=False )
2021-06-29 16:31:02 INFO    gaincal::calibrater::open   ****Using NEW VI2-driven calibrater tool****
2021-06-29 16:31:02 INFO    gaincal::calibrater::open   Opening MS: /content/ARIWS/1559937657.SECONDARY.1.0.1.1.ghz.avg832khz32s.12ants.ms for calibration.
2021-06-29 16:31:02 INFO    gaincal::Calibrater::   Initializing nominal selection to the whole MS.
2021-06-29 16:31:02 INFO    gaincal::::casa NB: gaincal automatically excludes auto-correlations.
2021-06-29 16:31:02 INFO    calibrater::setdata Beginning selectvis--(MSSelection version)-------
2021-06-29 16:31:02 INFO    calibrater::reset   Reseting solve/apply state
2021-06-29 16:31:02 INFO    Calibrater::selectvis   Performing selection on MeasurementSet
2021-06-29 16:31:02 INFO    Calibrater::selectvis+   Selecting on field: 'J1859-6615'
2021-06-29 16:31:02 INFO    Calibrater::selectvis+   Selecting with TaQL: 'ANTENNA1!=ANTENNA2'
2021-06-29 16:31:02 INFO    Calibrater::selectvis   Selection did not drop any rows
2021-06-29 16:31:02 INFO    Calibrater::selectvis   Frequency selection: Selecting all channels in all spws.
2021-06-29 16:31:02 INFO    calibrater::setdata chanmode=none nchan=1 start=0 step=1 mStart='0km/s' mStep='0km/s' msSelect='ANTENNA1!=ANTENNA2'
2021-06-29 16:31:02 INFO    calibrater::setapply    Beginning setapply--(MSSelection version)-------
2021-06-29 16:31:02 INFO    Calibrater::setapply(type, applypar)    Arranging to APPLY:
2021-06-29 16:31:02 INFO    Calibrater::setapply(type, applypar)    .   B Jones: table=prim.B0 select= interp=linear,linear spwmap=[-1] calWt=true
2021-06-29 16:31:02 INFO    calibrater::setapply    Beginning setapply--(MSSelection version)-------
2021-06-29 16:31:02 INFO    Calibrater::setapply(type, applypar)    Arranging to APPLY:
2021-06-29 16:31:02 INFO         (K Jones: Enforcing calWt()=false for phase/delay-like terms)
2021-06-29 16:31:02 INFO    Calibrater::setapply(type, applypar)    .   K Jones: table=sec.K0 select= interp=nearest spwmap=[-1] calWt=false
2021-06-29 16:31:02 INFO    calibrater::setsolve    Beginning setsolve--(MSSelection version)-------
2021-06-29 16:31:02 INFO    Calibrater::setsolve    Arranging to SOLVE:
2021-06-29 16:31:02 INFO    Calibrater::setsolve    .   G Jones: table=sec.G0 append=false solint=inf refantmode='flex' refant='m000' minsnr=3 apmode=AP solnorm=false
2021-06-29 16:31:02 INFO    calibrater::solve   Beginning solve-----------------------------
2021-06-29 16:31:02 INFO    Calibrater::solve   The following calibration terms are arranged for apply:
2021-06-29 16:31:02 INFO    Calibrater::solve   .   B Jones: table=prim.B0 select= interp=linear,linear spwmap=[-1] calWt=true
2021-06-29 16:31:02 INFO    Calibrater::solve   .   K Jones: table=sec.K0 select= interp=nearest spwmap=[-1] calWt=false
2021-06-29 16:31:02 INFO    Calibrater::solve   The following calibration term is arranged for solve:
2021-06-29 16:31:02 INFO    Calibrater::solve   .   G Jones: table=sec.G0 append=false solint=inf refantmode='flex' refant='m000' minsnr=3 apmode=AP solnorm=false
2021-06-29 16:31:02 INFO    ChannelAverageTVI::parseConfiguration   Channel bin is [-1]
2021-06-29 16:31:02 INFO    Calibrater::solve   For solint = inf, found 3 solution intervals.
2021-06-29 16:31:03 INFO    Calibrater::solve     Found good G Jones solutions in 3 solution intervals.
2021-06-29 16:31:03 INFO        Applying refant: m000 refantmode = flex (hold alternate refants' phase constant) when refant flagged
2021-06-29 16:31:03 INFO        Writing solutions to table: sec.G0
2021-06-29 16:31:03 INFO    calibrater::solve   Finished solving.
2021-06-29 16:31:03 INFO    gaincal::::casa Calibration solve statistics per spw:  (expected/attempted/succeeded):
2021-06-29 16:31:03 INFO    gaincal::::casa   Spw 0: 3/3/3
2021-06-29 16:31:03 INFO    gaincal::::casa Task gaincal complete. Start time: 2021-06-29 16:31:02.115577 End time: 2021-06-29 16:31:03.265190
2021-06-29 16:31:03 INFO    gaincal::::casa ##### End Task: gaincal              #####
2021-06-29 16:31:03 INFO    gaincal::::casa ##########################################

2021-06-29 16:31:03 INFO    gaincal::::casa ##########################################
2021-06-29 16:31:03 INFO    gaincal::::casa ##### Begin Task: gaincal            #####
2021-06-29 16:31:03 INFO    gaincal::::casa gaincal( vis='/content/ARIWS/1559937657.PRIMARY.1.0.1.1.ghz.avg832khz32s.12ants.scan2.3.ms', caltable='sec.G0', field='J1939-6342', spw='', intent='', selectdata=True, timerange='', uvrange='', antenna='', scan='', observation='', msselect='', solint='inf', combine='', preavg=-1.0, refant='m000', refantmode='flex', minblperant=4, minsnr=3.0, solnorm=False, normtype='mean', gaintype='G', smodel=[], calmode='ap', solmode='', rmsthresh=[], corrdepflags=False, append=True, splinetime=3600.0, npointaver=3, phasewrap=180.0, docallib=False, callib='', gaintable=['prim.K0', 'prim.B0'], gainfield=['J1939-6342', 'J1939-6342'], interp=['nearest', 'linear'], spwmap=[], parang=False )
2021-06-29 16:31:03 INFO    gaincal::calibrater::open   ****Using NEW VI2-driven calibrater tool****
2021-06-29 16:31:03 INFO    gaincal::calibrater::open   Opening MS: /content/ARIWS/1559937657.PRIMARY.1.0.1.1.ghz.avg832khz32s.12ants.scan2.3.ms for calibration.
2021-06-29 16:31:03 INFO    gaincal::Calibrater::   Initializing nominal selection to the whole MS.
2021-06-29 16:31:03 INFO    gaincal::::casa NB: gaincal automatically excludes auto-correlations.
2021-06-29 16:31:03 INFO    calibrater::setdata Beginning selectvis--(MSSelection version)-------
2021-06-29 16:31:03 INFO    calibrater::reset   Reseting solve/apply state
2021-06-29 16:31:03 INFO    Calibrater::selectvis   Performing selection on MeasurementSet
2021-06-29 16:31:03 INFO    Calibrater::selectvis+   Selecting on field: 'J1939-6342'
2021-06-29 16:31:03 INFO    Calibrater::selectvis+   Selecting with TaQL: 'ANTENNA1!=ANTENNA2'
2021-06-29 16:31:03 INFO    Calibrater::selectvis   Selection did not drop any rows
2021-06-29 16:31:03 INFO    Calibrater::selectvis   Frequency selection: Selecting all channels in all spws.
2021-06-29 16:31:03 INFO    calibrater::setdata chanmode=none nchan=1 start=0 step=1 mStart='0km/s' mStep='0km/s' msSelect='ANTENNA1!=ANTENNA2'
2021-06-29 16:31:03 INFO    calibrater::setapply    Beginning setapply--(MSSelection version)-------
2021-06-29 16:31:03 INFO    Calibrater::setapply(type, applypar)    Arranging to APPLY:
2021-06-29 16:31:03 INFO         (K Jones: Enforcing calWt()=false for phase/delay-like terms)
2021-06-29 16:31:03 INFO    Calibrater::setapply(type, applypar)    .   K Jones: table=prim.K0 select= interp=nearest spwmap=[-1] calWt=false
2021-06-29 16:31:03 INFO    calibrater::setapply    Beginning setapply--(MSSelection version)-------
2021-06-29 16:31:03 INFO    Calibrater::setapply(type, applypar)    Arranging to APPLY:
2021-06-29 16:31:03 INFO    Calibrater::setapply(type, applypar)    .   B Jones: table=prim.B0 select= interp=linear,linear spwmap=[-1] calWt=true
2021-06-29 16:31:03 INFO    calibrater::setsolve    Beginning setsolve--(MSSelection version)-------
2021-06-29 16:31:03 INFO    Calibrater::setsolve    Arranging to SOLVE:
2021-06-29 16:31:03 INFO    Calibrater::setsolve    .   G Jones: table=sec.G0 append=true solint=inf refantmode='flex' refant='m000' minsnr=3 apmode=AP solnorm=false
2021-06-29 16:31:03 INFO    calibrater::solve   Beginning solve-----------------------------
2021-06-29 16:31:03 INFO    Calibrater::solve   The following calibration terms are arranged for apply:
2021-06-29 16:31:03 INFO    Calibrater::solve   .   B Jones: table=prim.B0 select= interp=linear,linear spwmap=[-1] calWt=true
2021-06-29 16:31:03 INFO    Calibrater::solve   .   K Jones: table=prim.K0 select= interp=nearest spwmap=[-1] calWt=false
2021-06-29 16:31:03 INFO    Calibrater::solve   The following calibration term is arranged for solve:
2021-06-29 16:31:03 INFO    Calibrater::solve   .   G Jones: table=sec.G0 append=true solint=inf refantmode='flex' refant='m000' minsnr=3 apmode=AP solnorm=false
2021-06-29 16:31:03 INFO    ChannelAverageTVI::parseConfiguration   Channel bin is [-1]
2021-06-29 16:31:03 INFO    Calibrater::solve   For solint = inf, found 2 solution intervals.
2021-06-29 16:31:05 INFO    Calibrater::solve     Found good G Jones solutions in 2 solution intervals.
2021-06-29 16:31:05 INFO        Applying refant: m000 refantmode = flex (hold alternate refants' phase constant) when refant flagged
2021-06-29 16:31:05 INFO        Appending solutions to table: sec.G0
2021-06-29 16:31:05 SEVERE  Calibrater::solve   Caught exception: Cannot append solutions from different MS.
2021-06-29 16:31:05 INFO    Calibrater::solve   Reseting entire solve/apply state.
2021-06-29 16:31:05 SEVERE      Exception Reported: Error in Calibrater::solve.
2021-06-29 16:31:05 SEVERE  gaincal::::casa Task gaincal raised an exception of class RuntimeError with the following message: Error in Calibrater::solve.
2021-06-29 16:31:05 INFO    gaincal::::casa Task gaincal complete. Start time: 2021-06-29 16:31:03.269420 End time: 2021-06-29 16:31:04.744063
2021-06-29 16:31:05 INFO    gaincal::::casa ##### End Task: gaincal              #####
2021-06-29 16:31:05 INFO    gaincal::::casa ##########################################

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-34-928baadc0019> in <module>()
     32         interp=["nearest","linear",],
     33         gaintable=["prim.K0","prim.B0"],
---> 34         gainfield=["J1939-6342","J1939-6342"], combine="", solint="inf")
     35 
     36 # applycal(vis=SECDB, field="J1859-6615",

5 frames

/usr/local/lib/python3.7/dist-packages/casatools/__casac__/calibrater.py in solve(self)
   1083 
   1084         """
-> 1085         return _calibrater.calibrater_solve(self)
   1086 
   1087 

RuntimeError: Error in Calibrater::solve.

The kludgy workaround mentioned above will look something along the lines of

gaincal(vis=PRIMDB, field="J1939-6342", caltable="prim.F0", gaintype="G", refant="m000", 
        calmode="a",
        interp=["nearest"], gaintable=["prim.K0"], 
        gainfield=["J1939-6342"], combine="", solint="inf")
gaincal(vis=SECDB, field="J1859-6615", caltable="sec.K0", gaintype="K", refant="m000", 
        interp=["linear","linear"], gaintable=["prim.B0", "prim.F0"], 
        gainfield=["J1939-6342","J1939-6342"], combine="", solint="inf")
gaincal(vis=SECDB, field="J1859-6615", caltable="sec.G0", gaintype="G", refant="m000", 
        calmode="p",
        interp=["linear","linear","nearest"], gaintable=["prim.B0","prim.F0","sec.K0"], 
        gainfield=["J1939-6342","J1939-6342","J1859-6615"], combine="", solint="inf")
applycal(vis=SECDB, field="J1859-6615",
         interp=["nearest", "linear", "linear", "linear"], 
         gaintable=["prim.B0", "prim.F0", "sec.K0", "sec.G0"], 
         gainfield=["J1939-6342", "J1939-6342", "J1859-6615", "J1859-6615"])

where the bandpass is already normalized by AP solution stored in prim.G0 - here I just extract the A part for application to the secondary and other targets. The phases of course must only come from the secondary as we have had before.

[o-smirnov]

OK so:

• The assumption of "one MS, one full set of fields" is baked deep into the workers. I agree this was short-sighted design, but it's going to take a bit of surgery to fix it. Let's leave this to R2.

• The problem with concat, is presumably, clashing scan numbers so this can be worked around!

• The problem with virtualconcat is not in fact POINTING, but the fact that the MSs have different descriptions (tiled data managers and all that ~bullsh~arcana). Presumably this can be worked around by using split to make a copy of them (sampling down to 8s in the process).

I have written a script to renumber scans in a series of MSs. See /net/oates/home/oms/sinah-test/m-orig/renumber_scans.py. Run it with all the MSs that need to be renumbered as arguments.

So let's go back to a single MS. Two solutions, from here:

Option 1:

• @bennahugo runs my script to renumber his original MSs in the vault (brrrr a little scary, right?)

• @Sinah-astro runs concat to copy-and-concatenate the relevant ones (using the original MSs as the sources)

Option 2:

• @Sinah-astro runs split to split out her copies of 8s data from the original vault MSs

• @Sinah-astro runs my script to renumber the scans in her 8s copies

• @Sinah-astro runs virtualconcat to concatenate her renumbered 8s copies into one multi-MS.

Note also that the result will have redundant calibrator scans (multiple bandpass scans at the start, multiple gaincal scans in a consecutive block) e.g.

Renumbering into 18 scans:
  1: subset.1559937657.J1939-6342.ms scan 1
  2: subset.1559937657.J1331+3030.ms scan 2
  3: subset.1559937657.J1939-6342.ms scan 3
  4: subset.1559937657.J1939-6342.ms scan 4
  5: subset.1559937657.J1939-6342.ms scan 5
  6: subset.1559937657.J1939-6342.ms scan 6
  7: subset.1559937657.J1939-6342.ms scan 7
  8: subset.J0203-4349.ms scan 8
  9: subset.PSZ2G254.08-58.45.ms scan 9
  10: subset.PSZ2G254.08-58.45.ms scan 10
  11: subset.J0203-4349.ms scan 11
  12: subset.J0203-4349.ms scan 12
  13: subset.J0203-4349.ms scan 13
  14: subset.1559937657.J1939-6342.ms scan 14
  15: subset.J0203-4349.ms scan 15
  16: subset.J0203-4349.ms scan 16
  17: subset.J0203-4349.ms scan 17
  18: subset.J0203-4349.ms scan 18

...so @Sinah-astro might want to split out another copy anyway with just the relevant scans to save her some processing time.

[bennahugo]

I opt for option 2

I'm not comfortable touching the vault - I'm not prepared to ask Sean to restage the multiple tapes for this observation.

Yes there are redundant bandpass scans - there were 8 fields co-observed over a very long track. Select the good ones and get rid of the rest.

On Tue, Jun 29, 2021 at 8:33 PM Oleg Smirnov @.***> wrote:

OK so:

-

The assumption of "one MS, one full set of fields" is baked deep into the workers. I agree this was short-sighted design, but it's going to take a bit of surgery to fix it. Let's leave this to R2.

The problem with concat, is presumably, clashing scan numbers so this can be worked around!

The problem with virtualconcat is not in fact POINTING, but the fact that the MSs have different descriptions (tiled data managers and all that bullsharcana). Presumably this can be worked around by using split to make a copy of them (sampling down to 8s in the process).

I have written a script to renumber scans in a series of MSs. See /net/oates/home/oms/sinah-test/m-orig/renumber_scans.py. Run it with all the MSs that need to be renumbered as arguments.

So let's go back to a single MS. Two solutions, from here:

Option 1:

-

@bennahugo https://github.com/bennahugo runs my script to renumber his original MSs in the vault (brrrr a little scary, right?)

@Sinah-astro https://github.com/Sinah-astro runs concat to copy-and-concatenate the relevant ones (using the original MSs as the sources)

Option 2:

-

@Sinah-astro https://github.com/Sinah-astro runs split to split out her copies of 8s data from the original vault MSs

@Sinah-astro https://github.com/Sinah-astro runs my script to renumber the scans in her 8s copies

@Sinah-astro https://github.com/Sinah-astro runs virtualconcat to concatenate her renumbered 8s copies into one multi-MS.

Note also that the result will have redundant calibrator scans (multiple bandpass scans at the start, multiple gaincal scans in a consecutive block) e.g.

Renumbering into 18 scans: 1: subset.1559937657.J1939-6342.ms scan 1 2: subset.1559937657.J1331+3030.ms scan 2 3: subset.1559937657.J1939-6342.ms scan 3 4: subset.1559937657.J1939-6342.ms scan 4 5: subset.1559937657.J1939-6342.ms scan 5 6: subset.1559937657.J1939-6342.ms scan 6 7: subset.1559937657.J1939-6342.ms scan 7 8: subset.J0203-4349.ms scan 8 9: subset.PSZ2G254.08-58.45.ms scan 9 10: subset.PSZ2G254.08-58.45.ms scan 10 11: subset.J0203-4349.ms scan 11 12: subset.J0203-4349.ms scan 12 13: subset.J0203-4349.ms scan 13 14: subset.1559937657.J1939-6342.ms scan 14 15: subset.J0203-4349.ms scan 15 16: subset.J0203-4349.ms scan 16 17: subset.J0203-4349.ms scan 17 18: subset.J0203-4349.ms scan 18

...so @Sinah-astro https://github.com/Sinah-astro might want to split out another copy anyway with just the relevant scans to save her some processing time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-870822631, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6SGFAJ7GAXMOK63F4DTVIGVZANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[o-smirnov]

Yeah I thought you'd say that. All right @Sinah-astro, option 2 it is, do you know what to do now?

[bennahugo]

Just a note why we opt for 8 seconds here - this is offline discussion between me and Oleg. There are no strong DD effects for your fields. You also don't have strong BCGs needing very high rate selfcal to improve DR. Therefore you can dump at 8s - 10s. You can also choose to average all the way down to 1K before proceeding to speed things up. The flagging may need a bit of tweaking on the spike size in channels etc. but should be ok throughout.

I needed the high spectro temporal resolution because I was not sure what impact DDEs will have had on our fields, but retrospectively we can proceed at a quicker pace.

On Tue, Jun 29, 2021 at 8:52 PM Benna Hugo @.***> wrote:

I opt for option 2

I'm not comfortable touching the vault - I'm not prepared to ask Sean to restage the multiple tapes for this observation.

Yes there are redundant bandpass scans - there were 8 fields co-observed over a very long track. Select the good ones and get rid of the rest.

On Tue, Jun 29, 2021 at 8:33 PM Oleg Smirnov @.***> wrote:

OK so:

-

The assumption of "one MS, one full set of fields" is baked deep into the workers. I agree this was short-sighted design, but it's going to take a bit of surgery to fix it. Let's leave this to R2.

The problem with concat, is presumably, clashing scan numbers so this can be worked around!

The problem with virtualconcat is not in fact POINTING, but the fact that the MSs have different descriptions (tiled data managers and all that bullsharcana). Presumably this can be worked around by using split to make a copy of them (sampling down to 8s in the process).

I have written a script to renumber scans in a series of MSs. See /net/oates/home/oms/sinah-test/m-orig/renumber_scans.py. Run it with all the MSs that need to be renumbered as arguments.

So let's go back to a single MS. Two solutions, from here:

Option 1:

-

@bennahugo https://github.com/bennahugo runs my script to renumber his original MSs in the vault (brrrr a little scary, right?)

@Sinah-astro https://github.com/Sinah-astro runs concat to copy-and-concatenate the relevant ones (using the original MSs as the sources)

Option 2:

-

@Sinah-astro https://github.com/Sinah-astro runs split to split out her copies of 8s data from the original vault MSs

@Sinah-astro https://github.com/Sinah-astro runs my script to renumber the scans in her 8s copies

@Sinah-astro https://github.com/Sinah-astro runs virtualconcat to concatenate her renumbered 8s copies into one multi-MS.

Note also that the result will have redundant calibrator scans (multiple bandpass scans at the start, multiple gaincal scans in a consecutive block) e.g.

Renumbering into 18 scans: 1: subset.1559937657.J1939-6342.ms scan 1 2: subset.1559937657.J1331+3030.ms scan 2 3: subset.1559937657.J1939-6342.ms scan 3 4: subset.1559937657.J1939-6342.ms scan 4 5: subset.1559937657.J1939-6342.ms scan 5 6: subset.1559937657.J1939-6342.ms scan 6 7: subset.1559937657.J1939-6342.ms scan 7 8: subset.J0203-4349.ms scan 8 9: subset.PSZ2G254.08-58.45.ms scan 9 10: subset.PSZ2G254.08-58.45.ms scan 10 11: subset.J0203-4349.ms scan 11 12: subset.J0203-4349.ms scan 12 13: subset.J0203-4349.ms scan 13 14: subset.1559937657.J1939-6342.ms scan 14 15: subset.J0203-4349.ms scan 15 16: subset.J0203-4349.ms scan 16 17: subset.J0203-4349.ms scan 17 18: subset.J0203-4349.ms scan 18

...so @Sinah-astro https://github.com/Sinah-astro might want to split out another copy anyway with just the relevant scans to save her some processing time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-870822631, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6SGFAJ7GAXMOK63F4DTVIGVZANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[o-smirnov]

Right, so for XXX in J1939-6342, J1331+3030, PSZ2G254.08-58.45.ms, run

split('/net/ike/vault-ike/hugo/meerkat_galaxy_cluster_safari/1559937657.XXX.ms', outputvis='XXX.ms', datacolumn='data', timebin='8s')

Then do the same for 1559937657.phasecals.ms with field='J0203-4349'.

Then run my renumbering script on the four resulting MSs. Make a note of the output scans so that you can remove duplicates later.

Then run virtualconcat (keepcopy=False) on the four (now scan-renumbered) MSs. You now have a working single MS, hopefully.

Optionally, run split on this to split out only the needed scans. But consult with us first about which ones to leave out.

Here's a copy of the renumbering script as a gist: https://gist.github.com/o-smirnov/95c19ae54125e98ba4d4fc71c0511d38

[bennahugo]

And another final note - please double check the TIME column and field id for the various different scans in the final virtualconcat database match the original TIME boundaries of the raw data - after what I've seen the weekend with virtualconcat it won't hurt to tripple check the indexes before proceeding with any calibration.

You can read about how to select data from this memo: https://casacore.github.io/casacore-notes/199.html and use numpy min max and set operations to double check that the scan and time boundaries are respected in the re-indexed virtualconcat database.

Cheers,

On Tue, Jun 29, 2021 at 9:08 PM Oleg Smirnov @.***> wrote:

Right, so for XXX in J1939-6342, J1331+3030, PSZ2G254.08-58.45.ms, run

split('/net/ike/vault-ike/hugo/meerkat_galaxy_cluster_safari/1559937657.XXX.ms', outputvis='XXX.ms', datacolumn='data', timebin='8s')

Then do the same for 1559937657.phasecals.ms with field='J0203-4349'.

Then run my renumbering script on the four resulting MSs. Make a note of the output scans so that you can remove duplicates later.

Then run virtualconcat (keepcopy=False) on the four (now scan-renumbered) MSs. You now have a working single MS, hopefully.

Optionally, run split on this to split out only the needed scans. But consult with us first about which ones to leave out.

Here's a copy of the renumbering script as a gist: https://gist.github.com/o-smirnov/95c19ae54125e98ba4d4fc71c0511d38

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-870844901, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6WEE2JIKE6T3JUPVJDTVIK3HANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[Sinah-astro]

Yeah I thought you'd say that. All right @Sinah-astro, option 2 it is, do you know what to do now?

Yes, I know what to do. I've already started with the splitting. Thank you all.

[o-smirnov]

There's a make-elevation-plots script in owlcat which makes plots like these (based on TIME and SCAN_NUMBER and FIELD_ID). Useful to check if things make sense.

Of course, you could also hold your breath and drop it into the ~meat grinder~ Caracal, as it runs this script to make this plot almost first thing.

[bennahugo]

Ok but my point is to be careful not to just rely on the indexes blindly - we need to establish whether the reindixing hasn't yet again combned different fields into the same scans as we had before. When we plotted fields before it appeared that two fields are combined into one (ie target

• calibrator) even though we selected explicitly on FIELD_ID from the MAIN table. you can get a nice elevation plot like the above and still have misindexing in the field id for instance.

edit: starting point again is to very carefully inspect the autocorrelation power as a function of time.

On Wed, Jun 30, 2021 at 11:12 AM Oleg Smirnov @.***> wrote:

There's a make-elevation-plots script in owlcat which makes plots like these (based on TIME and SCAN_NUMBER and FIELD_ID). Useful to check if things make sense.

[image: image] https://user-images.githubusercontent.com/6470079/123934587-b846ce00-d993-11eb-9414-5a28ce1f7a51.png

Of course, you could also hold your breath and drop it into the meat grinder Caracal, as it runs this script to make this plot almost first thing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-871231502, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6RYKGUM46FQUPOINFTTVLNXBANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[Sinah-astro]

I've split, renumbered scans and virtually concatenated the datasets. So running lisobs() resulted in the following: which concerns me as J0203-4349's scans are numbered as 30, 31, 32, 33, 34, 35, 36 and 37. Should I be concerned, or is that what the renumbering script was aiming for? The full casa log file: casa-20210630-091228.log

[o-smirnov]

Hmmm, doesn't look like the scans got renumbered, I see duplicates still. Could you run the renumbering script, post the output, then do listobs on a renumbered MS to make sure they are renumbered? Before concatenating.

[Sinah-astro]

I ran the scan renumbering on all four MSs and produced

[o-smirnov]

OK, looks sensible. @bennahugo, take a look please, does the order of the scans look sensible to you, and consistent with the original observation?

Scans 1, 3, 4, 5, 6 are anyway surplus to requirements so you can take them out at some point. But it probably won't hurt to leave them in either. (Famous last words...)

@Sinah-astro when sharing console output, can I discourage you from using screenshots? Rather, copy-and-paste the text directly between triple backquotes like so..

``` pasted text ```

...which comes out looking like so:

pasted text

...and is a lot easier for others to work with (bits of text can be selected and copied, for instance).

[bennahugo]

Can we also please compute the minimum and maximum of the TIME column, convert them to UTC through quanta and print that next to the scan - it is hard to tell if things are correct otherwise as I mentioned before. @sinah can you please add that?

On Thu, Jul 1, 2021 at 2:27 PM Oleg Smirnov @.***> wrote:

OK, looks sensible. @bennahugo https://github.com/bennahugo, take a look please, does the order of the scans look sensible to you, and consistent with the original observation?

Scans 1, 3, 4, 5, 6 are anyway surplus to requirements so you can take them out at some point. But it probably won't hurt to leave them in either. (Famous last words...)

@Sinah-astro https://github.com/Sinah-astro when sharing console output, can I discourage you from using screenshots? Rather, copy-and-paste the text directly between triple backquotes like so..

pasted text

...which comes out looking like so:

pasted text

...and is a lot easier for others to work with (bits of text can be selected and copied, for instance).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-872203934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6XBGLK3ZOF4XNX6HGTTVRNJ3ANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[bennahugo]

In my opinion on the first statement: the more bandpass the better - you can average all of it to wash out the residual RFI as much as possible after flagging, so I would leave them, but that is just me :)

On Thu, Jul 1, 2021 at 2:37 PM Benna Hugo @.***> wrote:

Can we also please compute the minimum and maximum of the TIME column, convert them to UTC through quanta and print that next to the scan - it is hard to tell if things are correct otherwise as I mentioned before. @sinah can you please add that?

On Thu, Jul 1, 2021 at 2:27 PM Oleg Smirnov @.***> wrote:

OK, looks sensible. @bennahugo https://github.com/bennahugo, take a look please, does the order of the scans look sensible to you, and consistent with the original observation?

Scans 1, 3, 4, 5, 6 are anyway surplus to requirements so you can take them out at some point. But it probably won't hurt to leave them in either. (Famous last words...)

@Sinah-astro https://github.com/Sinah-astro when sharing console output, can I discourage you from using screenshots? Rather, copy-and-paste the text directly between triple backquotes like so..

pasted text

...which comes out looking like so:

pasted text

...and is a lot easier for others to work with (bits of text can be selected and copied, for instance).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1350#issuecomment-872203934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6XBGLK3ZOF4XNX6HGTTVRNJ3ANCNFSM47CDTVKA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[Sinah-astro]

@Sinah-astro when sharing console output, can I discourage you from using screenshots? Rather, copy-and-paste the text directly between triple backquotes like so..

pasted text

...which comes out looking like so:

pasted text

...and is a lot easier for others to work with (bits of text can be selected and copied, for instance).

Very well noted +1:

Can we also please compute the minimum and maximum of the TIME column, convert them to UTC through quanta and print that next to the scan - it is hard to tell if things are correct otherwise as I mentioned before. @Sinah can you please add that?

@bennahugo I thought the time is UTC -- looking at the product of listobs. Isn't it?

[o-smirnov]

Yes listobs prints UTC time. But I think @bennahugo meant modifying my renumber_scans script to also print the times in UTC as the scans are being renumbered. Which I don't have time to do at the moment, so I leave it as an exercise to the student. ;)

Of course, you can also just renumber, virtualconcat, then listobs and post the scan times here for verification.

[Sinah-astro]

Product of the virtual concatenation

07-Jun-2019/20:02:26.4 - 20:07:28.3     1      0 J1939-6342               74214  [0]  [7.93] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                      20:09:00.2 - 20:29:01.7     2      1 J1331+3030              294903  [0]  [7.93] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  22:27:22.7 - 22:32:20.6     3      0 J1939-6342               74214  [0]  [7.83] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
      08-Jun-2019/00:33:59.5 - 00:38:57.4     4      0 J1939-6342               74214  [0]  [7.81] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  02:40:18.3 - 02:40:22.3     5      0 J1939-6342                1953  [0]  [4] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  02:41:02.3 - 02:46:00.1     6      0 J1939-6342               74214  [0]  [7.79] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  04:46:47.1 - 04:51:45.0     7      0 J1939-6342               74214  [0]  [7.81] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  05:15:30.3 - 05:17:58.3     8      2 J0203-4349               37107  [0]  [7.74] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  05:17:58.3 - 05:18:02.3     9      3 PSZ2G254.08-58.45         1953  [0]  [4] [TARGET]
                  05:18:20.3 - 05:38:21.8    10      3 PSZ2G254.08-58.45       294903  [0]  [7.94] [TARGET]
                  05:38:41.8 - 05:41:09.7    11      2 J0203-4349               37107  [0]  [7.74] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  05:41:09.7 - 05:41:13.7    12      3 PSZ2G254.08-58.45         1953  [0]  [4] [TARGET]
                  05:41:31.7 - 06:01:33.2    13      3 PSZ2G254.08-58.45       294903  [0]  [7.94] [TARGET]
                  06:01:53.2 - 06:04:23.1    14      2 J0203-4349               37107  [0]  [7.84] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  06:04:43.1 - 06:24:40.6    15      3 PSZ2G254.08-58.45       292950  [0]  [7.97] [TARGET]
                  06:25:00.6 - 06:27:28.5    16      2 J0203-4349               37107  [0]  [7.75] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  06:27:28.5 - 06:27:32.5    17      3 PSZ2G254.08-58.45         1953  [0]  [4] [TARGET]
                  06:27:50.5 - 06:47:48.0    18      3 PSZ2G254.08-58.45       292950  [0]  [7.96] [TARGET]
                  06:53:53.8 - 06:58:51.7    19      0 J1939-6342               74214  [0]  [7.81] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  06:58:51.7 - 06:58:55.7    20      2 J0203-4349                1953  [0]  [4] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  06:59:51.7 - 07:02:21.6    21      2 J0203-4349               37107  [0]  [7.87] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  07:02:43.6 - 07:22:39.1    22      3 PSZ2G254.08-58.45       292950  [0]  [7.97] [TARGET]
                  07:22:41.1 - 07:22:45.1    23      2 J0203-4349                1953  [0]  [3.98] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  07:23:11.1 - 07:25:41.0    24      2 J0203-4349               37107  [0]  [7.86] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  07:26:07.0 - 07:46:08.5    25      3 PSZ2G254.08-58.45       294903  [0]  [7.94] [TARGET]

Secondary calibrator

Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
      08-Jun-2019/05:15:30.3 - 05:17:58.3    30      0 J0203-4349              144522  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  05:38:41.8 - 05:41:09.7    31      0 J0203-4349              144522  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  06:01:53.2 - 06:04:23.1    32      0 J0203-4349              146475  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  06:25:00.6 - 06:27:28.5    33      0 J0203-4349              144522  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  06:58:51.7 - 06:58:55.7    34      0 J0203-4349                3906  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  06:59:51.7 - 07:02:21.6    35      0 J0203-4349              146475  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  07:22:41.1 - 07:22:45.1    36      0 J0203-4349                3906  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]
                  07:23:11.1 - 07:25:41.0    37      0 J0203-4349              146475  [0]  [2] [CALIBRATE_AMPLI,CALIBRATE_PHASE]

Target

Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
      08-Jun-2019/05:17:58.3 - 05:18:02.3     1      0 PSZ2G254.08-58.45         3906  [0]  [2] [TARGET]
                  05:18:20.3 - 05:38:21.8     2      0 PSZ2G254.08-58.45      1173753  [0]  [2] [TARGET]
                  05:41:09.7 - 05:41:13.7     3      0 PSZ2G254.08-58.45         3906  [0]  [2] [TARGET]
                  05:41:31.7 - 06:01:33.2     4      0 PSZ2G254.08-58.45      1173753  [0]  [2] [TARGET]
                  06:04:43.1 - 06:24:42.6     5      0 PSZ2G254.08-58.45      1171800  [0]  [2] [TARGET]
                  06:27:28.5 - 06:27:32.5     6      0 PSZ2G254.08-58.45         3906  [0]  [2] [TARGET]
                  06:27:50.5 - 06:47:50.0     7      0 PSZ2G254.08-58.45      1171800  [0]  [2] [TARGET]
                  07:02:43.6 - 07:22:41.1     8      0 PSZ2G254.08-58.45      1169847  [0]  [2] [TARGET]
                  07:26:07.0 - 07:46:08.5     9      0 PSZ2G254.08-58.45      1173753  [0]  [2] [TARGET]

Polarimetry Please explain again why we're using this source

Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
      07-Jun-2019/20:09:00.2 - 20:29:01.7     1      0 J1331+3030             1173753  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]

Primary calibrator

Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
      07-Jun-2019/20:02:26.4 - 20:07:28.3     1      0 J1939-6342              294903  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  22:27:22.7 - 22:32:20.6     2      0 J1939-6342              290997  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
      08-Jun-2019/00:33:59.5 - 00:38:57.4     3      0 J1939-6342              290997  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  02:40:18.3 - 02:40:22.3     4      0 J1939-6342                3906  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
              02:41:02.3 - 02:46:00.1     5      0 J1939-6342              290997  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  04:46:47.1 - 04:51:45.0     6      0 J1939-6342              290997  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]
                  06:53:53.8 - 06:58:51.7     7      0 J1939-6342              290997  [0]  [2] [CALIBRATE_BANDPASS,CALIBRATE_FLUX]

The times in the virtual concatenated dataset seem to correspond to the times in the raw datasets.

Can we also please compute the minimum and maximum of the TIME column Does this refer to the amount of time spent on each source?

[o-smirnov]

The concatenation product looks sensible to me, @bennahugo can you run your eye over it?

Those repeated bandpass scans at the start... I guess it's really a question of personal faith whether we want to keep them or not. @bennahugo likes them because they increase the SNR in the bandpass solution. I think that if the bandpass is slightly time-variable, the returns will diminish rapidly. Arguing this without trying is all navel gazing though. Go forth and praxis.

The pol calibrator is going to be needed for polarization calibration (as the xcal), if and when you get to looking at polarimetry.

Does this refer to the amount of time spent on each source?

@bennahugo meant the min/max of the time column subset in the renumber_scans script. If everything is correct, this should be the same as the "Timerange" shown by listobs. But looking at your listobs output, it looks correct already, so I think we can forget about this action item.

[Sinah-astro]

Thank you for your response @o-smirnov

I'm looking at the autocorrelation plots and found that some scans are flagged when spw=0:1500~2000 and uvrange = <1m. Also the common thing is their SpwIds are less than or equal to 4. Why is this so?