Closed paoloserra closed 1 year ago
The relevant lines in the transform worker are:
and then
Everything non-pol works if the latter is replaced with
output_fields=[''],
But I suppose that would break polcal. @francescaLoi @o-smirnov could you please comment on this?
Alternatively, I suppose there is a callib-friendly way of changing L113 above to include all targets of the MS in output_fields
instead of just the first one?
It might be as simple as moving the resolve_calibration_library
call into this loop here:
where we already have a single target.
Indeed, I considered that, but I have no good understanding of what's needed for polcal. For regular calibration the solution above works fine, too, and it's simpler.
I have been puzzling about exactly this for the last day. I'll give the workaround suggested by Paolo a try.
@Athanaseus you also want to take a look at this.
For regular calibration the solution above works fine, too, and it's simpler.
Simpler doesn't always mean more correct. :) Setting output_fields=['']
means "return cal tables appropriate for default targets". This should be the same as specifying a target by name. However, the point of the resolve_calibration_library()
interface is to let that function decide what tables are most appropriate for a particular field. I'm thinking about future support of cases where we have several gain calibrators and several targets in the MS.
understood
Hm, this did not work for me I fear. I replaced in line 133 as suggested by Paolo: output_fields=['']. Then I deleted the corrected target field measurement sets and I continued caracal from the transform-target step on. Is this not sufficient and I need to re-run the whole thing?
This is what the transform-step logs for the otf-applycal:
MSTransformManager::generateIterator OTF calibration activated, using calibration record spec to generate iterator
Calibrater:: Arranging to calibrate MS: /stimela_mount/input/1662797070_sdp_l0.ms
OldCalibrater::setcallib2(callib) Arranging to APPLY:
. B Jones: table=/stimela_mount/output/caltables/mcvs-1662797070_sdp_l0-kgb_primary.B1 (by cal library) calWt=false
. 0:
+ MS: obs= fld= intent= spw=
+ CT: tinterp=linear finterp=linear
+ obsmap=[] fldmap=[1, 1, 1, 1, 1, 1, 1, 1]
+ spwmap=[] antmap=[]
OldCalibrater::setcallib2(callib) Arranging to APPLY:
. K Jones: table=/stimela_mount/output/caltables/mcvs-1662797070_sdp_l0-kgb_primary.K1 (by cal library) calWt=false
. 0:
+ MS: obs= fld= intent= spw=
+ CT: tinterp=nearest finterp=linear
+ obsmap=[] fldmap=[1, 1, 1, 1, 1, 1, 1, 1]
+ spwmap=[] antmap=[]
OldCalibrater::setcallib2(callib) Arranging to APPLY:
. G Jones: table=/stimela_mount/output/caltables/mcvs-1662797070_sdp_l0-kgb_secondary.F0 (by cal library) calWt=false
. 0:
+ MS: obs= fld= intent= spw=
+ CT: tinterp=linear finterp=linear
+ obsmap=[] fldmap=[0, 0, 0, 0, 0, 0, 0, 1]
+ spwmap=[] antmap=[]
OldCalibrater::setcallib2(callib) The following calibration terms are arranged for apply:
OldCalibrater::setcallib2(callib) . B Jones: table=/stimela_mount/output/caltables/mcvs-1662797070_sdp_l0-kgb_primary.B1 (by cal library) calWt=false
OldCalibrater::setcallib2(callib) . K Jones: table=/stimela_mount/output/caltables/mcvs-1662797070_sdp_l0-kgb_primary.K1 (by cal library) calWt=false
OldCalibrater::setcallib2(callib) . G Jones: table=/stimela_mount/output/caltables/mcvs-1662797070_sdp_l0-kgb_secondary.F0 (by cal library) calWt=false
Should I be able to see if things are going wrong here? "Field 1" is the one I want to correct.
Can you post the full log?
Sure, here you go: log-caracal.txt
Also, if you can post the calibration libraries please, i.e. the .txt and .yml files under caltables
in your output directory.
Ah damn, I deleted those in the meantime, sorry. I'll post them once I am at the step again.
Shame.
I deleted the corrected target field measurement sets and I continued caracal from the transform-target step on. Is this not sufficient and I need to re-run the whole thing?
That is fine. You don't need to re-run crosscal. In fact, you don't even need to delete the corrected MS, CARACal will take care of that.
I'm a little confused by your log. The initial transform and crosscal are enabled but are not running?
Anyway, the OTFcal part of the log looks fine. Have you looked at the first continuum image? Does that still show those scrambled phases?
BTW, in case you had not realised, you could leave the target unset in the yml file, and CARACal will loop through all targets.
I'm a little confused by your log. The initial transform and crosscal are enabled but are not running?
Yes, I used the flag --sw transform__2
to continue running from the troublesome step on after doing the change to the code.
Anyway, the OTFcal part of the log looks fine. Have you looked at the first continuum image? Does that still show those scrambled phases?
Yes, it looks like the phases are crazy, like before.
OK.
The problem I was having should show as in the example below:
Here's the OTF applycal log for the first target in the MS, where the calibration is applied alright and the phases look fine.
# 2022-11-08 10:13:02 INFO mstransform::::+ ##########################################
# 2022-11-08 10:13:02 INFO mstransform::::+ ##### Begin Task: mstransform #####
# 2022-11-08 10:13:02 INFO mstransform:::: mstransform(vis="/stimela_mount/input/1662968476_sdp_l0.ms",outputvis="/stimela_mount/msdir/1662968476_sdp_l0-virgo065-corr.ms",createmms=False,separationaxis="auto",numsubms="auto",
# 2022-11-08 10:13:02 INFO mstransform::::+ tileshape=[0],field="virgo065",spw="",scan="",antenna="",
# 2022-11-08 10:13:02 INFO mstransform::::+ correlation="",timerange="",intent="",array="",uvrange="",
# 2022-11-08 10:13:02 INFO mstransform::::+ observation="",feed="",datacolumn="corrected",realmodelcol=False,keepflags=True,
# 2022-11-08 10:13:02 INFO mstransform::::+ usewtspectrum=True,combinespws=False,chanaverage=False,chanbin=1,hanning=False,
# 2022-11-08 10:13:02 INFO mstransform::::+ regridms=False,mode="channel",nchan=-1,start=0,width=1,
# 2022-11-08 10:13:02 INFO mstransform::::+ nspw=1,interpolation="linear",phasecenter="",restfreq="",outframe="",
# 2022-11-08 10:13:02 INFO mstransform::::+ veltype="radio",preaverage=False,timeaverage=False,timebin="",timespan="",
# 2022-11-08 10:13:02 INFO mstransform::::+ maxuvwdistance=0.0,docallib=True,callib="/stimela_mount/output/caltables/callib-oct-1662968476_sdp_l0-1gc-transform.txt",douvcontsub=False,fitspw="",
# 2022-11-08 10:13:02 INFO mstransform::::+ fitorder=0,want_cont=False,denoising_lib=True,nthreads=1,niter=1,
# 2022-11-08 10:13:02 INFO mstransform::::+ disableparallel=False,ddistart=-1,taql="",monolithic_processing=False,reindex=True)
# 2022-11-08 10:13:02 INFO mstransform:::: Parse docallib parameters
# 2022-11-08 10:13:02 INFO MSTransformManager::parseMsSpecParams Input file name is /stimela_mount/input/1662968476_sdp_l0.ms
# 2022-11-08 10:13:02 INFO MSTransformManager::parseMsSpecParams Data column is CORRECTED
# 2022-11-08 10:13:02 INFO MSTransformManager::parseMsSpecParams Output file name is /stimela_mount/msdir/1662968476_sdp_l0-virgo065-corr.ms
# 2022-11-08 10:13:02 INFO MSTransformManager::parseMsSpecParams Re-index is enabled
# 2022-11-08 10:13:02 INFO MSTransformManager::parseMsSpecParams WEIGHT_SPECTRUM will be written in output MS
# 2022-11-08 10:13:02 INFO MSTransformManager::parseMsSpecParams Tile shape is [0]
# 2022-11-08 10:13:02 INFO MSTransformManager::parseDataSelParams field selection is virgo065
# 2022-11-08 10:13:02 INFO MSTransformManager::parseCalParams Calibration is activated
# 2022-11-08 10:13:02 INFO MSTransformManager::colCheckInfo Adding DATA column to output MS from input virtual CORRECTED_DATA column
# 2022-11-08 10:13:02 INFO MSTransformManager::initDataSelectionParams Selected Fields Ids are [1]
# 2022-11-08 10:13:02 INFO MSTransformManager::open Select data
# 2022-11-08 10:13:02 INFO MSTransformManager::createOutputMSStructure Create output MS structure
# 2022-11-08 10:13:07 INFO MSTransformDataHandler::makeSelection 552240 out of 3279810 rows are going to be considered due to the selection criteria.
# 2022-11-08 10:13:08 INFO MSTransformManager::checkFillWeightSpectrum Optional column WEIGHT_SPECTRUM found in input MS will be written to output MS
# 2022-11-08 10:13:08 INFO MSTransformManager::generateIterator OTF calibration activated, using calibration record spec to generate iterator
# 2022-11-08 10:13:09 INFO Calibrater:: Arranging to calibrate MS: /stimela_mount/input/1662968476_sdp_l0.ms
# 2022-11-08 10:13:09 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
# 2022-11-08 10:13:09 INFO . B Jones: table=/stimela_mount/output/caltables/oct-1662968476_sdp_l0-1gc_primary.B1 (by cal library) calWt=false
# 2022-11-08 10:13:09 INFO . 0:
# 2022-11-08 10:13:09 INFO + MS: obs= fld=virgo065 intent= spw=
# 2022-11-08 10:13:09 INFO + CT: tinterp=linear finterp=linear
# 2022-11-08 10:13:09 INFO + obsmap=[] fldmap=[1, 1, 1, 1, 1, 1, 1, 1]
# 2022-11-08 10:13:09 INFO + spwmap=[] antmap=[]
...
And here is the case of the second target, where you see that OTF applycal fails because of the mismatch in the field.
# 2022-11-08 10:17:20 INFO mstransform::::+ ##########################################
# 2022-11-08 10:17:20 INFO mstransform::::+ ##### Begin Task: mstransform #####
# 2022-11-08 10:17:20 INFO mstransform:::: mstransform(vis="/stimela_mount/input/1662968476_sdp_l0.ms",outputvis="/stimela_mount/msdir/1662968476_sdp_l0-virgo044-corr.ms",createmms=False,separationaxis="auto",numsubms="auto",
# 2022-11-08 10:17:20 INFO mstransform::::+ tileshape=[0],field="virgo044",spw="",scan="",antenna="",
# 2022-11-08 10:17:20 INFO mstransform::::+ correlation="",timerange="",intent="",array="",uvrange="",
# 2022-11-08 10:17:20 INFO mstransform::::+ observation="",feed="",datacolumn="corrected",realmodelcol=False,keepflags=True,
# 2022-11-08 10:17:20 INFO mstransform::::+ usewtspectrum=True,combinespws=False,chanaverage=False,chanbin=1,hanning=False,
# 2022-11-08 10:17:20 INFO mstransform::::+ regridms=False,mode="channel",nchan=-1,start=0,width=1,
# 2022-11-08 10:17:20 INFO mstransform::::+ nspw=1,interpolation="linear",phasecenter="",restfreq="",outframe="",
# 2022-11-08 10:17:20 INFO mstransform::::+ veltype="radio",preaverage=False,timeaverage=False,timebin="",timespan="",
# 2022-11-08 10:17:20 INFO mstransform::::+ maxuvwdistance=0.0,docallib=True,callib="/stimela_mount/output/caltables/callib-oct-1662968476_sdp_l0-1gc-transform.txt",douvcontsub=False,fitspw="",
# 2022-11-08 10:17:20 INFO mstransform::::+ fitorder=0,want_cont=False,denoising_lib=True,nthreads=1,niter=1,
# 2022-11-08 10:17:20 INFO mstransform::::+ disableparallel=False,ddistart=-1,taql="",monolithic_processing=False,reindex=True)
# 2022-11-08 10:17:20 INFO mstransform:::: Parse docallib parameters
# 2022-11-08 10:17:20 INFO MSTransformManager::parseMsSpecParams Input file name is /stimela_mount/input/1662968476_sdp_l0.ms
# 2022-11-08 10:17:20 INFO MSTransformManager::parseMsSpecParams Data column is CORRECTED
# 2022-11-08 10:17:20 INFO MSTransformManager::parseMsSpecParams Output file name is /stimela_mount/msdir/1662968476_sdp_l0-virgo044-corr.ms
# 2022-11-08 10:17:20 INFO MSTransformManager::parseMsSpecParams Re-index is enabled
# 2022-11-08 10:17:20 INFO MSTransformManager::parseMsSpecParams WEIGHT_SPECTRUM will be written in output MS
# 2022-11-08 10:17:20 INFO MSTransformManager::parseMsSpecParams Tile shape is [0]
# 2022-11-08 10:17:20 INFO MSTransformManager::parseDataSelParams field selection is virgo044
# 2022-11-08 10:17:20 INFO MSTransformManager::parseCalParams Calibration is activated
# 2022-11-08 10:17:20 INFO MSTransformManager::colCheckInfo Adding DATA column to output MS from input virtual CORRECTED_DATA column
# 2022-11-08 10:17:20 INFO MSTransformManager::initDataSelectionParams Selected Fields Ids are [2]
# 2022-11-08 10:17:20 INFO MSTransformManager::open Select data
# 2022-11-08 10:17:20 INFO MSTransformManager::createOutputMSStructure Create output MS structure
# 2022-11-08 10:17:20 INFO MSTransformDataHandler::makeSelection 554010 out of 3279810 rows are going to be considered due to the selection criteria.
# 2022-11-08 10:17:21 INFO MSTransformManager::checkFillWeightSpectrum Optional column WEIGHT_SPECTRUM found in input MS will be written to output MS
# 2022-11-08 10:17:21 INFO MSTransformManager::generateIterator OTF calibration activated, using calibration record spec to generate iterator
# 2022-11-08 10:17:22 INFO Calibrater:: Arranging to calibrate MS: /stimela_mount/input/1662968476_sdp_l0.ms
# 2022-11-08 10:17:22 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
# 2022-11-08 10:17:22 INFO . B Jones: table=/stimela_mount/output/caltables/oct-1662968476_sdp_l0-1gc_primary.B1 (by cal library) calWt=false
# 2022-11-08 10:17:22 WARN . The following callib entry matches no data
# 2022-11-08 10:17:22 WARN + . in the selected MS (1662968476_sdp_l0.ms) and will be ignored:
# 2022-11-08 10:17:22 WARN + . 0:
# 2022-11-08 10:17:22 WARN + MS: obs= fld=virgo065 intent= spw=
# 2022-11-08 10:17:22 WARN + CT: tinterp=linear finterp=linear
# 2022-11-08 10:17:22 WARN + obsmap=[] fldmap=[1, 1, 1, 1, 1, 1, 1, 1]
# 2022-11-08 10:17:22 WARN + spwmap=[] antmap=[]
...
If you don't see the warning above then OTF applycal is fine, at least as far as the bug reported in this issue goes.
Ok, indeed this warning I did not encounter before. Here the calibration libraries now: callib-mcvs-1662797070_sdp_l0-kgb.yml.txt callib-mcvs-1662797070_sdp_l0-kgb-transform__2.txt
Looks fine to me -- I don't think OTF is your problem in this case.
I don't think so either.
Do you see something like this for virgo048? (It's the dirty image.)
This is what I get as virgo048_0-MFS-image, you can see the artifacts of M87 but not really the source itself (FoV: 1.5x1.5deg). This is, as expected, very similar to imaging the data-column of the virgo048-corrected.MS directly after the transform__2 step. Similar results I also get for the other field I tried.
And this is what I get as image of the secondary calibrator, this still looks fine to me.
It could be that something went wrong in the pipeline by starting/re-starting and changing some of the parameters in-between (e.g. using all targets / just virgo048), or, alternatively, it is related to using the "nyq" method for the SPECTRAL_WEIGHTS (see https://github.com/caracal-pipeline/caracal/issues/1451#issuecomment-1302200235).
Do you have in mind any tests I could to to pin down the problem? Thanks a lot for the help already.
Could you please try field 113? It's an easier field.
This is for field 129, most distant from M87, still the same problem.
I also tried virgo065 from the second pilot, again with the same messed up phases. After a few iterations, sources indeed start to appear, so the data is probably fine but the initial phases are quite off.
There's definitely something wrong there.
This is what I get for field 129 in the freq range 1395 - 1437 MHz:
after cross-cal
after self-cal
It's not wonderful (and I don't need it to be), but it looks right.
Indeed ever so slightly better than what I am getting ;). Would you mind sharing your parameter set with me so I can compare them? And you are using the current master (except for the one change you described above)?
I'm on this branch: https://github.com/caracal-pipeline/caracal/tree/callib-multitarget (but yes, it's just that one change compared to master)
.yml files attached. I run them in the order:
I ran a test where I just applied the K and B solutions, and not the gaincal. Indeed, there I get a reasonable image (below: dirty/restored). So the point where things go wrong for me should be quite constrained. One thing I was wondering - for the gaincal solutions, I always see a "fldmap" that looks like this: fldmap=[0, 0, 0, 0, 0, 0, 0, 1]
. I wonder what that means, shouldn't this all be the same index (the one of the gaincal?). However, I see the same also in an archival MeerKat observation where the calibration works fine.
I'm not sure about fldmap. Have you looked at the gain plots? (G and F)
I think the last entry in fldmap
corresponds to what is used for the bpcal, so it uses itself rather than the gaincal.
For the record: I pinned down the issue, it was not related to caracal. I assumed the data set I had at hand was the raw data, when actually the calibrator solutions were already applied by the observatory (only to the target field, not to the cals, so the solutions appeared normal). So basically I was "corrupting" the already corrected data by correcting them a second time...
I re-downloaded the raw data, now running caracal gives the expected results.
phew!
Awesome! Now I wish someone would solve https://github.com/caracal-pipeline/caracal/issues/1456 :)
Well, we still need to solve #1454 too!!!! :)
@paoloserra can you please try the fix in branch apply_multiple_targets
.
Log section of mstransform now looks like:
2022-12-06 07:40:46 INFO mstransform:::: mstransform(vis="/stimela_mount/input/1543664600_sdp_l0.ms",outputvis="/stimela_mount/msdir/1543664600_sdp_l0-T16R02C04-corr.ms",createmms=False,separationaxis="auto",numsubms="auto",
2022-12-06 07:40:46 INFO mstransform::::+ tileshape=[0],field="T16R02C04",spw="",scan="",antenna="",
2022-12-06 07:40:46 INFO mstransform::::+ correlation="",timerange="",intent="",array="",uvrange="",
2022-12-06 07:40:46 INFO mstransform::::+ observation="",feed="",datacolumn="corrected",realmodelcol=False,keepflags=True,
2022-12-06 07:40:46 INFO mstransform::::+ usewtspectrum=True,combinespws=False,chanaverage=False,chanbin=1,hanning=False,
2022-12-06 07:40:46 INFO mstransform::::+ regridms=False,mode="channel",nchan=-1,start=0,width=1,
2022-12-06 07:40:46 INFO mstransform::::+ nspw=1,interpolation="linear",phasecenter="",restfreq="",outframe="",
2022-12-06 07:40:46 INFO mstransform::::+ veltype="radio",preaverage=False,timeaverage=False,timebin="",timespan="",
2022-12-06 07:40:46 INFO mstransform::::+ maxuvwdistance=0.0,docallib=True,callib="/stimela_mount/output/caltables/callib-mgps-hii-regions-1543664600-ar2-1543664600_sdp_l0-1kgb-transform__2.txt",douvcontsub=False,fitspw="",
2022-12-06 07:40:46 INFO mstransform::::+ fitorder=0,want_cont=False,denoising_lib=True,nthreads=1,niter=1,
2022-12-06 07:40:46 INFO mstransform::::+ disableparallel=False,ddistart=-1,taql="",monolithic_processing=False,reindex=True)
2022-12-06 07:40:46 INFO mstransform:::: Parse docallib parameters
2022-12-06 07:40:46 INFO MSTransformManager::parseMsSpecParams Input file name is /stimela_mount/input/1543664600_sdp_l0.ms
2022-12-06 07:40:46 INFO MSTransformManager::parseMsSpecParams Data column is CORRECTED
2022-12-06 07:40:46 INFO MSTransformManager::parseMsSpecParams Output file name is /stimela_mount/msdir/1543664600_sdp_l0-T16R02C04-corr.ms
2022-12-06 07:40:46 INFO MSTransformManager::parseMsSpecParams Re-index is enabled
2022-12-06 07:40:46 INFO MSTransformManager::parseMsSpecParams WEIGHT_SPECTRUM will be written in output MS
2022-12-06 07:40:46 INFO MSTransformManager::parseMsSpecParams Tile shape is [0]
2022-12-06 07:40:46 INFO MSTransformManager::parseDataSelParams field selection is T16R02C04
2022-12-06 07:40:46 INFO MSTransformManager::parseCalParams Calibration is activated
2022-12-06 07:40:46 INFO MSTransformManager::colCheckInfo Adding DATA column to output MS from input virtual CORRECTED_DATA column
2022-12-06 07:40:46 INFO MSTransformManager::initDataSelectionParams Selected Fields Ids are [3]
2022-12-06 07:40:46 INFO MSTransformManager::open Select data
2022-12-06 07:40:46 INFO MSTransformManager::createOutputMSStructure Create output MS structure
2022-12-06 07:40:46 INFO MSTransformDataHandler::makeSelection 249612 out of 3496459 rows are going to be considered due to the selection criteria.
2022-12-06 07:40:46 INFO MSTransformManager::checkFillWeightSpectrum Optional column WEIGHT_SPECTRUM found in input MS will be written to output MS
2022-12-06 07:40:46 INFO MSTransformManager::generateIterator OTF calibration activated, using calibration record spec to generate iterator
2022-12-06 07:40:47 INFO Calibrater:: Arranging to calibrate MS: /stimela_mount/input/1543664600_sdp_l0.ms
2022-12-06 07:40:47 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
2022-12-06 07:40:47 INFO . B Jones: table=/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.B1 (by cal library) calWt=false
2022-12-06 07:40:47 INFO . 0:
2022-12-06 07:40:47 INFO + MS: obs= fld=T16R02C02,T16R04C04,T16R03C01,T16R04C02,T16R04C06,T16R03C03,T16R02C04,T16R02C06,T16R03C05 intent= spw=
2022-12-06 07:40:47 INFO + CT: tinterp=linear finterp=linear
2022-12-06 07:40:47 INFO + obsmap=[] fldmap=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2022-12-06 07:40:47 INFO + spwmap=[] antmap=[]
2022-12-06 07:40:47 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
2022-12-06 07:40:47 INFO . K Jones: table=/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.K2 (by cal library) calWt=false
2022-12-06 07:40:47 INFO . 0:
2022-12-06 07:40:47 INFO + MS: obs= fld=T16R02C02,T16R04C04,T16R03C01,T16R04C02,T16R04C06,T16R03C03,T16R02C04,T16R02C06,T16R03C05 intent= spw=
2022-12-06 07:40:47 INFO + CT: tinterp=nearest finterp=linear
2022-12-06 07:40:47 INFO + obsmap=[] fldmap=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2022-12-06 07:40:47 INFO + spwmap=[] antmap=[]
2022-12-06 07:40:47 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
2022-12-06 07:40:47 INFO . G Jones: table=/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_secondary.F0 (by cal library) calWt=false
2022-12-06 07:40:47 INFO . 0:
2022-12-06 07:40:47 INFO + MS: obs= fld=T16R02C02,T16R04C04,T16R03C01,T16R04C02,T16R04C06,T16R03C03,T16R02C04,T16R02C06,T16R03C05 intent= spw=
NB: fld list all targets
And previously, it only listed index 0
target field only.
2022-11-15 09:11:13 INFO mstransform:::: mstransform(vis="/stimela_mount/input/1543664600_sdp_l0.ms",outputvis="/stimela_mount/msdir/1543664600_sdp_l0-T16R02C04-corr.ms",createmms=False,separationaxis="auto",numsubms="auto",
2022-11-15 09:11:13 INFO mstransform::::+ tileshape=[0],field="T16R02C04",spw="",scan="",antenna="",
2022-11-15 09:11:13 INFO mstransform::::+ correlation="",timerange="",intent="",array="",uvrange="",
2022-11-15 09:11:13 INFO mstransform::::+ observation="",feed="",datacolumn="corrected",realmodelcol=False,keepflags=True,
2022-11-15 09:11:13 INFO mstransform::::+ usewtspectrum=True,combinespws=False,chanaverage=False,chanbin=1,hanning=False,
2022-11-15 09:11:13 INFO mstransform::::+ regridms=False,mode="channel",nchan=-1,start=0,width=1,
2022-11-15 09:11:13 INFO mstransform::::+ nspw=1,interpolation="linear",phasecenter="",restfreq="",outframe="",
2022-11-15 09:11:13 INFO mstransform::::+ veltype="radio",preaverage=False,timeaverage=False,timebin="",timespan="",
2022-11-15 09:11:13 INFO mstransform::::+ maxuvwdistance=0.0,docallib=True,callib="/stimela_mount/output/caltables/callib-mgps-hii-regions-1543664600-ar2-1543664600_sdp_l0-1kgb-transform__2.txt",douvcontsub=False,fitspw="",
2022-11-15 09:11:13 INFO mstransform::::+ fitorder=0,want_cont=False,denoising_lib=True,nthreads=1,niter=1,
2022-11-15 09:11:13 INFO mstransform::::+ disableparallel=False,ddistart=-1,taql="",monolithic_processing=False,reindex=True)
2022-11-15 09:11:13 INFO mstransform:::: Parse docallib parameters
2022-11-15 09:11:13 INFO MSTransformManager::parseMsSpecParams Input file name is /stimela_mount/input/1543664600_sdp_l0.ms
2022-11-15 09:11:13 INFO MSTransformManager::parseMsSpecParams Data column is CORRECTED
2022-11-15 09:11:13 INFO MSTransformManager::parseMsSpecParams Output file name is /stimela_mount/msdir/1543664600_sdp_l0-T16R02C04-corr.ms
2022-11-15 09:11:13 INFO MSTransformManager::parseMsSpecParams Re-index is enabled
2022-11-15 09:11:13 INFO MSTransformManager::parseMsSpecParams WEIGHT_SPECTRUM will be written in output MS
2022-11-15 09:11:13 INFO MSTransformManager::parseMsSpecParams Tile shape is [0]
2022-11-15 09:11:13 INFO MSTransformManager::parseDataSelParams field selection is T16R02C04
2022-11-15 09:11:13 INFO MSTransformManager::parseCalParams Calibration is activated
2022-11-15 09:11:13 INFO MSTransformManager::colCheckInfo Adding DATA column to output MS from input virtual CORRECTED_DATA column
2022-11-15 09:11:13 INFO MSTransformManager::initDataSelectionParams Selected Fields Ids are [3]
2022-11-15 09:11:13 INFO MSTransformManager::open Select data
2022-11-15 09:11:13 INFO MSTransformManager::createOutputMSStructure Create output MS structure
2022-11-15 09:11:14 INFO MSTransformDataHandler::makeSelection 249612 out of 3496459 rows are going to be considered due to the selection criteria.
2022-11-15 09:11:14 INFO MSTransformManager::checkFillWeightSpectrum Optional column WEIGHT_SPECTRUM found in input MS will be written to output MS
2022-11-15 09:11:14 INFO MSTransformManager::generateIterator OTF calibration activated, using calibration record spec to generate iterator
2022-11-15 09:11:15 INFO Calibrater:: Arranging to calibrate MS: /stimela_mount/input/1543664600_sdp_l0.ms
2022-11-15 09:11:15 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
2022-11-15 09:11:15 INFO . B Jones: table=/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.B1 (by cal library) calWt=false
2022-11-15 09:11:15 WARN . The following callib entry matches no data
2022-11-15 09:11:15 WARN + . in the selected MS (1543664600_sdp_l0.ms) and will be ignored:
2022-11-15 09:11:15 WARN + . 0:
2022-11-15 09:11:15 WARN + MS: obs= fld=T16R02C02 intent= spw=
2022-11-15 09:11:15 WARN + CT: tinterp=linear finterp=linear
2022-11-15 09:11:15 WARN + obsmap=[] fldmap=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2022-11-15 09:11:15 WARN + spwmap=[] antmap=[]
2022-11-15 09:11:15 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
2022-11-15 09:11:15 INFO . K Jones: table=/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.K2 (by cal library) calWt=false
2022-11-15 09:11:15 WARN . The following callib entry matches no data
2022-11-15 09:11:15 WARN + . in the selected MS (1543664600_sdp_l0.ms) and will be ignored:
2022-11-15 09:11:15 WARN + . 0:
2022-11-15 09:11:15 WARN + MS: obs= fld=T16R02C02 intent= spw=
2022-11-15 09:11:15 WARN + CT: tinterp=nearest finterp=linear
2022-11-15 09:11:15 WARN + obsmap=[] fldmap=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2022-11-15 09:11:15 WARN + spwmap=[] antmap=[]
2022-11-15 09:11:15 INFO OldCalibrater::setcallib2(callib) Arranging to APPLY:
2022-11-15 09:11:15 INFO . G Jones: table=/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_secondary.F0 (by cal library) calWt=false
2022-11-15 09:11:15 WARN . The following callib entry matches no data
new calib:
caltable="/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_secondary.F0" calwt=False tinterp='linear' finterp='linear' fldmap='nearest' field='T16R02C02,T16R04C04,T16R03C01,T16R04C02,T16R04C06,T16R03C03,T16R02C04,T16R02C06,T16R03C05' spwmap=0
caltable="/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.B1" calwt=False tinterp='linear' finterp='linear' fldmap='nearest' field='T16R02C02,T16R04C04,T16R03C01,T16R04C02,T16R04C06,T16R03C03,T16R02C04,T16R02C06,T16R03C05' spwmap=0
caltable="/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.K2" calwt=False tinterp='nearest' finterp='linear' fldmap='nearest' field='T16R02C02,T16R04C04,T16R03C01,T16R04C02,T16R04C06,T16R03C03,T16R02C04,T16R02C06,T16R03C05' spwmap=0
old calib:
caltable="/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_secondary.F0" calwt=False tinterp='linear' finterp='linear' fldmap='nearest' field='T16R02C02' spwmap=0
caltable="/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.B1" calwt=False tinterp='linear' finterp='linear' fldmap='nearest' field='T16R02C02' spwmap=0
caltable="/stimela_mount/output/caltables/mgps-hii-regions-ar4-1543664600_sdp_l0-1kgb_primary.K2" calwt=False tinterp='nearest' finterp='linear' fldmap='nearest' field='T16R02C02' spwmap=0
Originally posted by @francescaLoi in https://github.com/caracal-pipeline/caracal/issues/1280#issuecomment-726142966
Since this we have been adding the field in the callib txt file. That works when you've got one field only. It does not when you have more than one, because we only write the first field.