mstransform worker can't handle whitespace

[AstroRipples]

As the title suggests, I'm encountering issues when trying to run CARACal on a dataset that has a space in the target field name. In this instance, I'm processing some MGCLS data from scratch, performing continuum calibration only. The primary & secondary calibration has been successful so now I'm splitting and processing the target data, and I'm getting the following error (full traceback below):

# 2023-03-07 13:50:42   INFO    mstransform::::+        ##########################################
# 2023-03-07 13:50:42   INFO    mstransform::::+        ##### Begin Task: mstransform        #####
# 2023-03-07 13:50:42   INFO    mstransform:::: mstransform(vis="/stimela_mount/input/1551621148.MS",outputvis="/stimela_mount/msdir/1551621148-Abell_3667-corr.MS",createmms=False,separationaxis="auto",numsubms="auto",
# 2023-03-07 13:50:42   INFO    mstransform::::+                tileshape=[0],field="Abell 3667",spw="",scan="",antenna="",
# 2023-03-07 13:50:42   INFO    mstransform::::+                correlation="",timerange="",intent="",array="",uvrange="",
# 2023-03-07 13:50:42   INFO    mstransform::::+                observation="",feed="",datacolumn="corrected",realmodelcol=False,keepflags=True,
# 2023-03-07 13:50:42   INFO    mstransform::::+                usewtspectrum=True,combinespws=False,chanaverage=False,chanbin=1,hanning=False,
# 2023-03-07 13:50:42   INFO    mstransform::::+                regridms=False,mode="channel",nchan=-1,start=0,width=1,
# 2023-03-07 13:50:42   INFO    mstransform::::+                nspw=1,interpolation="linear",phasecenter="",restfreq="",outframe="",
# 2023-03-07 13:50:42   INFO    mstransform::::+                veltype="radio",preaverage=False,timeaverage=False,timebin="",timespan="",
# 2023-03-07 13:50:42   INFO    mstransform::::+                maxuvwdistance=0.0,docallib=True,callib="/stimela_mount/output/caltables/callib-PipelineRun_Continuum-1551621148-2kgb-transform__2.txt",douvcontsub=False,fitspw="",
# 2023-03-07 13:50:42   INFO    mstransform::::+                fitorder=0,want_cont=False,denoising_lib=True,nthreads=1,niter=1,
# 2023-03-07 13:50:42   INFO    mstransform::::+                disableparallel=False,ddistart=-1,taql="",monolithic_processing=False,reindex=True)
# 2023-03-07 13:50:42   INFO    mstransform:::: Parse docallib parameters
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseMsSpecParams   Input file name is /stimela_mount/input/1551621148.MS
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseMsSpecParams   Data column is CORRECTED
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseMsSpecParams   Output file name is /stimela_mount/msdir/1551621148-Abell_3667-corr.MS
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseMsSpecParams   Re-index is enabled
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseMsSpecParams   WEIGHT_SPECTRUM will be written in output MS
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseMsSpecParams   Tile shape is [0]
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseDataSelParams  field selection is Abell 3667
# 2023-03-07 13:50:42   INFO    MSTransformManager::parseCalParams      Calibration is activated
# 2023-03-07 13:50:42   INFO    MSTransformManager::colCheckInfo        Adding DATA column to output MS from input CORRECTED_DATA column
# 2023-03-07 13:50:42   INFO    MSTransformManager::initDataSelectionParams     Selected Fields Ids are [1]
# 2023-03-07 13:50:42   INFO    MSTransformManager::open        Select data
# 2023-03-07 13:50:42   INFO    MSTransformManager::createOutputMSStructure     Create output MS structure
# 2023-03-07 13:50:43   INFO    MSTransformDataHandler::makeSelection   5985930 out of 7784820 rows are going to be considered due to the selection criteria.
# 2023-03-07 13:50:43   INFO    MSTransformManager::checkFillWeightSpectrum     Optional column WEIGHT_SPECTRUM found in input MS will be written to output MS
# 2023-03-07 13:50:43   INFO    MSTransformManager::generateIterator    OTF calibration activated, using calibration record spec to generate iterator
# 2023-03-07 13:50:48   INFO    Calibrater::    Arranging to calibrate MS: /stimela_mount/input/1551621148.MS
# 2023-03-07 13:50:48   INFO    OldCalibrater::setcallib2(callib)       Arranging to APPLY:
# 2023-03-07 13:50:48   INFO            .   B Jones: table=/stimela_mount/output/caltables/PipelineRun_Continuum-1551621148-2kgb_primary.B1 (by cal library) calWt=false
# 2023-03-07 13:50:48   SEVERE  OldCalibrater::setcallib2(callib)       Field Expression: No match found for name "Abell"
# 2023-03-07 13:50:48   SEVERE  OldCalibrater::setcallib2(callib)+      (near char. 5 in string "Abell,3667") Check inputs and try again.
# 2023-03-07 13:50:48   SEVERE  mstransform:::: Error in Calibrater::callib2.
# 2023-03-07 13:50:48   INFO    mstransform:::: ##### End Task: mstransform          #####
# 2023-03-07 13:50:48   INFO    mstransform::::+        ##########################################

My obsconf setup is as follows:

obsconf:
  target: 
    - all
  fcal: 
    - 'J1939-6342'
  bpcal: ['J1939-6342']
  gcal:
    - 'J1939gaincal'

where I've tried different variations on target, including Abell 3667 and ['Abell 3667'] (trying to explicitly define the field name here) or, as you see above, all. In each instance, while the field is correctly found in the MS, it seems that CARACal is either splitting the field name at the space when it comes to calling mstransform, or is inserting a comma in place of the space; either way, the net result is that the task fails.

I've also tried manually editing the FIELD and SOURCE tables using CASA's browsetable to remove the space, but that ends up with a different set of errors.

I can't be the first person to have encountered this issue, but as this isn't mentioned in the CARACal readthedocs page and I can't find anything among the GitHub issues, I'm creating this new one. Is there a workaround?

[KshitijT]

I think this might be related to a similar issue in ddcal worker : #1447 .

[AstroRipples]

Hi Kshitij 👋 ah yes, it seems like a similar problem... is there an easy fix I can make to my MS and/or an updated release of CARACal I can pull that's had this fixed? I've noticed I'm running v1.0.4 and I know I installed it some time ago, so perhaps there's something more recent?

[AstroRipples]

Just to provide a little update on this... I've changed the field name in the MS (using CASA'sbrowsetable) to remove the issue with the space, as you can see below:

However, when I call CARACal using Abell3667 as the target name, i.e.:

obsconf:
  target:
    - 'Abell3667'

The pipeline fails to find the field name:

...
2023-03-08 10:11:14 CARACal INFO: obsconf: initializing
2023-03-08 10:11:14 CARACal INFO: obsinfo file 1551621148-obsinfo.txt exists, not regenerating
2023-03-08 10:11:14 CARACal INFO: summary file 1551621148-summary.json exists, not regenerating
2023-03-08 10:11:14 CARACal INFO: elevation plot 1551621148-elevation-tracks.png exists, not regenerating
2023-03-08 10:11:14 CARACal INFO: MS #0: 1551621148.MS
2023-03-08 10:11:15 CARACal INFO: generated new obsinfo/1551621148-obsinfo.txt
2023-03-08 10:11:15 CARACal INFO: generated new obsinfo/1551621148-summary.json
2023-03-08 10:11:15 CARACal INFO: generated new obsinfo/1551621148-elevation-tracks.png
2023-03-08 10:11:15 CARACal INFO:   1 spectral windows, with NCHAN=4096
2023-03-08 10:11:15 CARACal INFO:   CHAN_FREQ from 856000000.0 Hz to 1711791015.625 Hz with average channel width of 208984.375 Hz
2023-03-08 10:11:15 CARACal ERROR: Can't find an appropriate FIELD for obsinfo: target: Abell3667. Please check this config setting. It may also be that your MS scan intents are not pupulated correctly, in which case you must set target to a list of explicit field names. [RuntimeError]
2023-03-08 10:11:15 CARACal INFO:   More information can be found in the logfile at OutputData/logs-20230308-101114/log-caracal.txt

despite the fact that it's very clearly there in the MS. The error suggests that CARACal is looking for the FIELD definition somewhere that I haven't modified, but I've changed the FIELD and SOURCE tables in the MS...

[KshitijT]

@AstroRipples , could you please share the full config you are using?

[AstroRipples]

Sure thing, here's the config and log:

Config file: caracal_initcal_continuum_a3667-1551621148_advanced.yml.txt Full log: log-caracal.txt

[KshitijT]

@AstroRipples , could you please confirm that the field names in the obsinfo file are the same as the ones in the config?

2023-03-08 10:11:14 CARACal INFO: summary file 1551621148-summary.json exists, not regenerating

This line means that if you have kept the output directory the same as previous attempts, the names in the obsinfo file would not be updated.

[AstroRipples]

Hi @KshitijT ... in the last couple of days I've purged and re-run CARACal a couple of times, and perhaps I uploaded the wrong log file after my latest run.

In any case, I've checked the names in the obsinfo file, and I can confirm they do not match the ones in my config. However, the ones in my config file do match the ones in the MS:

...
2023-03-08 09:08:13 INFO    listobs::ms::summary    Fields: 3
2023-03-08 09:08:13 INFO    listobs::ms::summary+     ID   Code Name                RA               Decl           Epoch   SrcId      nRows
2023-03-08 09:08:13 INFO    listobs::ms::summary+     0    T    J1939-6342          19:39:25.030000 -63.42.45.70000 J2000   0        1222440
2023-03-08 09:08:13 INFO    listobs::ms::summary+     1    T    Abell3667           20:12:33.680000 -56.50.26.30000 J2000   1        5985930
2023-03-08 09:08:13 INFO    listobs::ms::summary+     2    T    J1939gaincal        19:39:25.030000 -63.42.45.70000 J2000   2         576450
2023-03-08 09:08:13 INFO    listobs::ms::summary    Spectral Windows:  (1 unique spectral windows and 1 unique polarization setups)
2023-03-08 09:08:13 INFO    listobs::ms::summary+     SpwID  Name   #Chans   Frame   Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) CtrFreq(MHz)  Corrs          
2023-03-08 09:08:13 INFO    listobs::ms::summary+     0      none    4096   TOPO     856.000       208.984    856000.0   1283.8955   XX  XY  YX  YY
2023-03-08 09:08:13 INFO    listobs::ms::summary    Sources: 3
2023-03-08 09:08:13 INFO    listobs::ms::summary+     ID   Name                SpwId RestFreq(MHz)  
2023-03-08 09:08:13 INFO    listobs::ms::summary+     0    J1939-6342          0     -              
2023-03-08 09:08:13 INFO    listobs::ms::summary+     1    Abell3667           0     -              
2023-03-08 09:08:13 INFO    listobs::ms::summary+     2    J1939gaincal        0     -              
2023-03-08 09:08:13 INFO    listobs::ms::summary+     NB: No systemic velocity information found in SOURCE table.
...

so I don't know where CARACal is populating the obsinfo file from. I would have expected CARACal to take it from the MS, but that doesn't seem to be the case, as the obsinfo file (see below) still has the space in the target field name:

...
Fields: 3
  ID   Code Name                RA               Decl           Epoch   SrcId      nRows
  0    T    J1939-6342          19:39:25.030000 -63.42.45.70000 J2000   0        1222440
  1    T    Abell 3667          20:12:33.680000 -56.50.26.30000 J2000   1        5985930
  2    T    J1939gaincal        19:39:25.030000 -63.42.45.70000 J2000   2         576450
Spectral Windows:  (1 unique spectral windows and 1 unique polarization setups)
  SpwID  Name   #Chans   Frame   Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) CtrFreq(MHz)  Corrs          
  0      none    4096   TOPO     856.000       208.984    856000.0   1283.8955   XX  XY  YX  YY
Sources: 3
  ID   Name                SpwId RestFreq(MHz)  
  0    J1939-6342          0     -              
  1    Abell 3667          0     -              
  2    J1939gaincal        0     -              
  NB: No systemic velocity information found in SOURCE table.

When I match the information in the obsinfo file, we end up back at the initial problem of CARACal not being able to handle having a space in the field name 🤯

[KshitijT]

I think instead of purging CARACal (poor kitty) you should purge the existing obsinfo files from your output directory (maybe just copy them elsewhere).

My guess as to what's happening here is that you the obsinfo files are from the time when the field name still had the whitespace in it; CARACal doesn't update the obsinfo files if they are already in the output folder from a previous run (in your case the very first run before you edited the field name in the ms).

CARACal essentially runs listobs on the MS to generate the obsinfo files - then it tries to match the fieldnames you have provided in the config file with the ones in the (old) obsinfo files and then complains when they don't match..

You could confirm if this is the case very quickly - by just removing the obsinfo files from the output directory and running the config only till obsconf worker; if the fieldnames are updated (and without the whitespace) I think you should be sorted?

[AstroRipples]

Hey Kshitij ... ah sorry, I should've been clearer - no purging of the software has happened (kitty is ok 🐈 ) but I purged the output folder (in this case called OutputData) between runs so I was definitely proceeding from a clean slate...

However... in making sure everything was purged, I've also noticed that CARACal is also dropping a lot of stuff in the msdir folder, which it turns out I hadn't purged 🤦 it seems like the pipeline was getting the obsinfo from there rather than from the OutputData folder, as I've just purged that folder and the pipeline calibration is proceeding.

So TL;DR: it seems like things are okay now 👌 I'll keep this issue open for now until I've got through the pipeline (and while there's a workaround, not being able to handle a space in a field name is a feature/bug users should be aware of).

[KshitijT]

Awesome !

I'll keep on working on the whitespace issue, definitely a bug which we need to sort out.

[KshitijT]

@AstroRipples , I have checked this out and this does not seem a CARACal fault but a mstransform issue - it keeps complaining when there is a gap in the fieldname even when run manually. The workaround is to use field ID throughout instead. I'll try and enable that in the pipeline.

[Athanaseus]

@KshitijT please check this out!