Pass the same spectral windows for processing by Cubical and wsclean in selfcal worker

[viralp]

Hi I am getting the following error in wsclean (after the cubical run)

+ An exception occured:
+ >>> Table DataManager error: Invalid operation: TSM: no array in row 43864 of column CORRECTED_DATA in /stimela_mount/msdir/A3376A-Abell_3376-corr.ms/table.f27

The previous cubical run was completed without severe error except the following.

ERROR     20:58:40 - casa_db_adaptor    [0.2 0.2 71.9Gb] Gaintables cannot be written in Python 3 mode due to current casacore implementation issues

log-selfcal-calibrate-cubical-field0-iter1-20210725-160301.txt log-selfcal-image-field0-iter2-20210725-160301.txt

[o-smirnov]

It looks like your MS has 8 spectral windows. Not sure how/if the selfcal worker copes with this (@KshitijT?) I can see cubical is using spw 0, while wsclean is presumably trying to read them all, hence the error.

Come to think of it, I'm not even sure how wsclean itself copes with multiple SPWs?

[KshitijT]

It looks like your MS has 8 spectral windows. Not sure how/if the selfcal worker copes with this (@KshitijT?) I can see cubical is using spw 0, while wsclean is presumably trying to read them all, hence the error.

Come to think of it, I'm not even sure how wsclean itself copes with multiple SPWs?

There's a "spwid" parameter for the selfcal worker which maps to Cubical and should take care of multiple spws. As you say, @o-smirnov , wsclean seems to be imaging all the spws, so that should be fine; though there is a 'spws' option for wsclean for selecting which spectral windows to image - we don't have that in CARACal yet, however.

[viralp]

@KshitijT, I am not specifying any particular spws, so by default does cubical take all spws?

On Thu, Jul 29, 2021 at 11:52 AM Kshitij Thorat @.***> wrote:

It looks like your MS has 8 spectral windows. Not sure how/if the selfcal worker copes with this @.*** https://github.com/KshitijT?) I can see cubical is using spw 0, while wsclean is presumably trying to read them all, hence the error.

Come to think of it, I'm not even sure how wsclean itself copes with multiple SPWs?

There's a "spwid" parameter for the selfcal worker which maps to Cubical and should take care of multiple spws. As you say, @o-smirnov https://github.com/o-smirnov , wsclean seems to be imaging all the spws, so that should be fine; though there is a 'spws' option for wsclean for selecting which spectral windows to image - we don't have that in CARACal yet, however.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1360#issuecomment-888976181, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBBM44NJQIBWPWD3XZU3P3T2EQGTANCNFSM5BFISP2Q .

-- Postdoctoral Research Fellow South African Radio Astronomy Observatory (SARAO) and Rhodes University Square Kilometre Array, Cape Town South Africa

[o-smirnov]

Well I can see from the log that it's been told to process spw 0 only (--sel-ddid 0). It should be able to take multiple spws if you say --sel-ddid None, which is actually the default.

[KshitijT]

@KshitijT, I am not specifying any particular spws, so by default does cubical take all spws? … On Thu, Jul 29, 2021 at 11:52 AM Kshitij Thorat @.> wrote: It looks like your MS has 8 spectral windows. Not sure how/if the selfcal worker copes with this @. https://github.com/KshitijT?) I can see cubical is using spw 0, while wsclean is presumably trying to read them all, hence the error. Come to think of it, I'm not even sure how wsclean itself copes with multiple SPWs? There's a "spwid" parameter for the selfcal worker which maps to Cubical and should take care of multiple spws. As you say, @o-smirnov https://github.com/o-smirnov , wsclean seems to be imaging all the spws, so that should be fine; though there is a 'spws' option for wsclean for selecting which spectral windows to image - we don't have that in CARACal yet, however. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1360 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBBM44NJQIBWPWD3XZU3P3T2EQGTANCNFSM5BFISP2Q . -- Postdoctoral Research Fellow South African Radio Astronomy Observatory (SARAO) and Rhodes University Square Kilometre Array, Cape Town South Africa

https://github.com/caracal-pipeline/caracal/blob/master/caracal/schema/selfcal_schema.yml#L57

The default is "0" - so only the 1st spw.

[o-smirnov]

Right, but then wsclean must be told to only process the first spw too, otherwise the calibration and imaging invocations within selfcal are not mutually consistent.

[KshitijT]

Right, but then wsclean must be told to only process the first spw too, otherwise the calibration and imaging invocations within selfcal are not mutually consistent.

Sorry, I was not clear. I meant that @viralp can give a list of all the spws manually and then Cubical should calibrate them fine and since wsclean is anyway imaging all the spws; it should be sorted?

[o-smirnov]

Yes, that'll sort @viralp but it still leaves a trap for the next unwary user. I think if the spw is set, then the same value should be passed to cubical and wsclean, otherwise you get these incomprehensible errors.

[KshitijT]

Yes, that'll sort @viralp but it still leaves a trap for the next unwary user. I think if the spw is set, then the same value should be passed to cubical and wsclean, otherwise you get these incomprehensible errors.

Yes, we should definitely do that !

[viralp]

Yes I agree with @o-smirnov, spwid should be common for both cubical and wsclean. Thanks.

[KshitijT]

While the branch hotfix_#1360 has fixed @viralp 's original issue; Cubical seems to give trouble if 'sel-ddid' of anything except 0 or null is provided:

INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]     found 1 frequency chunks: 0 11
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   applying TAQL query 'FIELD_ID == 0 && DATA_DESC_ID IN [1]' (5/40 rows selected)
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   64 antennas, 5 rows, 1/8 DDIDs, 2 timeslots, 2 corrs
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   DDID central frequencies are at 1.03 GHz
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb] Input model feed rotation disabled, PA rotation disabled
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb] Output visibilities derotation disabled
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb] Using 1 model(s) for 1 directions(s) (DDEs explicitly disabled)
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   model 0 (weight WEIGHT_SPECTRUM):
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]     direction 0: MODEL_DATA
INFO      10:55:23 - main               [0.2 4.4 0.0Gb] subtraction directions set to slice(None, None, None)
INFO      10:55:23 - param_db           [0.2 4.4 0.0Gb] creating /stimela_mount/output/phase-0.parmdb in fragmented mode
INFO      10:55:23 - gain_machine       [0.2 4.4 0.0Gb] G solutions will be saved to /stimela_mount/output/phase-0.parmdb
ERROR     10:55:23 - casa_db_adaptor    [0.2 4.4 0.0Gb] Gaintables cannot be written in Python 3 mode due to current casacore implementation issues
INFO      10:55:23 - param_db           [0.2 4.4 0.0Gb] creating /stimela_mount/output/delayself_0-BBC-field_0-ddid_1.parmdb in fragmented mode
INFO      10:55:23 - gain_machine       [0.2 4.4 0.0Gb] will compute & save suggested baseline-based corrections (BBCs) to /stimela_mount/output/delayself_0-BBC-field_0-ddid_1.parmdb
INFO      10:55:23 - gain_machine       [0.2 4.4 0.0Gb]   (these can optionally be applied in a subsequent CubiCal run)
INFO      10:55:23 - main               [0.2 4.4 0.0Gb] defining chunks (time 1, freq 0, also when SCAN_NUMBER jumps > 1.0)
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   read indexing columns (5 total rows)
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   built timeslot index (2 unique timestamps)
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   max chunk size is 1 timeslots and/or -- seconds
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   found 2 time chunks: 0:0:0 1:1:1 2
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   generated 2 row chunks based on time and DDID
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   row chunks yield 2 potential tiles
INFO      10:55:23 - data_handler       [0.2 4.4 0.0Gb]   coarsening this to 1 tiles (max 2 chunks per tile, based on 1/0 requested)
INFO      10:55:23 - main               [0.2 4.4 0.0Gb] single-process, single-thread mode
INFO      10:55:23 - main               [0.2 4.4 0.0Gb] Exiting with exception: KeyError(0)
 Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/cubical/main.py", line 548, in main
    estimate_mem(ms, tile_list, GD["data"], GD["dist"])
  File "/usr/local/lib/python3.6/dist-packages/cubical/data_handler/wisdom.py", line 24, in estimate_mem
    along_freq = len(data_handler.freqchunks[0])
KeyError: 0

Funny that it's the wisdom file throwing up the error, though. Any clues @o-smirnov , @SpheMakh ?

[o-smirnov]

@JSKenyon take a look at this please?

[JSKenyon]

Yeah, doing so. The problem is that chanfreqs or freqchunks is a dictionary, not a list. Should be easy to fix.

[JSKenyon]

@o-smirnov https://github.com/ratt-ru/CubiCal/pull/454 needs your review. Should fix this problem.

[JSKenyon]

The fix has been merged into CubiCal master. Is it possible for you to try it out and confirm that it fixes the issue? Once it is confirmed, I can do a release for you.

[KshitijT]

The fix has been merged into CubiCal master. Is it possible for you to try it out and confirm that it fixes the issue? Once it is confirmed, I can do a release for you.

Will check it out and report back. Thanks for a super quick fix !

[KshitijT]

The stimela branch cubical-16Nov2021 (thanks @SpheMakh !) which has the new Cubical (thanks @JSKenyon !) seems perfectly happy with sel-ddid with non-zero values !

[KshitijT]

The branch in question was merged last year in stimela master so I am closing this issue. Please reopen if needed.