secondary gaincal failing

[kendak333]

Got some weird CASA behaviour when I run crosscal after populating primary model via crystalball. gaincal fails when searching for MODEL_DATA, but it shouldn't be looking as smodel is set. Log output below (lines 9507 to 9541)

log-caracal.txt

[KshitijT]

@kendak333 , how was the primary model populated via crystalball ? Was it done within the pipeline or manually?

[kendak333]

@KshitijT manually - ran the yaml up to end of flag__rfi, then ran crystalball -sm ~/mkatmodels/fitted.PKS0407.UBand.wsclean.cat.txt -f <fieldID> <path-to-cal.ms>

and then restarted the yaml from crosscal

[KshitijT]

Ah, I see now why the error seemed so familiar, this is the same one encountered a while ago by @o-smirnov : https://github.com/caracal-pipeline/caracal/issues/1408#issuecomment-1031385295 . It's been a while since I looked at the code; but I think creating the scratch column (EDITED: for the secondary) (maybe via running clearcal with addmodel? ) would work. Alternatively, you could try the inbuilt UHF primary models, but I am not sure the model you are using is the same as the one CARACal has.. I'll take a more detailed look tomorrow and update.

[kendak333]

I'm specifically trying to run it with the model from Ben, so not the inbuilt models. Had a look at the thread you referenced - the smodel parameter is included, but CASA is still looking for the MODEL_DATA anyway. Unless the syntax for the smodel is wrong? e.g. the Stokes parameters as strings, not numbers?

[KshitijT]

--> I'm specifically trying to run it with the model from Ben, so not the inbuilt models.

The "in-built" models are from Ben too ! :)

--> - the smodel parameter is included, but CASA is still looking for the MODEL_DATA anyway. Unless the syntax for the smodel is wrong? e.g. the Stokes parameters as strings, not numbers?

I am not sure the smodel parameter fills in the MODEL_DATA, but I'll take a look again and try out with UHF data.

[kendak333]

The "in-built" models are from Ben too ! :)

From here?

[kendak333]

I've checked with Ben and caracal is still using the old LSM models, so at the moment crystalball is the way to go for me. He's given the suggestion to run clearcal with usescratch=True before doing the crystalball step in order to populate the MODEL_DATA for the secondary. @KshitijT My question then is how do I do that in the yaml, or do I need to do it outside of caracal?

[KshitijT]

@kendak333, you can do it offline as per Ben's suggestion.

Fyi CARACal does have access to the crystalball models: https://github.com/caracal-pipeline/caracal/blob/master/caracal/schema/crosscal_schema.yml#L72 ; but I just had a chat with @bennahugo so I'll first confirm if I need to update the models.

[o-smirnov]

Seeing a similar problem with:

2023-03-27 16:17:52 CARACal INFO:     set_model:
2023-03-27 16:17:52 CARACal INFO:       enable:                       True
2023-03-27 16:17:52 CARACal INFO:       meerkat_band:                 L
2023-03-27 16:17:52 CARACal INFO:       meerkat_skymodel:             False
2023-03-27 16:17:52 CARACal INFO:       meerkat_crystalball_skymodel: True
2023-03-27 16:17:52 CARACal INFO:       meerkat_crystalball_memory_fraction:0.5
2023-03-27 16:17:52 CARACal INFO:       meerkat_crystalball_num_sources:0
2023-03-27 16:17:52 CARACal INFO:       meerkat_crystalball_row_chunks:0

It falls over with a missing MODEL_DATA when it gets to the secondary.

Looking at the crosscal code (and the set_model output), I'm a bit suspicious of the logic. If meerkat_crystalball_skymodel is set, it seems to me that the secondary model is not set up at all. @KshitijT do you concur?

[o-smirnov]

@elizabethww @SihlanguI

[KshitijT]

@o-smirnov , I am travelling right now, but will check it out as soon as possible.

[kendak333]

I've put together a simple bash that sorts this out, will share when back at my computer. But yes, it should get fixed in the code at some point.

On Fri, 31 Mar 2023, 12:36 Kshitij Thorat, @.***> wrote:

@o-smirnov https://github.com/o-smirnov , I am travelling right now, but will check it out as soon as possible.

— Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1450#issuecomment-1491708040, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH66KMWEIP2NW7Z7Y4Z2EC3W62XUBANCNFSM6AAAAAARI53UDY . You are receiving this because you were mentioned.Message ID: @.***>

[o-smirnov]

@kendak333 could you please share the fix so long.

[kendak333]

It's not elegant but I run ./caracal_modscript.sh

where the bash script is #!/bin/bash nice caracal -c jwst0723_u.yml -ct singularity -sid /home/knowles/singularity-pull/ -ew flag__rfi casa -c setcalmods.py crystalball -sm ~/mkatmodels/fitted.PKS0407.UBand.wsclean.cat.txt -f 0 /scratch/knowles/jwstu/msdir/jwst0723_u-cal.ms nice caracal -c jwst0723_u.yml -ct singularity -sid /home/knowles/singularity-pull/ -sw crosscal -ew selfcal

and setcalmods.py is clearcal(vis='/scratch/knowles/jwstu/msdir/jwst0723_u-cal.ms',field='J0943-7620',addmodel=True) clearcal(vis='/scratch/knowles/jwstu/msdir/jwst0723_u-cal.ms',field='J0521+1638',addmodel=True)

Currently hardcoding the appropriate field names in the python script, but will figure out a neater way later.

The crosscal worker setup is here: crosscal.txt

[o-smirnov]

Thanks @kendak333!

@elizabethww, no need to run crystalball by hand here. I can guess that @kendak333's fix is from the previous version of Caracal which didn't have the crystallball UHF models in it. The crystalball part is now done correctly for you by Caracal. The critical remaining bit is this one:

clearcal(vis='/scratch/knowles/jwstu/msdir/jwst0723_u-cal.ms',field='J0943-7620',addmodel=True)

Do this in CASA, setting field to your secondary, and the MS path to your MS. Then run again with -sw crosscal.

[KshitijT]

@elizabethww and @o-smirnov this is now implemented in the latest master version of CARACal, please switch on clearcal in prep worker to do the process organically in the pipeline if that works better for you.

[kendak333]

@KshitijT I had tried that but it only set an empty model column for the primary calibrator.

[KshitijT]

@kendak333 the primary should have been filled later in crosscal worker?

[kendak333]

@KshitijT yes, but when implementing clearcal: true in the prep worker, it doesn't create an empty model column for all the calibrators, just the first one. Or at least it didn't for me, and so it still crashed during crosscal when using the crystalball model, which is why I needed to do this work around.

[KshitijT]

@KshitijT I had tried that but it only set an empty model column for the primary calibrator.

Also note that if you tried this with an older (i.e. anything more than half an hour older) version of CARACal, addmodel was explicitly set to False in clearcal, and there was no way to switch it on. This has now been changed, so if you run clearcal it should add scratch columns to your data, if that helps?

[KshitijT]

@KshitijT yes, but when implementing clearcal: true in the prep worker, it doesn't create an empty model column for all the calibrators, just the first one. Or at least it didn't for me, and so it still crashed during crosscal when using the crystalball model, which is why I needed to do this work around.

Very weird, let me recheck.

[kendak333]

Ah ha! That'd be it then :) So just to confirm, with the latest version of caracal (as of now) the following prep and crosscal workers should work without having to do anything external to the caracal run (no external casa, no external crystallball)?

prep: enable: true label_in: cal field: calibrators fixcalcoords: enable: true specweights: enable: true mode: uniform clearcal: true

crosscal: enable: true set_model: enable: true meerkat_skymodel: false meerkat_band: L meerkat_crystalball_skymodel: true

[KshitijT]

@kendak333 it should work fine now, but give me half an hour and I'll confirm once again.

But could you confirm that your data is L band? In that case you should not be using the crystalball models, which are for UHF band.

[kendak333]

Some of it is L band, some of it is UHF. When I started doing all this the updated L band crystalball models from Ben apparently weren't implemented yet or something, which is why I was implementing them externally.