search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

KeyError when using 'J1130-1449' as a polarization calibrator #1371

Closed Sinenhlanhla closed 2 years ago

Sinenhlanhla commented 2 years ago

I am trying to use 'J1130-1449', an unknown polcal source, to calibrate my data. After setting leakage_calib=pol_calib=xcal, I get this error: 2021-09-20 10:59:51 CARACal.Stimela.polcal INFO: --------------------------------- 2021-09-20 10:59:51 CARACal.Stimela.polcal INFO: Stimela version 1.7.0 2021-09-20 10:59:51 CARACal.Stimela.polcal INFO: Running: polcal 2021-09-20 10:59:51 CARACal.Stimela.polcal INFO: --------------------------------- 2021-09-20 10:59:51 CARACal.Stimela.polcal INFO: Config directory cannot be found. Will create /home/precious/merghers/.stimela_workdir-16321283915562663/stimela_parameter_files 2021-09-20 10:59:51 CARACal INFO: polcal: initializing 2021-09-20 10:59:51 CARACal INFO: RECEPTOR_ANGLE has been rotated by -90.0 degrees 2021-09-20 10:59:51 CARACal.Stimela.polcal INFO: Adding cab '/home/precious/stimela-images/stimela_casa_1.6.3.sif' (5.6.1-8) to recipe, container name 'save-cfp2020_4a_polcal_before-ms0' 2021-09-20 10:59:51 CARACal.Stimela.polcal INFO: Adding cab '/home/precious/stimela-images/stimela_casa_1.6.3.sif' (5.6.1-8) to recipe, container name 'extend_flags_polcal' 2021-09-20 10:59:51 CARACal INFO: You decided to calibrate the polarized angle and leakage with a polarized calibrator. 2021-09-20 10:59:51 CARACal INFO: Using a known model for the polarized calibrator. 2021-09-20 10:59:51 CARACal ERROR: 'J1130-1449' [KeyError] 2021-09-20 10:59:51 CARACal INFO: More information can be found in the logfile at output/logs-20210920-105950/log-caracal.txt 2021-09-20 10:59:51 CARACal INFO: You are running version 1.0.4 2021-09-20 10:59:51 CARACal ERROR: Traceback (most recent call last): 2021-09-20 10:59:51 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/main.py", line 183, in __run 2021-09-20 10:59:51 CARACal ERROR: pipeline.run_workers() 2021-09-20 10:59:51 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/workers/worker_administrator.py", line 441, in run_workers 2021-09-20 10:59:51 CARACal ERROR: worker.worker(self, recipe, config) 2021-09-20 10:59:51 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/workers/polcal_worker.py", line 1355, in worker 2021-09-20 10:59:51 CARACal log-caracal.txt ERROR: xcal_model_xcal_leak(msname, msinfo, prefix_msbase, recipe, config, pipeline, i, 2021-09-20 10:59:51 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/workers/polcal_worker.py", line 410, in xcal_model_xcal_leak 2021-09-20 10:59:51 CARACal ERROR: normfreq = (meanchanfreq / float(polarized_calibrators[field]["reffreq"][:-3])) 2021-09-20 10:59:51 CARACal ERROR: KeyError: 'J1130-1449' 2021-09-20 10:59:51 CARACal INFO: exiting with error code 1

o-smirnov commented 2 years ago

Use leakage_calib=bpcal, and pol_calib=xcal, and ask @francescaLoi to add J1130-1449 to the list of allowed polcals...

o-smirnov commented 2 years ago

ALso @Sinenhlanhla could you please post your .yml file so we can advise.

o-smirnov commented 2 years ago

And set_model_pol=false.

o-smirnov commented 2 years ago

@bennahugo will give us a link to Bill Cotton's model for this calibrator, we'll add it.

bennahugo commented 2 years ago

https://archive-gw-1.kat.ac.za/public/meerkat/MeerKAT-L-band-Polarimetric-Calibration.pdf

On Wed, Sep 29, 2021 at 11:30 AM Oleg Smirnov @.***> wrote:

@bennahugo https://github.com/bennahugo will give us a link to Bill Cotton's model for this calibrator, we'll add it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1371#issuecomment-930008731, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6WSP2LL4274LNSPIETUELMENANCNFSM5ELQAWIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

francescaLoi commented 2 years ago

@Sinenhlanhla I just add a new branch with bandwidth averaged properties of J1130-1449. Could you please check if it works? The changes are in the add_j1130 branch. Remember to indicate J1130-1449 as your xcal calibrator in the obsconf worker. Then something like:

polcal:
  enable: true
  label_in: cal
  label_cal: 1pcal
  otfcal:
    enable: true
    label_cal: 1kgb
  extendflags: True
  feed_angle_rotation: '-90'
  uvrange: '>150'
  reuse_existing_tables: false
  pol_calib: xcal
  leakage_calib: fcal
  set_model_leakage:
    enable: true
    meerkat_skymodel: true
  set_model_pol: true
  gain_solint: '60s'
  time_solint: 'inf'
  plotgains: true
  apply_pcal: true
  applyto:
    - gcal
    - bpcal
    - xcal

should work.

Sinenhlanhla commented 2 years ago

Thank you, @francescaLoi. I will test it and update you.

Sinenhlanhla commented 2 years ago

@francescaLoi adding the model helped, but now I'm getting an error at the setjy step:

TDL script successfully compiled. 9557 node definitions

(of which 2 are root nodes) sent to meqserver.

Running TDL job "_simulate_MS"

Job result: None

No more commands

Stopping the meqserver

All your batch are belong to us. Bye!

2021-10-04 11:01:05 CARACal.Stimela.set_model_cal-0 INFO: job complete at 2021-10-04 11:01:05.127751 after 0:54:22.008687 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 INFO: job started at 2021-10-04 11:01:05.128100 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: STEP 5 :: set_model_0 2021-10-04 11:01:05 CARACal.Stimela INFO: Validating parameters for cab casa_setjy (stimela/casa:1.6.3) 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: Parameter rotmeas is unknown. Run 'stimela cabs -i casa_setjy' to get help on this cab 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: job failed at 2021-10-04 11:01:05.129213 after 0:00:00.001113 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: Traceback (most recent call last): 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: File "/home/precious/venv/lib/python3.8/site-packages/stimela/recipe.py", line 710, in run 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: job.run_job() 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: File "/home/precious/venv/lib/python3.8/site-packages/stimela/recipe.py", line 417, in run_job 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: self.job._cab.update(self.job.config, 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: File "/home/precious/venv/lib/python3.8/site-packages/stimela/cargo/cab/init.py", line 391, in update 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: raise StimelaCabParameterError( 2021-10-04 11:01:05 CARACal.Stimela.set_model_0 ERROR: stimela.exceptions.StimelaCabParameterError: Parameter rotmeas is unknown. Run 'stimela cabs -i casa_setjy' to get help on this cab 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Completed jobs : ['save-cfp2020_4a_polcal_before-ms0', 'extend_flags_polcal', 'extend_flags_polcal', 'set_model_cal-0'] 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Remaining jobs : ['gain_xcal', 'crosshand_delay', 'crosshand_phase_freq', 'flag_phase', 'leakage_freq', 'plotgains-Df-0-cfp2020_4a-cfp2020_4a-1pcal.Df'] 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Logging remaining task: gain_xcal 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Logging remaining task: crosshand_delay 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Logging remaining task: crosshand_phase_freq 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Logging remaining task: flag_phase_freq 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Logging remaining task: leakage_freq 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Logging remaining task: plotgains-Df-0-cfp2020_4a-cfp2020_4a-1pcal.Df:: Plot gaincal phase 2021-10-04 11:01:05 CARACal.Stimela.polcal INFO: Saving pipeline information in .last_polcal.json 2021-10-04 11:01:05 CARACal ERROR: Job 'set_model_0' failed: Parameter rotmeas is unknown. Run 'stimela cabs -i casa_setjy' to get help on this cab [PipelineException] 2021-10-04 11:01:05 CARACal INFO: More information can be found in the logfile at output/logs-20211004-095220/log-caracal.txt 2021-10-04 11:01:05 CARACal INFO: You are running version 1.0.4 2021-10-04 11:01:05 CARACal ERROR: Traceback (most recent call last): 2021-10-04 11:01:05 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/main.py", line 183, in __run 2021-10-04 11:01:05 CARACal ERROR: pipeline.run_workers() 2021-10-04 11:01:05 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/workers/worker_administrator.py", line 441, in run_workers 2021-10-04 11:01:05 CARACal ERROR: worker.worker(self, recipe, config) 2021-10-04 11:01:05 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/workers/polcal_worker.py", line 1347, in worker 2021-10-04 11:01:05 CARACal ERROR: xcal_model_fcal_leak(msname, msinfo, prefix_msbase, recipe, config, pipeline, i, prefix, refant, polarized_calibrators, 2021-10-04 11:01:05 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/caracal/workers/polcal_worker.py", line 303, in xcal_model_fcal_leak 2021-10-04 11:01:05 CARACal ERROR: recipe.run() 2021-10-04 11:01:05 CARACal ERROR: File "/home/precious/venv/lib/python3.8/site-packages/stimela/recipe.py", line 761, in run 2021-10-04 11:01:05 CARACal ERROR: raise PipelineException(exc, self.completed, job, self.remaining) from None 2021-10-04 11:01:05 CARACal ERROR: stimela.exceptions.PipelineException: Job 'set_model_0' failed: Parameter rotmeas is unknown. Run 'stimela cabs -i casa_setjy' to get help on this cab 2021-10-04 11:01:05 CARACal INFO: exiting with error code 1

francescaLoi commented 2 years ago

Sorry @Sinenhlanhla , I didn't notice that we don't have a "rotmeas" parameter in casa_setjy. We should include it through stimela. I'll open an issue there and let you know when this will be fixed.

francescaLoi commented 2 years ago

Hey @Sinenhlanhla , could you refresh stimela and try again?

Sinenhlanhla commented 2 years ago

I'm still getting the same error after updating and re-installing stimela. I've tried installing the development version, but it stalls.

francescaLoi commented 2 years ago

can you see the "rotmeas" parameter if you type "stimela cabs -i casa_setjy"?

Sinenhlanhla commented 2 years ago

It's not there. Is there a particular branch that I need to chekout in order to see the updated parameters?

francescaLoi commented 2 years ago

@SpheMakh could you please help? How can @Sinenhlanhla obtain the updated version of stimela with the "rotmeas" parameter? I cloned the repository and installed it in the virtual environment but maybe there's a better way...

Sinenhlanhla commented 2 years ago

@francescaLoi I've also now cloned the repository and this seems to have worked, thanks.