ddcal worker not accepting target name

[kendak333]

The ddcal worker is replacing the point in a field name with a dash, and then throwing a KeyError (presumably because 'J2023-3..' is not the field name, 'J2023.3..' is. Output from the logfile:

2022-09-29 14:31:45 CARACal INFO: ddcal: initializing
2022-09-29 14:31:45 CARACal.Stimela.ddcal INFO: Adding cab '/home/knowles/singularity-pull/stimela_eidos_1.2.0.sif' (1.1.0) to recipe, container name 'make-pb'
2022-09-29 14:31:46 CARACal ERROR: 'J2023-3-5535' [KeyError]
2022-09-29 14:31:46 CARACal INFO:   More information can be found in the logfile at output-j2023u/logs-20220929-143135/log-caracal.txt
2022-09-29 14:31:46 CARACal INFO:   You are running version 1.0.6
2022-09-29 14:31:46 CARACal ERROR: Traceback (most recent call last):
2022-09-29 14:31:46 CARACal ERROR:   File "/home/knowles/caracal-venv/lib/python3.8/site-packages/caracal/main.py", line 187, in __run
2022-09-29 14:31:46 CARACal ERROR:     pipeline.run_workers()
2022-09-29 14:31:46 CARACal ERROR:   File "/home/knowles/caracal-venv/lib/python3.8/site-packages/caracal/workers/worker_administrator.py", line 441, in run_workers
2022-09-29 14:31:46 CARACal ERROR:     worker.worker(self, recipe, config)
2022-09-29 14:31:46 CARACal ERROR:   File "/home/knowles/caracal-venv/lib/python3.8/site-packages/caracal/workers/ddcal_worker.py", line 437, in worker
2022-09-29 14:31:46 CARACal ERROR:     dagga(field)
2022-09-29 14:31:46 CARACal ERROR:   File "/home/knowles/caracal-venv/lib/python3.8/site-packages/caracal/workers/ddcal_worker.py", line 253, in dagga
2022-09-29 14:31:46 CARACal ERROR:     sources_to_tag = de_dict[field.replace("_","-")]
2022-09-29 14:31:46 CARACal ERROR: KeyError: 'J2023-3-5535'
2022-09-29 14:31:46 CARACal INFO: exiting with error code 1

Here's the yaml input for the worker

ddcal:
  enable: true
  label_in: 'corr'
  use_pb: true
  image_dd:
    enable: false #true
    npix: 10000
    use_mask: true
    mask_sigma: 15
    cell: 1.6
    weight_robust: -0.5
  calibrate_dd:
    enable: True
    min_dist_from_phcentre: 900
    dist_ncpu: 4
    de_sources_mode: manual
    de_target_manual: ['J2023.3-5535']
    de_sources_manual: ['20h27m31.7s, -55d30m03.5s,size1; 20h18m02.04s, -55d39m40.99s,size2; 20h24m21.73s, -57d23m18.17s,size3; 20h21m37.37s, -54d35m29.11s,size4']
  copy_data:
    enable: true
  image_wsclean:
    enable: true
  transfer_model_dd:
    enable: true

[kendak333]

Hi @KshitijT has there been any progress on this?

[KshitijT]

@kendak333 , it's a bit weird because the ddcal worker should not replace a "." in the name of the target. If possible, could you please share the full log?

[kendak333]

@KshitijT here's the full log log-caracal.txt

[KshitijT]

Ok, I get how this is happening now. The filter_name function from dispatch_crew: https://github.com/caracal-pipeline/caracal/blob/4f716f7c7f16ff5b7face76af053fb459a06bd77/caracal/dispatch_crew/utils.py#L374 is the problem; it replaces the "." AND the "_" in the name with "-". This is supposed to be so that the output files are easy to write, but in this case the (slightly) non-standard field name gives the error above. Solution is probably to shuffle the call to filter_name around.