Flag summary exits with error code 1

[PeterKamphuis]

Running the current master version on a GMRT data set results in an error when producing the flag summary. This is due to the two fake GMRT antennas that throw a NaN in 'output/diagnostic_plots/Original-cal-flag-flagging-summary-0.json' (See full traceback below).

This is is despite setting the inspect to false in the yaml file.

It is easily solveable by running an mstransform that exclude the given antennas, antenna = '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29&' In my casa (v5.5) antenna ='!30,31' does not work.

Unfortunately this problem is not solveable within caracal as transform does not allow for the selection of antennas (Why not?). Making it difficult to create a caracal yaml file that can deal with GMRT observations out of the box.

# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 35, in <module>
#     flag_stats.plot_statistics(**args)
#   File "/usr/local/lib/python3.6/dist-packages/MSUtils/flag_stats.py", line 423, in plot_statistics
#     _plot_flag_stats(**flag_data, outfile=htmlfile)
#   File "/usr/local/lib/python3.6/dist-packages/MSUtils/flag_stats.py", line 417, in _plot_flag_stats
#     row(plot_list[1], plot_list[0])))
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/io/saving.py", line 86, in save
#     _save_helper(obj, filename, resources, title, template, theme)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/io/saving.py", line 148, in _save_helper
#     html = file_html(obj, resources, title=title, template=template, theme=theme)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/embed/standalone.py", line 304, in file_html
#     (docs_json, render_items) = standalone_docs_json_and_render_items(models_seq, suppress_callback_warning=suppress_callback_warning)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/embed/util.py", line 294, in standalone_docs_json_and_render_items
#     docs_json[docid] = doc.to_json()
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/document/document.py", line 902, in to_json
#     doc_json = self.to_json_string()
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/document/document.py", line 959, in to_json_string
#     return serialize_json(json, indent=indent)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/core/json_encoder.py", line 166, in serialize_json
#     return json.dumps(obj, cls=BokehJSONEncoder, allow_nan=False, indent=indent, separators=separators, sort_keys=True, **kwargs)
#   File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
#     **kw).encode(obj)
#   File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
#     chunks = self.iterencode(o, _one_shot=True)
#   File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
#     return _iterencode(o, 0)
# ValueError: Out of range float values are not JSON compliant
2022-05-18 10:55:16 CARACal.Stimela.flag-summary-ms0 ERROR: cd /home/peter/Current_Reduction/J1150_00_caracal/.stimela_workdir-16528640819505663 && singularity returns error code 1
2022-05-18 10:55:16 CARACal.Stimela.flag-summary-ms0 ERROR: job failed at 2022-05-18 10:55:16.196315 after 0:00:05.003909
2022-05-18 10:55:16 CARACal ERROR: Job 'flag-summary-ms0:: Flagging summary  ms=Original-cal.ms' failed: cd /home/peter/Current_Reduction/J1150_00_caracal/.stimela_workdir-16528640819505663 && singularity returns error code 1 [PipelineException]
2022-05-18 10:55:16 CARACal INFO:   More information can be found in the logfile at /home/peter/Current_Reduction/J1150_00_caracal/output/logs-20220518-105320/log-caracal.txt
2022-05-18 10:55:16 CARACal INFO: exiting with error code 1 

[paoloserra]

Thanks for reporting this. Could you maybe copy to full log?

[PeterKamphuis]

Here is the log

disabling the summary works but then it has to be disabled every where, i.e. when disabled in the first flag it crashes in crosscal on the same issue. I'll try to add the antenna's to the transform worker.

[paoloserra]

cool, thanks!

[o-smirnov]

The solution is to check for NaNs in MSUtils/flag_stats.py and replace them with 0s. @Athanaseus you perhaps?

[Athanaseus]

@PeterKamphuis is it possible to share/email me a subset of the similar GMRT data?

[PeterKamphuis]

@SpheMakh the cab/casa_mstransform does not have the antenna parameter while mstransform does. Is there a reason for this? Or am I missing something?

Also according to stimela cabs -i casa_mstransform the scan parameter is there twice. Maybe this is a stimela issue instead of caracal?

[PeterKamphuis]

Solved in #1428

you can add a transform after the obsconf with : transform: enable: false field: 'calibrators' label_out: 'cal' split_field: enable: true col: 'data' antennas: '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29&'

and caracal should run fine.

[rramij]

Facing the exact same 'Flag summary error'. Then added the antennas as follows:

transform:
  enable: true
  label_out: cal
  field: calibrators
  split_field:
    enable: true
    col: data
    antennas: 'C00,C01,C02,C03,C04,C05,C06,C08,C09,C10,C11,C12,C13,C14,E02,E03,E04,E05,E06,S01,S02,S03,S04,S06,W01,W02,W03,W04,W05,W06&'

Now the pipeline stops running. Returning below error:

2022-12-23` 15:23:12 CARACal INFO: Invoked as /home/rramij/venv/bin/caracal -ct singularity -c uGMRT_pipe.yml -sid /home/rramij/singularity_images
2022-12-23 15:23:13 CARACal ERROR: configuration file uGMRT_pipe.yml fails to validate, list of errors follows:
  transform:
    - Key 'antennas' was not defined. Path: '/transform/split_field'
``