search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

Calibrating uGMRT: plotting error #1449

Open phetogoportia opened 1 year ago

phetogoportia commented 1 year ago

@Athanaseus

I am trying to calibrate uGMRT data with caracal; however, it fails with an error instead:

# MSUtils.flag_stats - 2022-10-15 20:18:51,926 INFO - Computing correlation flag stats data...
# MSUtils.flag_stats - 2022-10-15 20:18:51,926 INFO - Correlation Names: ['RR', 'LL']
# MSUtils.flag_stats - 2022-10-15 20:18:58,439 INFO - Output json file: /stimela_mount/output/3c105-cal-flag-flagging-summary-0.json.
# MSUtils.flag_stats - 2022-10-15 20:18:58,443 INFO - Plotting flag stats data.
# MSUtils.flag_stats - 2022-10-15 20:18:58,473 INFO - Output plots: /stimela_mount/output/3c105-cal-flag-flagging-summary-plots-0.html.
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 35, in <module>
#     flag_stats.plot_statistics(**args)
#   File "/usr/local/lib/python3.6/dist-packages/MSUtils/flag_stats.py", line 423, in plot_statistics
#     _plot_flag_stats(**flag_data, outfile=htmlfile)
#   File "/usr/local/lib/python3.6/dist-packages/MSUtils/flag_stats.py", line 417, in _plot_flag_stats
#     row(plot_list[1], plot_list[0])))
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/io/saving.py", line 86, in save
#     _save_helper(obj, filename, resources, title, template, theme)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/io/saving.py", line 148, in _save_helper
#     html = file_html(obj, resources, title=title, template=template, theme=theme)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/embed/standalone.py", line 304, in file_html
#     (docs_json, render_items) = standalone_docs_json_and_render_items(models_seq, suppress_callback_warning=suppress_callback_warning)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/embed/util.py", line 294, in standalone_docs_json_and_render_items
#     docs_json[docid] = doc.to_json()
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/document/document.py", line 902, in to_json
#     doc_json = self.to_json_string()
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/document/document.py", line 959, in to_json_string
#     return serialize_json(json, indent=indent)
#   File "/usr/local/lib/python3.6/dist-packages/bokeh/core/json_encoder.py", line 166, in serialize_json
#     return json.dumps(obj, cls=BokehJSONEncoder, allow_nan=False, indent=indent, separators=separators, sort_keys=True, **kwargs)
#   File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
#     **kw).encode(obj)
#   File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
#     chunks = self.iterencode(o, _one_shot=True)
#   File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
#     return _iterencode(o, 0)
# ValueError: Out of range float values are not JSON compliant
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR: cd /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 && singularity run --workdir /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 --containall returns error code 1
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR: job failed at 2022-10-15 20:18:58.768665 after 0:00:41.322741
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR: Traceback (most recent call last):
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:   File "/home/phetogo/venv/cara/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:     job.run_job()
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:   File "/home/phetogo/venv/cara/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:     self.job.run(output_wrangler=self.apply_output_wranglers)
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:   File "/home/phetogo/venv/cara/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:     utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall",
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:   File "/home/phetogo/venv/cara/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR:     raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2022-10-15 20:18:58 CARACal.Stimela.flag-summary-ms0 ERROR: stimela.utils.StimelaCabRuntimeError: cd /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 && singularity run --workdir /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 --containall returns error code 1
2022-10-15 20:18:58 CARACal.Stimela.flag INFO: Completed jobs : ['save-3C105_flag_before-ms0', 'flag-autocorr-ms0', 'flag-spw-ms0', 'flag-manual-ms0-0', 'flag-manual-ms0-1', 'flag-manual-ms0-2', 'flag-manual-ms0-3', 'flag-rfi-ms0']
2022-10-15 20:18:58 CARACal.Stimela.flag INFO: Remaining jobs : []
2022-10-15 20:18:58 CARACal.Stimela.flag INFO: Saving pipeline information in .last_flag.json
2022-10-15 20:18:58 CARACal ERROR: Job 'flag-summary-ms0:: Flagging summary  ms=3c105-cal.ms' failed: cd /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 && singularity run --workdir /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 --containall returns error code 1 [PipelineException]
2022-10-15 20:18:58 CARACal INFO:   More information can be found in the logfile at output-3c/logs-20221015-195321/log-caracal.txt
2022-10-15 20:18:58 CARACal INFO:   You are running version 1.0.6-24-g63019ef0
2022-10-15 20:18:58 CARACal ERROR: Traceback (most recent call last):
2022-10-15 20:18:58 CARACal ERROR:   File "/home/phetogo/Software/caracal/caracal/main.py", line 187, in __run
2022-10-15 20:18:58 CARACal ERROR:     pipeline.run_workers()
2022-10-15 20:18:58 CARACal ERROR:   File "/home/phetogo/Software/caracal/caracal/workers/worker_administrator.py", line 441, in run_workers
2022-10-15 20:18:58 CARACal ERROR:     worker.worker(self, recipe, config)
2022-10-15 20:18:58 CARACal ERROR:   File "/home/phetogo/Software/caracal/caracal/workers/flag_worker.py", line 551, in worker
2022-10-15 20:18:58 CARACal ERROR:     recipe.run()
2022-10-15 20:18:58 CARACal ERROR:   File "/home/phetogo/venv/cara/lib/python3.8/site-packages/stimela/recipe.py", line 764, in run
2022-10-15 20:18:58 CARACal ERROR:     raise PipelineException(exc, self.completed, job, self.remaining) from None
2022-10-15 20:18:58 CARACal ERROR: stimela.exceptions.PipelineException: Job 'flag-summary-ms0:: Flagging summary  ms=3c105-cal.ms' failed: cd /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 && singularity run --workdir /home/phetogo/new_project/3C105/ugmrt/.stimela_workdir-16658570665552719 --containall returns error code 1
2022-10-15 20:18:58 CARACal INFO: exiting with error code 1
Athanaseus commented 1 year ago

Duplicate of #1422