search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

Inspect fails because of type error #1037

Closed gigjozsa closed 4 years ago

gigjozsa commented 4 years ago

This:

/home/jozsa/software/caracal_tests/thatest_new/caracal_venv/lib/python3.7/site-packages/caracal/workers/inspect_worker.py:209: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  corrs = yaml.load(stdr)['CORR']['CORR_TYPE']
2020-05-01 19:27:20 CARACal WARNING: The plotter 'shadems' cannot make the plot 'amp_ant'
2020-05-01 19:27:22 CARACal WARNING: The plotter 'shadems' cannot make the plot 'amp_scan'
2020-05-01 19:27:25 CARACal INFO: inspect: running
2020-05-01 19:27:25 CARACal.Stimela.plot-real_imag-0-0 INFO: job started at 2020-05-01 19:27:25.757633
2020-05-01 19:27:25 CARACal.Stimela.plot-real_imag-0-0 ERROR: Expecting any of types [<class 'str'>] for parameter 'corr', but got 'int'
2020-05-01 19:27:25 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-01 19:27:25.758125 after 0:00:00.000492
2020-05-01 19:27:25 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected ' failed: Expecting any of types [<class 'str'>] for parameter 'corr', but got 'int' [PipelineException]
2020-05-01 19:27:25 CARACal INFO:   More information can be found in the logfile at output/log-caracal-20200501-192152.txt
2020-05-01 19:27:25 CARACal INFO: exiting with error code 1
gigjozsa commented 4 years ago

Culprit is this: https://github.com/ska-sa/caracal/blob/c637b2aadaae318e5381a74bb318490460733fcf/caracal/workers/inspect_worker.py#L218

@SpheMakh @Athanaseus @o-smirnov could anyone working on the inspect worker correct this (if not done already)?

o-smirnov commented 4 years ago

Please replace with:

opts["corr"] = co

On my harlequin branch this is all moot anyway.

KshitijT commented 4 years ago

Please replace with:

opts["corr"] = co

On my harlequin branch this is all moot anyway.

Tried this, get another error down the road:

# 2020-05-03 01:15:04 - shadems - INFO - :   corrs/Stokes XX XY YX YY I Q U V
# 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------
# 2020-05-03 01:15:04 - shadems - INFO - : Data selected for plotting:
# 2020-05-03 01:15:04 - shadems - INFO - Antenna(s)       : all
# 2020-05-03 01:15:04 - shadems - INFO - Baseline(s)      : all
# 2020-05-03 01:15:04 - shadems - INFO - Field(s)         : J1939-6342
# 2020-05-03 01:15:04 - shadems - INFO - SPW(s)           : all
# 2020-05-03 01:15:04 - shadems - INFO - Scan(s)          : all
# 2020-05-03 01:15:04 - shadems - INFO - Channels         : all
# 2020-05-03 01:15:04 - shadems - INFO - Corr/Stokes      : XX
# 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------
# Traceback (most recent call last):
#   File "/usr/local/bin/shadems", line 8, in <module>
#     main.main([a for a in sys.argv[1:]])
#   File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 423, in main
#     xspecs = DataAxis.parse_datum_spec(xaxis, default_column, ms=ms)
#   File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_mappers.py", line 115, in parse_datum_spec
#     raise ValueError(f"invalid axis specification '{axis_spec}'")
# ValueError: invalid axis specification 'r'
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 43, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['shadems', '--xaxis', 'r', '--yaxis', 'i', '--col', 'CORRECTED_DATA', '--ant', 'all', '--baseline', 'all', '--spw', 'all', '--field', '0', '--scan', 'all', '--corr', '0', '--xcanvas', '1280', '--ycanvas', '900', '--norm', 'auto', '--cmap', 'bkr', '--bmap', 'bkr', '--dmap', 'glasbey_dark', '--spread-pix', '0', '--spread-thr', '0.5', '--bgcol', 'FFFFFF', '--png', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX.png', '--title', '{ms}{_field}{_Spw}{_Scan}{_Ant}{_title}{_Alphatitle}{_Colortitle}', '--xlabel', '{xname}{_xunit}', '--ylabel', '{yname}{_yunit}', '--row-chunk-size', '100000', '--num-parallel', '1', '/stimela_mount/msdir/1565287561_sdp_l0.ms']' returned non-zero exit status 1.
2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1
2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:15:05.499307 after 0:00:19.223499
2020-05-03 01:15:05 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected ' failed: singularity returns error code 1 [PipelineException]
2020-05-03 01:15:05 CARACal INFO:   More information can be found in the logfile at output_caracal/log-caracal-20200503-011221.txt
2020-05-03 01:15:05 CARACal INFO: exiting with error code 1
o-smirnov commented 4 years ago

Set standard_plotter to ragavi. Shadems is only called in the new way for now.

Cheers, Oleg

Sent from my phone. Quality of spelling inversely proportional to finger size.

On Sun, 03 May 2020, 01:22 Kshitij Thorat, notifications@github.com wrote:

Please replace with:

opts["corr"] = co

On my harlequin branch this is all moot anyway.

Tried this, get another error down the road:

2020-05-03 01:15:04 - shadems - INFO - : corrs/Stokes XX XY YX YY I Q U V

2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------

2020-05-03 01:15:04 - shadems - INFO - : Data selected for plotting:

2020-05-03 01:15:04 - shadems - INFO - Antenna(s) : all

2020-05-03 01:15:04 - shadems - INFO - Baseline(s) : all

2020-05-03 01:15:04 - shadems - INFO - Field(s) : J1939-6342

2020-05-03 01:15:04 - shadems - INFO - SPW(s) : all

2020-05-03 01:15:04 - shadems - INFO - Scan(s) : all

2020-05-03 01:15:04 - shadems - INFO - Channels : all

2020-05-03 01:15:04 - shadems - INFO - Corr/Stokes : XX

2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------

Traceback (most recent call last):

File "/usr/local/bin/shadems", line 8, in

main.main([a for a in sys.argv[1:]])

File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 423, in main

xspecs = DataAxis.parse_datum_spec(xaxis, default_column, ms=ms)

File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_mappers.py", line 115, in parse_datum_spec

raise ValueError(f"invalid axis specification '{axis_spec}'")

ValueError: invalid axis specification 'r'

Traceback (most recent call last):

File "/stimela_mount/code/run.py", line 43, in

subprocess.check_call(shlex.split(_runc))

File "/usr/lib/python3.6/subprocess.py", line 311, in check_call

raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['shadems', '--xaxis', 'r', '--yaxis', 'i', '--col', 'CORRECTED_DATA', '--ant', 'all', '--baseline', 'all', '--spw', 'all', '--field', '0', '--scan', 'all', '--corr', '0', '--xcanvas', '1280', '--ycanvas', '900', '--norm', 'auto', '--cmap', 'bkr', '--bmap', 'bkr', '--dmap', 'glasbey_dark', '--spread-pix', '0', '--spread-thr', '0.5', '--bgcol', 'FFFFFF', '--png', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX.png', '--title', '{ms}{_field}{_Spw}{_Scan}{_Ant}{_title}{_Alphatitle}{_Colortitle}', '--xlabel', '{xname}{_xunit}', '--ylabel', '{yname}{_yunit}', '--row-chunk-size', '100000', '--num-parallel', '1', '/stimela_mount/msdir/1565287561_sdp_l0.ms']' returned non-zero exit status 1.

2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1 2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:15:05.499307 after 0:00:19.223499 2020-05-03 01:15:05 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected ' failed: singularity returns error code 1 [PipelineException] 2020-05-03 01:15:05 CARACal INFO: More information can be found in the logfile at output_caracal/log-caracal-20200503-011221.txt 2020-05-03 01:15:05 CARACal INFO: exiting with error code 1

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/caracal/issues/1037#issuecomment-623028494, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTP4XZATUUJ2YQCGRB73RPSTM7ANCNFSM4MXILRWQ .

gigjozsa commented 4 years ago

This is done in the harlequin branch, right?

o-smirnov commented 4 years ago

By default yes.

Cheers, Oleg

Sent from my phone. Quality of spelling inversely proportional to finger size.

On Sun, 03 May 2020, 10:33 Gyula Jozsa (Josh), notifications@github.com wrote:

This is done in the harlequin branch, right?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/caracal/issues/1037#issuecomment-623074723, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTPYABSDBKDMNVVLMX3LRPUT4LANCNFSM4MXILRWQ .

KshitijT commented 4 years ago

Set standard_plotter to ragavi. Shadems is only called in the new way for now. Cheers, Oleg … --- Sent from my phone. Quality of spelling inversely proportional to finger size. On Sun, 03 May 2020, 01:22 Kshitij Thorat, @.***> wrote: Please replace with: opts["corr"] = co On my harlequin branch this is all moot anyway. Tried this, get another error down the road: # 2020-05-03 01:15:04 - shadems - INFO - : corrs/Stokes XX XY YX YY I Q U V # 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------ # 2020-05-03 01:15:04 - shadems - INFO - : Data selected for plotting: # 2020-05-03 01:15:04 - shadems - INFO - Antenna(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Baseline(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Field(s) : J1939-6342 # 2020-05-03 01:15:04 - shadems - INFO - SPW(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Scan(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Channels : all # 2020-05-03 01:15:04 - shadems - INFO - Corr/Stokes : XX # 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------ # Traceback (most recent call last): # File "/usr/local/bin/shadems", line 8, in # main.main([a for a in sys.argv[1:]]) # File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 423, in main # xspecs = DataAxis.parse_datum_spec(xaxis, default_column, ms=ms) # File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_mappers.py", line 115, in parse_datum_spec # raise ValueError(f"invalid axis specification '{axis_spec}'") # ValueError: invalid axis specification 'r' # Traceback (most recent call last): # File "/stimela_mount/code/run.py", line 43, in # subprocess.check_call(shlex.split(_runc)) # File "/usr/lib/python3.6/subprocess.py", line 311, in check_call # raise CalledProcessError(retcode, cmd) # subprocess.CalledProcessError: Command '['shadems', '--xaxis', 'r', '--yaxis', 'i', '--col', 'CORRECTED_DATA', '--ant', 'all', '--baseline', 'all', '--spw', 'all', '--field', '0', '--scan', 'all', '--corr', '0', '--xcanvas', '1280', '--ycanvas', '900', '--norm', 'auto', '--cmap', 'bkr', '--bmap', 'bkr', '--dmap', 'glasbey_dark', '--spread-pix', '0', '--spread-thr', '0.5', '--bgcol', 'FFFFFF', '--png', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX.png', '--title', '{ms}{_field}{_Spw}{_Scan}{_Ant}{_title}{_Alphatitle}{_Colortitle}', '--xlabel', '{xname}{_xunit}', '--ylabel', '{yname}{_yunit}', '--row-chunk-size', '100000', '--num-parallel', '1', '/stimela_mount/msdir/1565287561_sdp_l0.ms']' returned non-zero exit status 1. 2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1 2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:15:05.499307 after 0:00:19.223499 2020-05-03 01:15:05 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected ' failed: singularity returns error code 1 [PipelineException] 2020-05-03 01:15:05 CARACal INFO: More information can be found in the logfile at output_caracal/log-caracal-20200503-011221.txt 2020-05-03 01:15:05 CARACal INFO: exiting with error code 1 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1037 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTP4XZATUUJ2YQCGRB73RPSTM7ANCNFSM4MXILRWQ .

I did that, but end up with the following error:

# Traceback (most recent call last):
#   File "/usr/local/bin/ragavi-vis", line 11, in <module>
#     main(options=options)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1440, in main
#     colour_axis=colour_axis)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1169, in hv_plotter
#     ms_name=ms_name, xds_table_obj=xds_table_obj)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 906, in gen_grid
#     s_agg = agg.sel(**{cat: c_val})
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataarray.py", line 1061, in sel
#     **indexers_kwargs,
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataset.py", line 2066, in sel
#     self, indexers=indexers, method=method, tolerance=tolerance
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/coordinates.py", line 397, in remap_label_indexers
#     obj, v_indexers, method=method, tolerance=tolerance
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 270, in remap_label_indexers
#     idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 190, in convert_label_indexer
#     label.item(), method=method, tolerance=tolerance
#   File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2648, in get_loc
#     return self._engine.get_loc(self._maybe_cast_indexer(key))
#   File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
#   File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
#   File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
#   File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
# KeyError: 18
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 38, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-vis', '--ms', '/stimela_mount/msdir/1565287561_sdp_l0.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1
2020-05-03 01:34:47 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1
2020-05-03 01:34:47 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:34:47.818132 after 0:09:07.729361
2020-05-03 01:34:47 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected real_imag' failed: singularity returns error code 1 [PipelineException]

This is on singularity and on master branch, not the harlequin branch, colourful though that sounds.

Maybe I should open a different issue, unless this is already addressed in the harlequin branch, in which case I'll just switch to that, @o-smirnov ?

gigjozsa commented 4 years ago

I'm testing this now on the harlequin branch

o-smirnov commented 4 years ago

This looks like something deep inside ragavi. I suggest opening a new issue for this and assigning to @Mulan-94. @KshitijT, could you please also paste the ragavi-vis command, which will be just a bit above in the log.

KshitijT commented 4 years ago

@o-smirnov , I guess this is the command:

subprocess.CalledProcessError: Command '['ragavi-vis', '--ms', '/stimela_mount/msdir/1565287561_sdp_l0.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1
o-smirnov commented 4 years ago

Effectively, yes -- although I think (hope) it prints the whole thing in slightly more readable form earlier.

KshitijT commented 4 years ago

Erm, no, all it prints at the start is:

job started at 2020-05-03 01:25:40.088771
Starting container [plot_real_imag_0_0-14021650460760015884619168365507]. Timeout set to -1. The container ID is printed below.
running singularity run --workdir /stimela_mount/output --home /users/kshitij:/stimela_home  --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/stimela_parameter_files/plot_real_imag_0_0-14021650460760015884619168365507.json:/stimela_mount/configfile:ro --bind /scratch/users/kshitij/Stimela/stimela/cargo/cab/ragavi_vis/src:/stimela_mount/code:ro --bind /scratch/users/kshitij/caracal-venv/bin/stimela_runscript:/singularity:ro --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/msdir:/stimela_mount/msdir:rw --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/input:/stimela_mount/input:ro --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/output_caracal/diagnostic_plots/crosscal:/stimela_mount/output:rw --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/output_caracal/diagnostic_plots/crosscal/tmp:/stimela_mount/output/tmp:rw ../../STIMELA_IMAGES//stimela_ragavi_1.2.6.sif /singularity
03.05.2020@01:25:48 - ragavi       - INFO       - Chunk sizes: (100000, 4096, 4)
03.05.2020@01:25:48 - ragavi       - INFO       - Number of Partitions: 6
03.05.2020@01:25:48 - ragavi       - INFO       - Starting data processing.
03.05.2020@01:25:48 - ragavi       - INFO       - Starting plotting function.
03.05.2020@01:25:48 - ragavi       - INFO       - Calculating x and y Min and Max ranges

[                                        ] | 0% Completed |  0.0s
[                                        ] | 0% Completed | 20.7s
[#                                       ] | 2% Completed | 43.8s
[#                                       ] | 2% Completed | 45.1s
[#                                       ] | 3% Completed | 46.6s
[#                                       ] | 3% Completed |  1min 10.7s
[##                                      ] | 5% Completed |  1min 32.6s
[#####                                   ] | 12% Completed |  1min 55.8s
[#####                                   ] | 12% Completed |  2min 14.8s
[###########                             ] | 27% Completed |  2min 14.9s
gigjozsa commented 4 years ago

No, Kshitij, this is a new issue.

KshitijT commented 4 years ago

No, Kshitij, this is a new issue.

Yeah, I have opened a new issue : #1041 .