Closed gigjozsa closed 4 years ago
Culprit is this: https://github.com/ska-sa/caracal/blob/c637b2aadaae318e5381a74bb318490460733fcf/caracal/workers/inspect_worker.py#L218
@SpheMakh @Athanaseus @o-smirnov could anyone working on the inspect worker correct this (if not done already)?
Please replace with:
opts["corr"] = co
On my harlequin branch this is all moot anyway.
Please replace with:
opts["corr"] = co
On my harlequin branch this is all moot anyway.
Tried this, get another error down the road:
# 2020-05-03 01:15:04 - shadems - INFO - : corrs/Stokes XX XY YX YY I Q U V
# 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------
# 2020-05-03 01:15:04 - shadems - INFO - : Data selected for plotting:
# 2020-05-03 01:15:04 - shadems - INFO - Antenna(s) : all
# 2020-05-03 01:15:04 - shadems - INFO - Baseline(s) : all
# 2020-05-03 01:15:04 - shadems - INFO - Field(s) : J1939-6342
# 2020-05-03 01:15:04 - shadems - INFO - SPW(s) : all
# 2020-05-03 01:15:04 - shadems - INFO - Scan(s) : all
# 2020-05-03 01:15:04 - shadems - INFO - Channels : all
# 2020-05-03 01:15:04 - shadems - INFO - Corr/Stokes : XX
# 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------
# Traceback (most recent call last):
# File "/usr/local/bin/shadems", line 8, in <module>
# main.main([a for a in sys.argv[1:]])
# File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 423, in main
# xspecs = DataAxis.parse_datum_spec(xaxis, default_column, ms=ms)
# File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_mappers.py", line 115, in parse_datum_spec
# raise ValueError(f"invalid axis specification '{axis_spec}'")
# ValueError: invalid axis specification 'r'
# Traceback (most recent call last):
# File "/stimela_mount/code/run.py", line 43, in <module>
# subprocess.check_call(shlex.split(_runc))
# File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
# raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['shadems', '--xaxis', 'r', '--yaxis', 'i', '--col', 'CORRECTED_DATA', '--ant', 'all', '--baseline', 'all', '--spw', 'all', '--field', '0', '--scan', 'all', '--corr', '0', '--xcanvas', '1280', '--ycanvas', '900', '--norm', 'auto', '--cmap', 'bkr', '--bmap', 'bkr', '--dmap', 'glasbey_dark', '--spread-pix', '0', '--spread-thr', '0.5', '--bgcol', 'FFFFFF', '--png', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX.png', '--title', '{ms}{_field}{_Spw}{_Scan}{_Ant}{_title}{_Alphatitle}{_Colortitle}', '--xlabel', '{xname}{_xunit}', '--ylabel', '{yname}{_yunit}', '--row-chunk-size', '100000', '--num-parallel', '1', '/stimela_mount/msdir/1565287561_sdp_l0.ms']' returned non-zero exit status 1.
2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1
2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:15:05.499307 after 0:00:19.223499
2020-05-03 01:15:05 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected ' failed: singularity returns error code 1 [PipelineException]
2020-05-03 01:15:05 CARACal INFO: More information can be found in the logfile at output_caracal/log-caracal-20200503-011221.txt
2020-05-03 01:15:05 CARACal INFO: exiting with error code 1
Set standard_plotter to ragavi. Shadems is only called in the new way for now.
Cheers, Oleg
Sent from my phone. Quality of spelling inversely proportional to finger size.
On Sun, 03 May 2020, 01:22 Kshitij Thorat, notifications@github.com wrote:
Please replace with:
opts["corr"] = co
On my harlequin branch this is all moot anyway.
Tried this, get another error down the road:
2020-05-03 01:15:04 - shadems - INFO - : corrs/Stokes XX XY YX YY I Q U V
2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------
2020-05-03 01:15:04 - shadems - INFO - : Data selected for plotting:
2020-05-03 01:15:04 - shadems - INFO - Antenna(s) : all
2020-05-03 01:15:04 - shadems - INFO - Baseline(s) : all
2020-05-03 01:15:04 - shadems - INFO - Field(s) : J1939-6342
2020-05-03 01:15:04 - shadems - INFO - SPW(s) : all
2020-05-03 01:15:04 - shadems - INFO - Scan(s) : all
2020-05-03 01:15:04 - shadems - INFO - Channels : all
2020-05-03 01:15:04 - shadems - INFO - Corr/Stokes : XX
2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/bin/shadems", line 8, in
main.main([a for a in sys.argv[1:]])
File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 423, in main
xspecs = DataAxis.parse_datum_spec(xaxis, default_column, ms=ms)
File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_mappers.py", line 115, in parse_datum_spec
raise ValueError(f"invalid axis specification '{axis_spec}'")
ValueError: invalid axis specification 'r'
Traceback (most recent call last):
File "/stimela_mount/code/run.py", line 43, in
subprocess.check_call(shlex.split(_runc))
File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['shadems', '--xaxis', 'r', '--yaxis', 'i', '--col', 'CORRECTED_DATA', '--ant', 'all', '--baseline', 'all', '--spw', 'all', '--field', '0', '--scan', 'all', '--corr', '0', '--xcanvas', '1280', '--ycanvas', '900', '--norm', 'auto', '--cmap', 'bkr', '--bmap', 'bkr', '--dmap', 'glasbey_dark', '--spread-pix', '0', '--spread-thr', '0.5', '--bgcol', 'FFFFFF', '--png', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX.png', '--title', '{ms}{_field}{_Spw}{_Scan}{_Ant}{_title}{_Alphatitle}{_Colortitle}', '--xlabel', '{xname}{_xunit}', '--ylabel', '{yname}{_yunit}', '--row-chunk-size', '100000', '--num-parallel', '1', '/stimela_mount/msdir/1565287561_sdp_l0.ms']' returned non-zero exit status 1.
2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1 2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:15:05.499307 after 0:00:19.223499 2020-05-03 01:15:05 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected ' failed: singularity returns error code 1 [PipelineException] 2020-05-03 01:15:05 CARACal INFO: More information can be found in the logfile at output_caracal/log-caracal-20200503-011221.txt 2020-05-03 01:15:05 CARACal INFO: exiting with error code 1
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/caracal/issues/1037#issuecomment-623028494, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTP4XZATUUJ2YQCGRB73RPSTM7ANCNFSM4MXILRWQ .
This is done in the harlequin branch, right?
By default yes.
Cheers, Oleg
Sent from my phone. Quality of spelling inversely proportional to finger size.
On Sun, 03 May 2020, 10:33 Gyula Jozsa (Josh), notifications@github.com wrote:
This is done in the harlequin branch, right?
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/caracal/issues/1037#issuecomment-623074723, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTPYABSDBKDMNVVLMX3LRPUT4LANCNFSM4MXILRWQ .
Set standard_plotter to ragavi. Shadems is only called in the new way for now. Cheers, Oleg … --- Sent from my phone. Quality of spelling inversely proportional to finger size. On Sun, 03 May 2020, 01:22 Kshitij Thorat, @.***> wrote: Please replace with: opts["corr"] = co On my harlequin branch this is all moot anyway. Tried this, get another error down the road: # 2020-05-03 01:15:04 - shadems - INFO - : corrs/Stokes XX XY YX YY I Q U V # 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------ # 2020-05-03 01:15:04 - shadems - INFO - : Data selected for plotting: # 2020-05-03 01:15:04 - shadems - INFO - Antenna(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Baseline(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Field(s) : J1939-6342 # 2020-05-03 01:15:04 - shadems - INFO - SPW(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Scan(s) : all # 2020-05-03 01:15:04 - shadems - INFO - Channels : all # 2020-05-03 01:15:04 - shadems - INFO - Corr/Stokes : XX # 2020-05-03 01:15:04 - shadems - INFO - ------------------------------------------------------ # Traceback (most recent call last): # File "/usr/local/bin/shadems", line 8, in
# main.main([a for a in sys.argv[1:]]) # File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 423, in main # xspecs = DataAxis.parse_datum_spec(xaxis, default_column, ms=ms) # File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_mappers.py", line 115, in parse_datum_spec # raise ValueError(f"invalid axis specification '{axis_spec}'") # ValueError: invalid axis specification 'r' # Traceback (most recent call last): # File "/stimela_mount/code/run.py", line 43, in # subprocess.check_call(shlex.split(_runc)) # File "/usr/lib/python3.6/subprocess.py", line 311, in check_call # raise CalledProcessError(retcode, cmd) # subprocess.CalledProcessError: Command '['shadems', '--xaxis', 'r', '--yaxis', 'i', '--col', 'CORRECTED_DATA', '--ant', 'all', '--baseline', 'all', '--spw', 'all', '--field', '0', '--scan', 'all', '--corr', '0', '--xcanvas', '1280', '--ycanvas', '900', '--norm', 'auto', '--cmap', 'bkr', '--bmap', 'bkr', '--dmap', 'glasbey_dark', '--spread-pix', '0', '--spread-thr', '0.5', '--bgcol', 'FFFFFF', '--png', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX.png', '--title', '{ms}{_field}{_Spw}{_Scan}{_Ant}{_title}{_Alphatitle}{_Colortitle}', '--xlabel', '{xname}{_xunit}', '--ylabel', '{yname}{_yunit}', '--row-chunk-size', '100000', '--num-parallel', '1', '/stimela_mount/msdir/1565287561_sdp_l0.ms']' returned non-zero exit status 1. 2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1 2020-05-03 01:15:05 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:15:05.499307 after 0:00:19.223499 2020-05-03 01:15:05 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected ' failed: singularity returns error code 1 [PipelineException] 2020-05-03 01:15:05 CARACal INFO: More information can be found in the logfile at output_caracal/log-caracal-20200503-011221.txt 2020-05-03 01:15:05 CARACal INFO: exiting with error code 1 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1037 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTP4XZATUUJ2YQCGRB73RPSTM7ANCNFSM4MXILRWQ .
I did that, but end up with the following error:
# Traceback (most recent call last):
# File "/usr/local/bin/ragavi-vis", line 11, in <module>
# main(options=options)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1440, in main
# colour_axis=colour_axis)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1169, in hv_plotter
# ms_name=ms_name, xds_table_obj=xds_table_obj)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 906, in gen_grid
# s_agg = agg.sel(**{cat: c_val})
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataarray.py", line 1061, in sel
# **indexers_kwargs,
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataset.py", line 2066, in sel
# self, indexers=indexers, method=method, tolerance=tolerance
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/coordinates.py", line 397, in remap_label_indexers
# obj, v_indexers, method=method, tolerance=tolerance
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 270, in remap_label_indexers
# idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 190, in convert_label_indexer
# label.item(), method=method, tolerance=tolerance
# File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2648, in get_loc
# return self._engine.get_loc(self._maybe_cast_indexer(key))
# File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
# File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
# File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
# File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
# KeyError: 18
# Traceback (most recent call last):
# File "/stimela_mount/code/run.py", line 38, in <module>
# subprocess.check_call(shlex.split(_runc))
# File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
# raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-vis', '--ms', '/stimela_mount/msdir/1565287561_sdp_l0.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1
2020-05-03 01:34:47 CARACal.Stimela.plot-real_imag-0-0 ERROR: singularity returns error code 1
2020-05-03 01:34:47 CARACal.Stimela.plot-real_imag-0-0 ERROR: job failed at 2020-05-03 01:34:47.818132 after 0:09:07.729361
2020-05-03 01:34:47 CARACal ERROR: Job 'plot-real_imag-0-0:: Plotting corrected real_imag' failed: singularity returns error code 1 [PipelineException]
This is on singularity and on master branch, not the harlequin branch, colourful though that sounds.
Maybe I should open a different issue, unless this is already addressed in the harlequin branch, in which case I'll just switch to that, @o-smirnov ?
I'm testing this now on the harlequin branch
This looks like something deep inside ragavi. I suggest opening a new issue for this and assigning to @Mulan-94. @KshitijT, could you please also paste the ragavi-vis command, which will be just a bit above in the log.
@o-smirnov , I guess this is the command:
subprocess.CalledProcessError: Command '['ragavi-vis', '--ms', '/stimela_mount/msdir/1565287561_sdp_l0.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1
Effectively, yes -- although I think (hope) it prints the whole thing in slightly more readable form earlier.
Erm, no, all it prints at the start is:
job started at 2020-05-03 01:25:40.088771
Starting container [plot_real_imag_0_0-14021650460760015884619168365507]. Timeout set to -1. The container ID is printed below.
running singularity run --workdir /stimela_mount/output --home /users/kshitij:/stimela_home --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/stimela_parameter_files/plot_real_imag_0_0-14021650460760015884619168365507.json:/stimela_mount/configfile:ro --bind /scratch/users/kshitij/Stimela/stimela/cargo/cab/ragavi_vis/src:/stimela_mount/code:ro --bind /scratch/users/kshitij/caracal-venv/bin/stimela_runscript:/singularity:ro --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/msdir:/stimela_mount/msdir:rw --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/input:/stimela_mount/input:ro --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/output_caracal/diagnostic_plots/crosscal:/stimela_mount/output:rw --bind /scratch/users/kshitij/X-shaped-galaxy/3C403/output_caracal/diagnostic_plots/crosscal/tmp:/stimela_mount/output/tmp:rw ../../STIMELA_IMAGES//stimela_ragavi_1.2.6.sif /singularity
03.05.2020@01:25:48 - ragavi - INFO - Chunk sizes: (100000, 4096, 4)
03.05.2020@01:25:48 - ragavi - INFO - Number of Partitions: 6
03.05.2020@01:25:48 - ragavi - INFO - Starting data processing.
03.05.2020@01:25:48 - ragavi - INFO - Starting plotting function.
03.05.2020@01:25:48 - ragavi - INFO - Calculating x and y Min and Max ranges
[ ] | 0% Completed | 0.0s
[ ] | 0% Completed | 20.7s
[# ] | 2% Completed | 43.8s
[# ] | 2% Completed | 45.1s
[# ] | 3% Completed | 46.6s
[# ] | 3% Completed | 1min 10.7s
[## ] | 5% Completed | 1min 32.6s
[##### ] | 12% Completed | 1min 55.8s
[##### ] | 12% Completed | 2min 14.8s
[########### ] | 27% Completed | 2min 14.9s
No, Kshitij, this is a new issue.
No, Kshitij, this is a new issue.
Yeah, I have opened a new issue : #1041 .
This: