Closed KshitijT closed 4 years ago
I think 1Gb is the "aspirational" limit, it's not strictly enforced.
If you're feeling adventurous, could you try repeating the same ragavi command outside the container?
I think 1Gb is the "aspirational" limit, it's not strictly enforced.
If you're feeling adventurous, could you try repeating the same ragavi command outside the container?
I was born adventurous ;). Will do.
Ok, so I tested with standalone Ragavi as:
ragavi-vis --ms msdir/1565287561_sdp_l0.ms --xaxis real --yaxis imaginary --canvas-height 300 --canvas-width 300 --cmap blues --corr 0 --data-column CORRECTED_DATA --field 0 --htmlname output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX_test --iter-axis scan --mem-limit 1GB
Got a different error:
[########################################] | 100% Completed | 2min 25.4s
03.05.2020@22:16:20 - ragavi - INFO - Done
03.05.2020@22:16:20 - ragavi - INFO - Creating Dataframe
03.05.2020@22:20:01 - ragavi - INFO - Launching datashader
03.05.2020@22:20:01 - ragavi - INFO - Datashader aggregation starting
03.05.2020@22:20:01 - ragavi - ERROR - Oops ... !
Traceback (most recent call last):
File "/scratch/users/kshitij/caracal-venv/bin/ragavi-vis", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/scratch/users/kshitij/ragavi/ragavi/bin/ragavi-vis", line 11, in <module>
main(options=options)
File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 1440, in main
colour_axis=colour_axis)
File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 1169, in hv_plotter
ms_name=ms_name, xds_table_obj=xds_table_obj)
File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 853, in gen_grid
y=y_name, cat=cat)
File "/scratch/users/kshitij/ragavi/ragavi/utils.py", line 583, in timer
ans = func(*args, **kwargs)
File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 769, in image_callback
agg = cvs.points(xy_df, x, y, ds.count_cat(cat))
File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/core.py", line 224, in points
return bypixel(source, self, glyph, agg)
File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/core.py", line 1192, in bypixel
return bypixel.pipeline(source, schema, canvas, glyph, agg)
File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/utils.py", line 94, in __call__
return lk[typ](head, *rest, **kwargs)
File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/data_libraries/dask.py", line 21, in dask_pipeline
dsk, name = glyph_dispatch(glyph, df, schema, canvas, summary, cuda=cuda)
File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/utils.py", line 97, in __call__
return lk[cls](head, *rest, **kwargs)
File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/data_libraries/dask.py", line 82, in default
dtype = np.result_type(*df.dtypes)
File "<__array_function__ internals>", line 6, in result_type
TypeError: data type not understood
Hey @KshitijT , could you retry this with the current dev branch in ragavi please?
Hey @KshitijT , could you retry this with the current dev branch in ragavi please?
Will do.
P.S. @Mulan-94 , I get this (non-critical?) error while installing the dev branch:
ERROR: xova 0.1.1 has requirement dask-ms>=0.2.4, but you'll have dask-ms 0.2.3 which is incompatible.
Is the correct version of dask being installed?
@KshitijT Yes but xova is not a dependency now ...
Let me try again with a fresh version.
Mine just fell over with:
2020-05-06 21:28:50 CARACal.Stimela.plot-shadems-ms0 INFO: job complete at 2020-05-06 21:28:50.937517 after 0:58:31.765331
2020-05-06 21:28:50 CARACal.Stimela.plot-amp_uvwave-0-0 INFO: job started at 2020-05-06 21:28:50.937775
# 0abd8d41b49076692bb13dd3c292d9df8c8ea8a5361954d410e5f050fcd240bb
# usage: ragavi-vis [options] <value>
# ragavi-vis: error: unrecognized arguments: --cols 5
# Traceback (most recent call last):
Same problem, new problem?
Looks like new problem, went to master stimela and rebuilt the cab, still getting the error above.
And now falling over in ragavi-gains
as well:
# b49c89c24233c60e58c4c47221a8eabb0822be6e919d3edcadc78f6529a35f83
# 06.05.2020@20:37:06 - ragavi - ERROR - Oops ... !
# Traceback (most recent call last):
# File "/usr/local/bin/ragavi-gains", line 19, in <module>
# main(options=options)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/ragavi.py", line 1485, in main
# options.fields.isnumeric()):
# AttributeError: 'list' object has no attribute 'isnumeric'
# Traceback (most recent call last):
# File "/stimela_mount/code/run.py", line 38, in <module>
# subprocess.check_call(shlex.split(_runc))
# File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
# raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-gains', '--table', '/stimela_mount/msdir/pica-1576687564_sdp_l0-1gc1_primary_cal.K0', '--gaintype', 'K', '--cmap', 'coolwarm', '--doplot', 'ap', '--field', '0', '--htmlname', '/stimela_mount/output/pica-1576687564_sdp_l0-1gc1_primary_cal.K0', '--k-xaxis', 'time']' returned non-zero exit status 1
2020-05-06 22:37:06 CARACal.Stimela.plotgains-K-0-0 ERROR: docker returns error code 1
@o-smirnov is this problem with the ragavi:1.4.4
image?
Mine just fell over with:
2020-05-06 21:28:50 CARACal.Stimela.plot-shadems-ms0 INFO: job complete at 2020-05-06 21:28:50.937517 after 0:58:31.765331 2020-05-06 21:28:50 CARACal.Stimela.plot-amp_uvwave-0-0 INFO: job started at 2020-05-06 21:28:50.937775 # 0abd8d41b49076692bb13dd3c292d9df8c8ea8a5361954d410e5f050fcd240bb # usage: ragavi-vis [options] <value> # ragavi-vis: error: unrecognized arguments: --cols 5 # Traceback (most recent call last):
I also have some ragavi error crashing my test run (using the workflow_new.yml). Sorry, I can't make any sense of it, so I'll just report here.
# 07.05.2020@07:34:38 - ragavi - INFO - Launching datashader
# 07.05.2020@07:34:38 - ragavi - INFO - Datashader aggregation starting
#
# [ ] | 0% Completed | 0.0s
# [############################### ] | 77% Completed | 0.1s
# [################################### ] | 88% Completed | 0.2s
# [################################### ] | 88% Completed | 0.4s
# [################################### ] | 88% Completed | 0.5s
# [################################### ] | 88% Completed | 0.6s
# [################################### ] | 88% Completed | 0.7s
# [########################################] | 100% Completed | 0.8s
# 07.05.2020@07:34:39 - ragavi - INFO - Aggregation done
# 07.05.2020@07:34:39 - ragavi - INFO - image_callback executed in: 0.8952 sec.
# 07.05.2020@07:34:39 - ragavi - INFO - Creating Bokeh grid
# 07.05.2020@07:34:39 - ragavi - ERROR - Oops ... !
# Traceback (most recent call last):
# File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2646, in get_loc
# return self._engine.get_loc(key)
# File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
# File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
# File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
# File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
# KeyError: 2
#
# During handling of the above exception, another exception occurred:
#
# Traceback (most recent call last):
# File "/usr/local/bin/ragavi-vis", line 11, in <module>
# main(options=options)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1440, in main
# colour_axis=colour_axis)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1169, in hv_plotter
# ms_name=ms_name, xds_table_obj=xds_table_obj)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 906, in gen_grid
# s_agg = agg.sel(**{cat: c_val})
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataarray.py", line 1061, in sel
# **indexers_kwargs,
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataset.py", line 2066, in sel
# self, indexers=indexers, method=method, tolerance=tolerance
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/coordinates.py", line 397, in remap_label_indexers
# obj, v_indexers, method=method, tolerance=tolerance
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 270, in remap_label_indexers
# idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
# File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 190, in convert_label_indexer
# label.item(), method=method, tolerance=tolerance
# File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2648, in get_loc
# return self._engine.get_loc(self._maybe_cast_indexer(key))
# File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
# File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
# File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
# File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
# KeyError: 2
# Traceback (most recent call last):
# File "/stimela_mount/code/run.py", line 38, in <module>
# subprocess.check_call(shlex.split(_runc))
# File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
# raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-vis', '--ms', '/stimela_mount/msdir/gps1_cal.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '1', '--htmlname', '/stimela_mount/output/workflow2-gps1-1gc-J0825-5010-real_imag-gcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1
2020-05-07 07:34:40 CARACal.Stimela.plot-real_imag-0-1 ERROR: docker returns error code 1
2020-05-07 07:34:40 CARACal.Stimela.plot-real_imag-0-1 ERROR: job failed at 2020-05-07 07:34:40.396482 after 0:00:08.154299
2020-05-07 07:34:40 CARACal ERROR: Job 'plot-real_imag-0-1:: Plotting corrected real_imag' failed: docker returns error code 1 [PipelineException]
2020-05-07 07:34:40 CARACal INFO: More information can be found in the logfile at output/logs-20200507-073041/log-caracal.txt
2020-05-07 07:34:40 CARACal INFO: exiting with error code 1
(caracal_venv) dmolnar@escondida:/24TB/dmolnar/caracal_prerelease_tests/test_tiny$ nano output/logs-20200507-073041/log-caracal.txt
GNU nano 2.9.3 output/logs-20200507-073041/log-caracal.txt Modified
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_uvwave-0-0:: Plotting corrected phase_uvwave
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_uvwave-0-1:: Plotting corrected phase_uvwave
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-0:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-1:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-0:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-1:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-0:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-1:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-0:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-1:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-0:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-1:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-0:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-1:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Saving pipeline information in .last_inspect.json
2020-05-07 07:34:40 CARACal ERROR: Job 'plot-real_imag-0-1:: Plotting corrected real_imag' failed: docker returns error code 1 [PipelineException]
2020-05-07 07:34:40 CARACal INFO: More information can be found in the logfile at output/logs-20200507-073041/log-caracal.txt
2020-05-07 07:34:40 CARACal INFO: You are running version 0.1.0-2271-gb8d236b
2020-05-07 07:34:40 CARACal ERROR: Traceback (most recent call last):
2020-05-07 07:34:40 CARACal ERROR: File "/home/dmolnar/caracal_test/caracal/caracal/main.py", line 241, in __run
2020-05-07 07:34:40 CARACal ERROR: pipeline.run_workers()
2020-05-07 07:34:40 CARACal ERROR: File "/home/dmolnar/caracal_test/caracal/caracal/workers/worker_administrator.py", line 316, in run_workers
2020-05-07 07:34:40 CARACal ERROR: recipe.run()
2020-05-07 07:34:40 CARACal ERROR: File "/home/dmolnar/caracal_test/caracal_venv/lib/python3.6/site-packages/stimela/recipe.py", line 681, in run
2020-05-07 07:34:40 CARACal ERROR: raise PipelineException(exc, self.completed, job, self.remaining) from None
2020-05-07 07:34:40 CARACal ERROR: stimela.exceptions.PipelineException: Job 'plot-real_imag-0-1:: Plotting corrected real_imag' failed: docker returns er$
2020-05-07 07:34:40 CARACal INFO: exiting with error code 1
@molnard89 , sorry about this, which ragavi
image are you using?
@Mulan-94 sorry, I'm ignorant - how do I check it? I'm using the workflow_new.yml from the sample configs on the plant_your_flag branch.
@molnard89 if you're using singularity please check in the folder where you pulled the images for this stimela_ragavi_
the image number is at the end, otherwise for docker, I think this:
docker image ls | grep -i stimela/ragavi
should show you
I have
dmolnar@escondida:~/caracal_test/caracal$ docker image ls | grep -i stimela/ragavi
stimela/ragavi 1.2.6 991ac0ace0bd 4 weeks ago 2.34GB
stimela/ragavi 1.2.3 c20ee154e4b0 5 months ago 2.26GB
stimela/ragavi 1.2.4 c20ee154e4b0 5 months ago 2.26GB
stimela/ragavi 1.2.0 c4eb6f8ac0b2 8 months ago 1.44GB
Great. I would suggest to please update stimela
(if not yet) and then pull the latest ragavi
image. The most recent one is 1.4.4
. However, this image does have some issues which I'm working to fix. There will be a newer version soon...
Hey @KshitijT , could you retry this with the current dev branch in ragavi please?
Ok, I finally (succesfully) ran ragavi with the dev version:
ragavi-vis --ms test.ms/ --xaxis real --yaxis imaginary --canvas-height 300 --canvas-width 300 --cmap blues --data-column DATA --htmlname testdata-real_imag-bpcal-_Corr_XX_test --iter-axis scan --mem-limit 1GB
to get:
Seems to take a long time on the full data - this is just a single scan, done in a couple of minutes. I guess I should increase the mem-limit?
@KshitijT thanks for this. You may try that, as well as the cores and chunk sizes (it may or may not improve things). But also specifying the x and y limits so that they don't have to be calculated will also cut down some of the time (this may take a really long time for large datasets).
Is this version already in stimela?
Is this version already in stimela?
Yes this is the version in stimela
but there is the error that @o-smirnov pointed out so there will be a new image soon.
What is the status?
What is the status?
Waiting for new stimela image, I think.
@Mulan-94 , I am getting another error with the stimela release version of ragavi:
# eca8c5035186a631b813817d80252362feed2040ecb1476ae8be927c21618d44
# 08.05.2020@10:30:32 - ragavi - ERROR - Oops ... !
# Traceback (most recent call last):
# File "/usr/local/bin/ragavi-gains", line 19, in <module>
# main(options=options)
# File "/usr/local/lib/python3.6/dist-packages/ragavi/ragavi.py", line 1485, in main
# options.fields.isnumeric()):
# AttributeError: 'list' object has no attribute 'isnumeric'
# Traceback (most recent call last):
# File "/stimela_mount/code/run.py", line 38, in <module>
# subprocess.check_call(shlex.split(_runc))
# File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
# raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-gains', '--table', '/stimela_mount/msdir/mypipelinerun-1532022061_subset-1gc1_primary_cal.K0', '--gaintype', 'K', '--cmap', 'coolwarm', '--doplot', 'ap', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1532022061_subset-1gc1_primary_cal.K0', '--k-xaxis', 'time']' returned non-zero exit status 1
2020-05-08 10:30:33 CARACal.Stimela.plotgains-K-0-0 ERROR: docker returns error code 1
A similar issue was fixed a while ago, right? Is this something unrelated?
I assume this will fix the above issue too: https://github.com/ratt-ru/Stimela/pull/601 ?
Yes that PR should fix it
More woes: https://github.com/ratt-ru/ragavi/issues/62
@o-smirnov Do we close this or do we only remove the milestone tag?
Woebegone.
Not really nice to reopen this again, but on my laptop running since 20 mins:
2020-05-12 10:12:03 CARACal.Stimela.plotgains-B-0-0 INFO: job started at 2020-05-12 10:12:03.093540
# 4f4125360079e6259ceac32e134a9dab3e9f6c70250ba246a7f6d9557aba6227
# 12.05.2020@08:12:06 - ragavi - INFO - Acquiring table: mypipelinerun-1532022061_subset-1gc1_primary.B0
# 12.05.2020@08:12:06 - ragavi - INFO - Spw: 0, Field: J1331+3030, Corr: 0 amplitude
# 12.05.2020@08:12:10 - ragavi - INFO - Spw: 0, Field: J1331+3030, Corr: 1 amplitude
At this stage we should at least put out a warning...
Stimela release: 1.4.6
Stimela build: a0af56a43950dedf949ebce7ff2a00fa8a278ef4
f091b75b459c2cb31d4c6b057e9f68db46562fb5
from: https://github.com/ratt-ru/Stimela
I see that the release version is wrong.
@SpheMakh could you update the release number in Stimela/setup.py ?
This is slightly urgent I guess... I issued a PR on Stimela, which might fix this: https://github.com/ratt-ru/Stimela/pull/612
It is urgent, but another PR has to go in for a true fix. Give it an hour.
@o-smirnov @SpheMakh ok let me know when I can test again.
Not really nice to reopen this again, but on my laptop running since 20 mins
Docker or Singularity? I've had no problems with ragavi-gains on Docker since Caracal release, but maybe I've been lucky so far?
ragavi-gains has been working well for me since yesterday as well on two different measument sets (docker version).
Well the fix we figured out yesterday actually didn't make it into a Stimela release until now... ooops... so you have been a couple of very lucky boys.
But yes, now that Stimela 1.5.1 is out, another round of hard testing will be good (running for me fine so far...)
Fixed in stimela 1.5.1
When the plotter is set to 'ragavi_vis', I get the following error:
The options are given by the lines (as above, so below):
Just as a curiosity, is 1GB memory enough for this? Putting at defcon-0, lower if @o-smirnov and @Mulan-94 think it is not that dire.