search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

Ragavi plotting woes #1041

Closed KshitijT closed 4 years ago

KshitijT commented 4 years ago

When the plotter is set to 'ragavi_vis', I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/ragavi-vis", line 11, in <module>
    main(options=options)
  File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1440, in main
    colour_axis=colour_axis)
  File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1169, in hv_plotter
    ms_name=ms_name, xds_table_obj=xds_table_obj)
  File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 906, in gen_grid
    s_agg = agg.sel(**{cat: c_val})
  File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataarray.py", line 1061, in sel
    **indexers_kwargs,
  File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataset.py", line 2066, in sel
    self, indexers=indexers, method=method, tolerance=tolerance
  File "/usr/local/lib/python3.6/dist-packages/xarray/core/coordinates.py", line 397, in remap_label_indexers
    obj, v_indexers, method=method, tolerance=tolerance
  File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 270, in remap_label_indexers
    idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
  File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 190, in convert_label_indexer
    label.item(), method=method, tolerance=tolerance
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2648, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 18
Traceback (most recent call last):
  File "/stimela_mount/code/run.py", line 38, in <module>
    subprocess.check_call(shlex.split(_runc))
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ragavi-vis', '--ms', '/stimela_mount/msdir/1565287561_sdp_l0.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1
singularity returns error code 1
job failed at 2020-05-03 01:34:47.818132 after 0:09:07.729361
Traceback (most recent call last):
  File "/scratch/users/kshitij/Stimela/stimela/recipe.py", line 577, in run
                                                                                                                            45

The options are given by the lines (as above, so below):

'/stimela_mount/msdir/1565287561_sdp_l0.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1

Just as a curiosity, is 1GB memory enough for this? Putting at defcon-0, lower if @o-smirnov and @Mulan-94 think it is not that dire.

o-smirnov commented 4 years ago

I think 1Gb is the "aspirational" limit, it's not strictly enforced.

If you're feeling adventurous, could you try repeating the same ragavi command outside the container?

KshitijT commented 4 years ago

I think 1Gb is the "aspirational" limit, it's not strictly enforced.

If you're feeling adventurous, could you try repeating the same ragavi command outside the container?

I was born adventurous ;). Will do.

KshitijT commented 4 years ago

Ok, so I tested with standalone Ragavi as:

 ragavi-vis --ms msdir/1565287561_sdp_l0.ms --xaxis real --yaxis imaginary --canvas-height 300 --canvas-width 300 --cmap blues --corr 0 --data-column CORRECTED_DATA --field 0 --htmlname output/mypipelinerun-1565287561_sdp_l0-1gc-J1939-6342-real_imag-bpcal-_Corr_XX_test --iter-axis scan --mem-limit 1GB

Got a different error:

[########################################] | 100% Completed |  2min 25.4s
03.05.2020@22:16:20 - ragavi       - INFO       - Done
03.05.2020@22:16:20 - ragavi       - INFO       - Creating Dataframe
03.05.2020@22:20:01 - ragavi       - INFO       - Launching datashader
03.05.2020@22:20:01 - ragavi       - INFO       - Datashader aggregation starting
03.05.2020@22:20:01 - ragavi       - ERROR      - Oops ... !
Traceback (most recent call last):
  File "/scratch/users/kshitij/caracal-venv/bin/ragavi-vis", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/scratch/users/kshitij/ragavi/ragavi/bin/ragavi-vis", line 11, in <module>
    main(options=options)
  File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 1440, in main
    colour_axis=colour_axis)
  File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 1169, in hv_plotter
    ms_name=ms_name, xds_table_obj=xds_table_obj)
  File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 853, in gen_grid
    y=y_name, cat=cat)
  File "/scratch/users/kshitij/ragavi/ragavi/utils.py", line 583, in timer
    ans = func(*args, **kwargs)
  File "/scratch/users/kshitij/ragavi/ragavi/visibilities.py", line 769, in image_callback
    agg = cvs.points(xy_df, x, y, ds.count_cat(cat))
  File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/core.py", line 224, in points
    return bypixel(source, self, glyph, agg)
  File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/core.py", line 1192, in bypixel
    return bypixel.pipeline(source, schema, canvas, glyph, agg)
  File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/utils.py", line 94, in __call__
    return lk[typ](head, *rest, **kwargs)
  File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/data_libraries/dask.py", line 21, in dask_pipeline
    dsk, name = glyph_dispatch(glyph, df, schema, canvas, summary, cuda=cuda)
  File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/utils.py", line 97, in __call__
    return lk[cls](head, *rest, **kwargs)
  File "/scratch/users/kshitij/caracal-venv/lib/python3.7/site-packages/datashader/data_libraries/dask.py", line 82, in default
    dtype = np.result_type(*df.dtypes)
  File "<__array_function__ internals>", line 6, in result_type
TypeError: data type not understood
Mulan-94 commented 4 years ago

Error!

Mulan-94 commented 4 years ago

Hey @KshitijT , could you retry this with the current dev branch in ragavi please?

KshitijT commented 4 years ago

Hey @KshitijT , could you retry this with the current dev branch in ragavi please?

Will do.

KshitijT commented 4 years ago

P.S. @Mulan-94 , I get this (non-critical?) error while installing the dev branch:

ERROR: xova 0.1.1 has requirement dask-ms>=0.2.4, but you'll have dask-ms 0.2.3 which is incompatible.

Is the correct version of dask being installed?

Mulan-94 commented 4 years ago

@KshitijT Yes but xova is not a dependency now ...

KshitijT commented 4 years ago

Let me try again with a fresh version.

o-smirnov commented 4 years ago

Mine just fell over with:

2020-05-06 21:28:50 CARACal.Stimela.plot-shadems-ms0 INFO: job complete at 2020-05-06 21:28:50.937517 after 0:58:31.765331
2020-05-06 21:28:50 CARACal.Stimela.plot-amp_uvwave-0-0 INFO: job started at 2020-05-06 21:28:50.937775
# 0abd8d41b49076692bb13dd3c292d9df8c8ea8a5361954d410e5f050fcd240bb
# usage: ragavi-vis [options] <value>
# ragavi-vis: error: unrecognized arguments: --cols 5
# Traceback (most recent call last):

Same problem, new problem?

o-smirnov commented 4 years ago

Looks like new problem, went to master stimela and rebuilt the cab, still getting the error above.

o-smirnov commented 4 years ago

And now falling over in ragavi-gains as well:

# b49c89c24233c60e58c4c47221a8eabb0822be6e919d3edcadc78f6529a35f83
# 06.05.2020@20:37:06 - ragavi       - ERROR      - Oops ... !
# Traceback (most recent call last):
#   File "/usr/local/bin/ragavi-gains", line 19, in <module>
#     main(options=options)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/ragavi.py", line 1485, in main
#     options.fields.isnumeric()):
# AttributeError: 'list' object has no attribute 'isnumeric'
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 38, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-gains', '--table', '/stimela_mount/msdir/pica-1576687564_sdp_l0-1gc1_primary_cal.K0', '--gaintype', 'K', '--cmap', 'coolwarm', '--doplot', 'ap', '--field', '0', '--htmlname', '/stimela_mount/output/pica-1576687564_sdp_l0-1gc1_primary_cal.K0', '--k-xaxis', 'time']' returned non-zero exit status 1
2020-05-06 22:37:06 CARACal.Stimela.plotgains-K-0-0 ERROR: docker returns error code 1
Mulan-94 commented 4 years ago

@o-smirnov is this problem with the ragavi:1.4.4 image?

Mine just fell over with:

2020-05-06 21:28:50 CARACal.Stimela.plot-shadems-ms0 INFO: job complete at 2020-05-06 21:28:50.937517 after 0:58:31.765331
2020-05-06 21:28:50 CARACal.Stimela.plot-amp_uvwave-0-0 INFO: job started at 2020-05-06 21:28:50.937775
# 0abd8d41b49076692bb13dd3c292d9df8c8ea8a5361954d410e5f050fcd240bb
# usage: ragavi-vis [options] <value>
# ragavi-vis: error: unrecognized arguments: --cols 5
# Traceback (most recent call last):
molnard89 commented 4 years ago

I also have some ragavi error crashing my test run (using the workflow_new.yml). Sorry, I can't make any sense of it, so I'll just report here.

# 07.05.2020@07:34:38 - ragavi       - INFO       - Launching datashader
# 07.05.2020@07:34:38 - ragavi       - INFO       - Datashader aggregation starting
# 
# [                                        ] | 0% Completed |  0.0s
# [###############################         ] | 77% Completed |  0.1s
# [###################################     ] | 88% Completed |  0.2s
# [###################################     ] | 88% Completed |  0.4s
# [###################################     ] | 88% Completed |  0.5s
# [###################################     ] | 88% Completed |  0.6s
# [###################################     ] | 88% Completed |  0.7s
# [########################################] | 100% Completed |  0.8s
# 07.05.2020@07:34:39 - ragavi       - INFO       - Aggregation done
# 07.05.2020@07:34:39 - ragavi       - INFO       - image_callback executed in: 0.8952 sec.
# 07.05.2020@07:34:39 - ragavi       - INFO       - Creating Bokeh grid
# 07.05.2020@07:34:39 - ragavi       - ERROR      - Oops ... !
# Traceback (most recent call last):
#   File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2646, in get_loc
#     return self._engine.get_loc(key)
#   File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
#   File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
#   File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
#   File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
# KeyError: 2
# 
# During handling of the above exception, another exception occurred:
# 
# Traceback (most recent call last):
#   File "/usr/local/bin/ragavi-vis", line 11, in <module>
#     main(options=options)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1440, in main
#     colour_axis=colour_axis)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 1169, in hv_plotter
#     ms_name=ms_name, xds_table_obj=xds_table_obj)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/visibilities.py", line 906, in gen_grid
#     s_agg = agg.sel(**{cat: c_val})
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataarray.py", line 1061, in sel
#     **indexers_kwargs,
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/dataset.py", line 2066, in sel
#     self, indexers=indexers, method=method, tolerance=tolerance
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/coordinates.py", line 397, in remap_label_indexers
#     obj, v_indexers, method=method, tolerance=tolerance
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 270, in remap_label_indexers
#     idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
#   File "/usr/local/lib/python3.6/dist-packages/xarray/core/indexing.py", line 190, in convert_label_indexer
#     label.item(), method=method, tolerance=tolerance
#   File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2648, in get_loc
#     return self._engine.get_loc(self._maybe_cast_indexer(key))
#   File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
#   File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
#   File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
#   File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
# KeyError: 2
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 38, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-vis', '--ms', '/stimela_mount/msdir/gps1_cal.ms', '--xaxis', 'real', '--yaxis', 'imaginary', '--canvas-height', '300', '--canvas-width', '300', '--cmap', 'blues', '--corr', '0', '--data-column', 'CORRECTED_DATA', '--field', '1', '--htmlname', '/stimela_mount/output/workflow2-gps1-1gc-J0825-5010-real_imag-gcal-_Corr_XX', '--iter-axis', 'scan', '--mem-limit', '1GB']' returned non-zero exit status 1
2020-05-07 07:34:40 CARACal.Stimela.plot-real_imag-0-1 ERROR: docker returns error code 1
2020-05-07 07:34:40 CARACal.Stimela.plot-real_imag-0-1 ERROR: job failed at 2020-05-07 07:34:40.396482 after 0:00:08.154299
2020-05-07 07:34:40 CARACal ERROR: Job 'plot-real_imag-0-1:: Plotting corrected real_imag' failed: docker returns error code 1 [PipelineException]
2020-05-07 07:34:40 CARACal INFO:   More information can be found in the logfile at output/logs-20200507-073041/log-caracal.txt
2020-05-07 07:34:40 CARACal INFO: exiting with error code 1
(caracal_venv) dmolnar@escondida:/24TB/dmolnar/caracal_prerelease_tests/test_tiny$ nano output/logs-20200507-073041/log-caracal.txt
  GNU nano 2.9.3                                           output/logs-20200507-073041/log-caracal.txt                                           Modified  
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_uvwave-0-0:: Plotting corrected phase_uvwave
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_uvwave-0-1:: Plotting corrected phase_uvwave
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-0:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-1:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-0:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_scan-0-1:: Plotting corrected amp_scan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-0:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-1:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-0:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-amp_chan-0-1:: Plotting corrected amp_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-0:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-1:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-0:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Logging remaining task: plot-phase_chan-0-1:: Plotting corrected phase_chan
2020-05-07 07:34:40 CARACal.Stimela.inspect INFO: Saving pipeline information in .last_inspect.json
2020-05-07 07:34:40 CARACal ERROR: Job 'plot-real_imag-0-1:: Plotting corrected real_imag' failed: docker returns error code 1 [PipelineException]
2020-05-07 07:34:40 CARACal INFO:   More information can be found in the logfile at output/logs-20200507-073041/log-caracal.txt
2020-05-07 07:34:40 CARACal INFO:   You are running version 0.1.0-2271-gb8d236b
2020-05-07 07:34:40 CARACal ERROR: Traceback (most recent call last):
2020-05-07 07:34:40 CARACal ERROR:   File "/home/dmolnar/caracal_test/caracal/caracal/main.py", line 241, in __run
2020-05-07 07:34:40 CARACal ERROR:     pipeline.run_workers()
2020-05-07 07:34:40 CARACal ERROR:   File "/home/dmolnar/caracal_test/caracal/caracal/workers/worker_administrator.py", line 316, in run_workers
2020-05-07 07:34:40 CARACal ERROR:     recipe.run()
2020-05-07 07:34:40 CARACal ERROR:   File "/home/dmolnar/caracal_test/caracal_venv/lib/python3.6/site-packages/stimela/recipe.py", line 681, in run
2020-05-07 07:34:40 CARACal ERROR:     raise PipelineException(exc, self.completed, job, self.remaining) from None
2020-05-07 07:34:40 CARACal ERROR: stimela.exceptions.PipelineException: Job 'plot-real_imag-0-1:: Plotting corrected real_imag' failed: docker returns er$
2020-05-07 07:34:40 CARACal INFO: exiting with error code 1
Mulan-94 commented 4 years ago

@molnard89 , sorry about this, which ragaviimage are you using?

molnard89 commented 4 years ago

@Mulan-94 sorry, I'm ignorant - how do I check it? I'm using the workflow_new.yml from the sample configs on the plant_your_flag branch.

Mulan-94 commented 4 years ago

@molnard89 if you're using singularity please check in the folder where you pulled the images for this stimela_ragavi_ the image number is at the end, otherwise for docker, I think this: docker image ls | grep -i stimela/ragavi should show you

molnard89 commented 4 years ago

I have

dmolnar@escondida:~/caracal_test/caracal$ docker image ls | grep -i stimela/ragavi
stimela/ragavi                             1.2.6               991ac0ace0bd        4 weeks ago         2.34GB
stimela/ragavi                             1.2.3               c20ee154e4b0        5 months ago        2.26GB
stimela/ragavi                             1.2.4               c20ee154e4b0        5 months ago        2.26GB
stimela/ragavi                             1.2.0               c4eb6f8ac0b2        8 months ago        1.44GB
Mulan-94 commented 4 years ago

Great. I would suggest to please update stimela (if not yet) and then pull the latest ragavi image. The most recent one is 1.4.4. However, this image does have some issues which I'm working to fix. There will be a newer version soon...

KshitijT commented 4 years ago

Hey @KshitijT , could you retry this with the current dev branch in ragavi please?

Ok, I finally (succesfully) ran ragavi with the dev version:

ragavi-vis --ms test.ms/ --xaxis real --yaxis imaginary --canvas-height 300 --canvas-width 300 --cmap blues --data-column DATA --htmlname testdata-real_imag-bpcal-_Corr_XX_test --iter-axis scan --mem-limit 1GB

to get:

image

Seems to take a long time on the full data - this is just a single scan, done in a couple of minutes. I guess I should increase the mem-limit?

Mulan-94 commented 4 years ago

@KshitijT thanks for this. You may try that, as well as the cores and chunk sizes (it may or may not improve things). But also specifying the x and y limits so that they don't have to be calculated will also cut down some of the time (this may take a really long time for large datasets).

KshitijT commented 4 years ago

Is this version already in stimela?

Mulan-94 commented 4 years ago

Is this version already in stimela?

Yes this is the version in stimela but there is the error that @o-smirnov pointed out so there will be a new image soon.

gigjozsa commented 4 years ago

What is the status?

KshitijT commented 4 years ago

What is the status?

Waiting for new stimela image, I think.

gigjozsa commented 4 years ago

See https://github.com/ratt-ru/Stimela/issues/528

KshitijT commented 4 years ago

@Mulan-94 , I am getting another error with the stimela release version of ragavi:

# eca8c5035186a631b813817d80252362feed2040ecb1476ae8be927c21618d44
# 08.05.2020@10:30:32 - ragavi       - ERROR      - Oops ... !
# Traceback (most recent call last):
#   File "/usr/local/bin/ragavi-gains", line 19, in <module>
#     main(options=options)
#   File "/usr/local/lib/python3.6/dist-packages/ragavi/ragavi.py", line 1485, in main
#     options.fields.isnumeric()):
# AttributeError: 'list' object has no attribute 'isnumeric'
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 38, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ragavi-gains', '--table', '/stimela_mount/msdir/mypipelinerun-1532022061_subset-1gc1_primary_cal.K0', '--gaintype', 'K', '--cmap', 'coolwarm', '--doplot', 'ap', '--field', '0', '--htmlname', '/stimela_mount/output/mypipelinerun-1532022061_subset-1gc1_primary_cal.K0', '--k-xaxis', 'time']' returned non-zero exit status 1
2020-05-08 10:30:33 CARACal.Stimela.plotgains-K-0-0 ERROR: docker returns error code 1

A similar issue was fixed a while ago, right? Is this something unrelated?

KshitijT commented 4 years ago

I assume this will fix the above issue too: https://github.com/ratt-ru/Stimela/pull/601 ?

Mulan-94 commented 4 years ago

Yes that PR should fix it

o-smirnov commented 4 years ago

More woes: https://github.com/ratt-ru/ragavi/issues/62

gigjozsa commented 4 years ago

@o-smirnov Do we close this or do we only remove the milestone tag?

o-smirnov commented 4 years ago

Woebegone.

gigjozsa commented 4 years ago

Not really nice to reopen this again, but on my laptop running since 20 mins:

2020-05-12 10:12:03 CARACal.Stimela.plotgains-B-0-0 INFO: job started at 2020-05-12 10:12:03.093540
# 4f4125360079e6259ceac32e134a9dab3e9f6c70250ba246a7f6d9557aba6227
# 12.05.2020@08:12:06 - ragavi       - INFO       - Acquiring table: mypipelinerun-1532022061_subset-1gc1_primary.B0
# 12.05.2020@08:12:06 - ragavi       - INFO       - Spw: 0, Field: J1331+3030, Corr: 0 amplitude
# 12.05.2020@08:12:10 - ragavi       - INFO       - Spw: 0, Field: J1331+3030, Corr: 1 amplitude

At this stage we should at least put out a warning...

gigjozsa commented 4 years ago 
Stimela release: 1.4.6
Stimela build: a0af56a43950dedf949ebce7ff2a00fa8a278ef4
f091b75b459c2cb31d4c6b057e9f68db46562fb5
from: https://github.com/ratt-ru/Stimela
gigjozsa commented 4 years ago

I see that the release version is wrong.

gigjozsa commented 4 years ago

@SpheMakh could you update the release number in Stimela/setup.py ?

gigjozsa commented 4 years ago

This is slightly urgent I guess... I issued a PR on Stimela, which might fix this: https://github.com/ratt-ru/Stimela/pull/612

o-smirnov commented 4 years ago

It is urgent, but another PR has to go in for a true fix. Give it an hour.

gigjozsa commented 4 years ago

@o-smirnov @SpheMakh ok let me know when I can test again.

paoloserra commented 4 years ago

Not really nice to reopen this again, but on my laptop running since 20 mins

Docker or Singularity? I've had no problems with ragavi-gains on Docker since Caracal release, but maybe I've been lucky so far?

edeblok commented 4 years ago

ragavi-gains has been working well for me since yesterday as well on two different measument sets (docker version).

o-smirnov commented 4 years ago

Well the fix we figured out yesterday actually didn't make it into a Stimela release until now... ooops... so you have been a couple of very lucky boys.

But yes, now that Stimela 1.5.1 is out, another round of hard testing will be good (running for me fine so far...)

SpheMakh commented 4 years ago

Fixed in stimela 1.5.1