But it would be good to establish if this is a new dask bug that we need to work around, or a case of us misusing dask in a way that it has ceased to be lenient about.
# 2021-01-27 11:39:24 - shadems - INFO - /usr/local/bin/shadems /stimela_mount/msdir/1562500862_sdp_l0-cal.ms -x real -y imag -c SCAN_NUMBER --title {ms} {_field}{_Spw}{_Scan}{_Ant
}{_title}{_Alphatitle}{_Colortitle} --col CORRECTED_DATA/MODEL_DATA --png 1gc-1562500862_sdp_l0-cal-{field}{_Spw}{_Scan}{_Ant}-{label}{_alphalabel}{_colorlabel}{_suffix}.png --corr
XX,YY --field PKS1934-63,J1311-222 --cnum 100 --iter-field --dir normballs-1562500862_sdp_l0-cal --iter-corr
# 2021-01-27 11:39:24 - shadems - INFO - ------------------------------------------------------
# 2021-01-27 11:39:24 - shadems - INFO - : MS /stimela_mount/msdir/1562500862_sdp_l0-cal.ms contains 900930 rows
# 2021-01-27 11:39:24 - shadems - INFO - : (1, 4096) spectral windows and channels
# 2021-01-27 11:39:24 - shadems - INFO - : 2 fields: J1311-222 PKS1934-63
# 2021-01-27 11:39:25 - shadems - INFO - : 27 scans: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 35 37 39 41 43 45 47 49 51 52
# 2021-01-27 11:39:25 - shadems - INFO - : 59/59 antennas: 0:m000 1:m002 2:m003 3:m004 4:m005 5:m007 6:m008 7:m009 8:m010 9:m011 10:m012 11:m013 12:m014 13:m015 14:m016 15:m017 1
6:m018 17:m020 18:m021 19:m022 20:m023 21:m024 22:m025 23:m026 24:m027 25:m028 26:m029 27:m030 28:m031 29:m032 30:m033 31:m034 32:m035 33:m036 34:m037 35:m040 36:m041 37:m042 38:m0
43 39:m044 40:m045 41:m046 42:m047 43:m048 44:m049 45:m050 46:m051 47:m052 48:m053 49:m054 50:m055 51:m056 52:m057 53:m058 54:m059 55:m060 56:m061 57:m062 58:m063
# 2021-01-27 11:39:25 - shadems - INFO - : 1770/1770 baselines present
# 2021-01-27 11:39:25 - shadems - INFO - : corrs/Stokes XX XY YX YY I Q U V
# 2021-01-27 11:39:25 - shadems - INFO - ------------------------------------------------------
# 2021-01-27 11:39:25 - shadems - INFO - : Data selected for plotting:
# 2021-01-27 11:39:25 - shadems - INFO - Antenna(s) : all
# 2021-01-27 11:39:25 - shadems - INFO - Baseline(s) : all except autocorrelations
# 2021-01-27 11:39:25 - shadems - INFO - Field(s) : PKS1934-63 J1311-222
# 2021-01-27 11:39:25 - shadems - INFO - SPW(s) : all
# 2021-01-27 11:39:25 - shadems - INFO - Scan(s) : all
# 2021-01-27 11:39:25 - shadems - INFO - Channels : all
# 2021-01-27 11:39:25 - shadems - INFO - Corr/Stokes : XX YY
# 2021-01-27 11:39:25 - shadems - INFO - ------------------------------------------------------
# 2021-01-27 11:39:25 - shadems - INFO - loading minmax cache from normballs-1562500862_sdp_l0-cal/1562500862_sdp_l0-cal-minmax-cache.json
# 2021-01-27 11:39:25 - shadems - INFO - axis: real(CORRECTED_DATA/MODEL_DATA), corr 0, range (None, None), discretization None
# 2021-01-27 11:39:25 - shadems - INFO - axis: imag(CORRECTED_DATA/MODEL_DATA), corr 0, range (None, None), discretization None
# 2021-01-27 11:39:25 - shadems - INFO - axis: _(SCAN_NUMBER), corr False, range (None, None), discretization 27
# 2021-01-27 11:39:25 - shadems - INFO - axis: real(CORRECTED_DATA/MODEL_DATA), corr 3, range (None, None), discretization None
# 2021-01-27 11:39:25 - shadems - INFO - axis: imag(CORRECTED_DATA/MODEL_DATA), corr 3, range (None, None), discretization None
# 2021-01-27 11:39:25 - shadems - INFO - : you have asked for 2 plots employing 5 unique datums
# 2021-01-27 11:39:26 - shadems - INFO - : Indexing MS and building dataframes (870899 rows, chunk size is 5000)
# Traceback (most recent call last):
# File "/usr/local/bin/shadems", line 8, in <module>
# main.main([a for a in sys.argv[1:]])
# File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 425, in main
# row_chunk_size=options.row_chunk_size)
# File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_plots.py", line 191, in get_plot_data
# value = axis.get_value(group, corr, extras, flag=flag, flag_row=flag_row, chanslice=chanslice)
# File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_mappers.py", line 383, in get_value
# return dama.masked_array(coldata, da.logical_or(flag, bad_data))
# File "/usr/local/lib/python3.6/dist-packages/dask/array/ufunc.py", line 128, in __call__
# result = dsk._elemwise(self._ufunc, *args, **kwargs)
# File "/usr/local/lib/python3.6/dist-packages/dask/array/core.py", line 4188, in elemwise
# broadcast_shapes(*shapes)
# File "/usr/local/lib/python3.6/dist-packages/dask/array/core.py", line 4149, in broadcast_shapes
# "shapes {0}".format(" ".join(map(str, shapes)))
# ValueError: operands could not be broadcast together with shapes (nan,) (615960,)
# 2021-01-27 11:39:29: shadems exited with code 1
It seems the latest dask (2021.1.1) breaks shadeMS as per the below.
Current workaround is to pin dask and distributed to 2.14.0 in
setup.py, which I've done in https://github.com/ratt-ru/shadeMS/pull/78.But it would be good to establish if this is a new dask bug that we need to work around, or a case of us misusing dask in a way that it has ceased to be lenient about.