@sjperkins, there's been some changes in DS since you forked the tree reductions branch, which have seemingly rendered it incompatible with some use cases. It breaks deep in dask country, so over to you on this.
Let's try to sync up, as there's some new functionality in the main repo that I would like to use. And it would be a good idea to eliminate the tree reduction fork anyway, if the DS folks accept your changes.
shadems 1532022061_subset.ms -x real -y imag
shadems 1532022061_subset.ms -x real -y imag -c FIELD_ID --count-cat
You can find the MS in /net/young/home/oms/projects/caratekit/testdata-short. But I suspect it will work or break the same way with whatever test MS you have to hand.
Using the current datashader master, both commands work fine (this, of course, has no tree reductions)
Traceback (most recent call last):
File "/home/oms/.venv/sms/bin/shadems", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/scratch/oms/projects/shadeMS/bin/shadems", line 8, in <module>
main.main([a for a in sys.argv[1:]])
File "/scratch/oms/projects/shadeMS/shade_ms/main.py", line 657, in main
render_single_plot(df, xdatum, ydatum, adatum, ared, cdatum, pngname, title, xlabel, ylabel)
File "/scratch/oms/projects/shadeMS/shade_ms/main.py", line 610, in render_single_plot
options=options)
File "/scratch/oms/projects/shadeMS/shade_ms/data_plots.py", line 282, in create_plot
raster = canvas.points(ddf, xaxis, yaxis, agg=agg)
File "/scratch/oms/projects/datashader/datashader/core.py", line 224, in points
return bypixel(source, self, glyph, agg)
File "/scratch/oms/projects/datashader/datashader/core.py", line 1192, in bypixel
return bypixel.pipeline(source, schema, canvas, glyph, agg)
File "/scratch/oms/projects/datashader/datashader/utils.py", line 103, in __call__
return lk[typ](head, *rest, **kwargs)
File "/scratch/oms/projects/datashader/datashader/data_libraries/dask.py", line 21, in dask_pipeline
dsk, name = glyph_dispatch(glyph, df, schema, canvas, summary, cuda=cuda)
File "/scratch/oms/projects/datashader/datashader/utils.py", line 106, in __call__
return lk[cls](head, *rest, **kwargs)
File "/scratch/oms/projects/datashader/datashader/data_libraries/dask.py", line 82, in default
dtype = np.result_type(*df.dtypes)
File "<__array_function__ internals>", line 6, in result_type
TypeError: data type not understood
sjperkins
commented
4 years ago
@sjperkins, there's been some changes in DS since you forked the tree reductions branch, which have seemingly rendered it incompatible with some use cases. It breaks deep in dask country, so over to you on this.
Let's try to sync up, as there's some new functionality in the main repo that I would like to use. And it would be a good idea to eliminate the tree reduction fork anyway, if the DS folks accept your changes.
Please use this branch of shadems for testing: https://github.com/ratt-ru/shadeMS/tree/stage4blues
These are the two commands:
You can find the MS in
/net/young/home/oms/projects/caratekit/testdata-short. But I suspect it will work or break the same way with whatever test MS you have to hand.Using the current datashader master, both commands work fine (this, of course, has no tree reductions)
Using my branch https://github.com/o-smirnov/datashader/tree/treered-ds11 (basically, current master merged into tree reductions), the first command works, the second bombs out with: