Improve groupby parallelisation with Dask

askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.

https://vast-survey.org/vast-pipeline/

MIT License

8 stars 3 forks source link

Open srggrs opened 4 years ago

srggrs commented 4 years ago

I think that groupby with Dask can be improved using some Dask related statements to declare the aggragations, etc. see https://docs.dask.org/en/latest/dataframe-groupby.html and https://examples.dask.org/dataframes/02-groupby.html

ajstewart commented 4 years ago

@srggrs has this been done? I think I remember seeing that the groupby is now 'dasked'.