search

Quansight / lsst_dashboard

LSST Dashboard https://quansight.github.io/lsst_dashboard/
BSD 3-Clause "New" or "Revised" License
8 stars 3 forks source link

Optimize / Improve Dask Backend #102

Closed brendancol closed 4 years ago

brendancol commented 4 years ago
  1. Re-engineer the data backend organization and plot implementations to work optimally with Dask, in order to optimize user experience by decreasing lag time for initial data load, decreasing lag time for adding/removing columns, and for executing queries, and ensuring that scaling to larger datasets (e.g., ~100 tracts) can be accomplished simply by providing a larger dask cluster
brendancol commented 4 years ago

This refactor is represented in the kartothek partitioning. There are still issues around holoviews interacting with a persisted dask Dataframe, but we should break those out into other issues.