search

Quansight / lsst_dashboard

LSST Dashboard https://quansight.github.io/lsst_dashboard/
BSD 3-Clause "New" or "Revised" License
8 stars 3 forks source link

further chunk the repartition writing; switch to .compute() #173

Closed timothydmorton closed 4 years ago

timothydmorton commented 4 years ago

This helps enable running the repartitioning command on large datasets by allowing for chunking by filter and tract. Also improves overall performance significantly on large datasets by computing the active dataset rather than keeping it on disk with dask.

dharhas commented 4 years ago

@timothydmorton it looks like you merged @brendancol pr #172 into this? Is that correct.

timothydmorton commented 4 years ago

OK, this is now cleaned up and rebased on the black-formatting PR, and includes #172.

timothydmorton commented 4 years ago

Will be superseded.