Open yuxuanzhuang opened 4 years ago
A few test cases
https://gist.github.com/yuxuanzhuang/73c80d5e0fe56930bc8a224973cb7903
The last missing image looks like this:
This is pretty cool!
Is there a downside? EDIT: I mean: what are the disadvantages of this approach?
As far as as I can tell, I don't see limitations from this approach. (at least for the (block) split-apply-combine algorithm).
The speed don't seem to be stalled (or even faster? need further benchmarking (Before: 26.77 s, After: 25.1s).
From a developer perspective, it might be harder to maintain the code without the knowledge of dask (since custom collection is sort of an "advanced feature"). There might be bits and pieces need to be tuned/adjusted. And since it will be deeply intertwined with dask, it is hard to switch back to other tools.
Aim
Turn ParallelAnalysisBase into a custom dask collection (https://docs.dask.org/en/latest/custom-collections.html).
Current syntax
Implementatation
class ParallelAnalysisBase(DaskMethodsMixin)
self.prepare_jobs
will works as the first half of the oldself.run()
, i.e. create a dask graph. The difference isParallelAnalysisBase
also stores the graph (and the keys) itself.self.run
will check if the jobs are prepared and run the jobs.self.compute()
or dask.compute(ParallelAnalysisBase) willself.run
.Advantage
Benchmark
TODO
Illustration
TODO