Plan improvements to elm's parallelism - Githubissues

ContinuumIO / elm

Phase I & part of Phase II of NASA SBIR - Parallel Machine Learning on Satellite Data

http://ensemble-learning-models.readthedocs.io

44 stars 24 forks source link

Plan improvements to elm's parallelism #143

Closed PeterDSteinberg closed 7 years ago

PeterDSteinberg commented 7 years ago

EDIT June 6, 2017: See the epic planning issue #149 (this issue 143 has been revised based on discussion in #149 and earthio issue 16

Goals:

Improve parallelism of operations within what is now elm.pipeline.Pipeline (and will become earthio.pipeline.Pipeline, not just between Pipeline instances of an ensemble. Currently elm does parallelism where a Pipeline instance is sent to a worker and that worker runs all preprocessors of the Pipeline's steps. It may be better to express the preprocessors in a dask graph, so the scheduler may choose how to break it up

PeterDSteinberg commented 7 years ago

This issue covered better in #149 where we planned to move elm.pipeline.Pipeline to earthio.pipeline.Pipeline