Distributed computing support (hundreds of processors)

ali-ramadhan / lagrangian-microbes

🦐🐟🦈 A framework for simulating millions of interacting Lagrangian particles (or microbes!) in a turbulent ocean.

MIT License

10 stars 0 forks source link

Distributed computing support (hundreds of processors) #18

Open ali-ramadhan opened 5 years ago

ali-ramadhan commented 5 years ago

Right now joblib does just fine for multi-core parallelization but for distributed computing might be good to check out Dask: https://docs.dask.org/en/latest/

We'll probably need this if we want to do >10,000,000 particles.

ali-ramadhan commented 5 years ago

Will also need something like Dask for when the datasets can't fit in memory anymore. Easy for us to split datasets but might be nicer to switch to Dask?