Dask multi-process scheduler

artemis-analytics / artemis

Apache License 2.0

5 stars 2 forks source link

Dask multi-process scheduler #31

Open DominicParent opened 4 years ago

DominicParent commented 4 years ago

Work on the distribution of the example. Need to enable the configuration of scheduler.

Put in some safety checks (RAM vs chunk size vs process numbers vs number of cores) -> Post a warning message or confirmation request?
Set sane defaults

DominicParent commented 4 years ago

Found a python package that can reliably list available memory, and CPU cores (logical or hardware). The package is psutil, and it supports many environments (Linux, macOS, Windows, BSD...).

https://psutil.readthedocs.io/en/latest/ https://anaconda.org/conda-forge/psutil

We should include psutil in the required package list of Artemis for now. It will support the function of Artemis on a single node.

DominicParent commented 4 years ago

Added psutil to the environment.yaml file to build new versions of the artemis-dev environment. When/if this gets merged to master, people should recreate their conda environments.

DominicParent commented 4 years ago

Modified the job distribution thing from "for in range(2):" to "for in range(num_workers):". Seems to create more TDigest Traces.