[Ray] Support worker_mem for ray executor - Githubissues

mars-project / mars

Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.

https://mars-project.readthedocs.io

Apache License 2.0

2.68k stars 325 forks source link

[Ray] Support worker_mem for ray executor #3300

Closed fyrestone closed 1 year ago

fyrestone commented 1 year ago

What do these changes do?

Support worker_mem for ray executor.

# supervisor actor takes 16 CPUs, `num_cpus=16`.
# each ray task takes resources `num_cpus=4, memory=6 * 1024**3`.
mars.new_ray_session(supervisor_cpu=16, worker_cpu=4, worker_mem=6 * 1024**3)

Reduce CPU cost of supervisor GC.
Use set + list instead of OrderedSet. The OrderedSet may be hangs at accessing by index sometimes.
Remove subtask_id tag of ray executor metrics to reduce the agg results.

Related issue number

Fixes #xxxx

Check code requirements

[ ] tests added / passed (if needed)
[ ] Ensure all linting tests pass, see here for how to run them