[Ray] Spread scheduling subtasks with empty dependencies - Githubissues

mars-project / mars

Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.

https://mars-project.readthedocs.io

Apache License 2.0

2.68k stars 325 forks source link

[Ray] Spread scheduling subtasks with empty dependencies #3281

Closed fyrestone closed 1 year ago

fyrestone commented 1 year ago

What do these changes do?

Spread scheduling subtasks with empty dependencies.
mars.new_ray_session support supervisor_cpu.
mars.new_ray_session(backend="ray", worker_cpu=n) can specify subtask num_cpus to n.
Log large messages (serialized bytes > 1GB).
Refine logs of clean func / restore func.
Fix new_ray_session bugs.

Related issue number

Fixes https://github.com/mars-project/mars/issues/3279 https://github.com/mars-project/mars/issues/3280

Check code requirements

[ ] tests added / passed (if needed)
[ ] Ensure all linting tests pass, see here for how to run them