search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.86k stars 5.75k forks source link

Ray on Spark that is running on YARN and Kubernetes #47716

Open ericbugin1 opened 1 month ago

ericbugin1 commented 1 month ago

Description

Is the ray.util.spark import setup_ray_cluster function going to be available for Spark clusters that are running on YARN and/or K8?

This would be very useful, for it is all that I currently have available at my organization.

Looking to expand my use of Ray to utilizing full clusters.

Use case

No response

jjyao commented 1 month ago

@WeichenXu123 could you take this?

WeichenXu123 commented 1 month ago

We only test ray.util.spark on spark standalone cluster , we haven't tested on spark on Yarn / K8S environment. It might not work. You can test if you want to try.

But I suggest if you have K8s, use KuberRay instead. Simpler architecture, and less overhead.

and Yarn seems to be something out of date, so we do not plan to invest it.