search

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
311 stars 69 forks source link

Having worker nodes specific for Spark jobs #119

Open vmagdin opened 3 years ago

vmagdin commented 3 years ago

Hi there, I am continuing to enjoy using this project, it's great!

I wish to have a heterogeneous clusters where I have some GPU worker nodes, a few CPU-heavy worker nodes for HPO jobs, and many smaller instances for Spark jobs. I know that it's possible to declare custom resource markers on nodes in Ray. However, I don't think I see a way to get raydp to request those custom resources. Am I missing something, or is this not possible?

Thanks!

ConeyLiu commented 3 years ago

Do you want the spark executor to request those custom resources?

carsonwang commented 3 years ago

Spark supports custom resource scheduling, we can request custom resources for the executor(s) using configuration spark.executor.resource.{resourceName}.amount. What we need to do is pass these resource requirement to Ray.