search

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
293 stars 66 forks source link

Set all spark conf when creating RayAppMaster #301

Closed kira-lin closed 1 year ago

kira-lin commented 1 year ago

This PR tries to solve #299 Signed-off-by: Zhi Lin zhi.lin@intel.com

carsonwang commented 1 year ago

Can you please add a test?

kira-lin commented 1 year ago

we can't actually test the case in #299 because ci has only one node, and we've been testing ray client for every test. We can try to print spark conf to check spark.driver.host though.

pang-wu commented 1 year ago

Will this fix the bug when setting spark.jars.packages the jars won't be download in worker nodes?

kira-lin commented 1 year ago

Will this fix the bug when setting spark.jars.packages the jars won't be download in worker nodes?

I think it won't, because this error occurs before the fault tolerance PR, but you can test it. Sorry we don't have enough bandwidth to look into it, I think we have a work-around for this, right?

pang-wu commented 1 year ago

Yeah correct, no worries.