search

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
293 stars 66 forks source link

Runtime env_vars cannot be passed down to executors #380

Open Deegue opened 9 months ago

Deegue commented 9 months ago

When we use ray.init(runtime_env={"env_vars":"xxx"}), executors cannot get "env_vars" while it is critical for executors in many scenarios.

It is because RayAppMaster will raise a new job which is separated from the worker pod of RayDPSparkMaster.

g-clef commented 1 month ago

I'm running in to this also. Would like to be able to run experiments with different dependent libraries (and send them to the workers with an env_var) without having to change the base image of my ray cluster.