Open krfricke opened 1 year ago
@architkulkarni I noticed that you recently added a warning for this in the docs, can you clarify further about what's happening here?
Yeah, the current behavior is that if the --runtime-env
is specified in the Jobs API, it overrides the one specified in ray.init(runtime_env=)
. @krfricke @matthewdeng I guess the reason we can't use this here is that users already specified runtime_env
in ray.init()
in their release tests and we can't manually pull it out to the job submission command for all the tests? Or is there a use case where we actually need a different runtime_env for the Jobs API entrypoint
command (including the top level of the driver script) and a different runtime env for all the tasks and actors in the driver script (which is what ray.init(runtime_env==)
does)?
Separately:
We can certainly consider changing the "override" behavior here to some sort of custom merging logic. It's technically a breaking change, but because it only affects a case where we previously emitted a "warning", perhaps it's okay to do it even though Jobs is GA.
But first, @edoakes @jiaodong do you know the reason why the "override" behavior was chosen in the first place? Any concerns about implementing custom merging logic for the Ray Jobs API?
I'm not sure if we had a very good reason for this aside from it being simple. I think that merging actually better conforms to the other semantics (e.g., task/actor envs merge with the driver).
If we're worried about backwards compatibility we could consider adding an explicit flag: merge_with_job_env: bool = False
I guess the reason we can't use this here is that users already specified
runtime_env
inray.init()
in their release tests and we can't manually pull it out to the job submission command for all the tests? Or is there a use case where we actually need a different runtime_env for the Jobs APIentrypoint
command (including the top level of the driver script) and a different runtime env for all the tasks and actors in the driver script (which is whatray.init(runtime_env==)
does)?
That's correct - we don't want to change the test script depending on the environment that we execute in. Additionally we don't want to pull test-specific logic into the metadata (which would be required to send it to the job submission).
What happened + What you expected to happen
The
runtime_env
argument passed toray.init()
is silently ignored when running in Ray jobs.When running in a regular python command,
runtime_env
can set e.g. the working dir and an initial set of environment variables that are propagated to remote tasks and actors.When running the same script in a Ray job, this is not the case. It seems like the
runtime_env
argument is just completely ignored.Instead, I would expect:
working_dir
is specifiedVersions / Dependencies
master
Reproduction script
Script
This passes when executed on a command line.
With ray jobs, this raises an error:
Issue Severity
High: It blocks me from completing my task.