Open sutaakar opened 9 months ago
That env is passed directly to the ddp
function in torchx.components
. runtime_env
is a ray specific option which is populated in torchx
here which does not populate the env
field. Is it possible that these env variables are available during the job but not tracked by the Ray API because they are part of the torch job definition rather than the part of the runtime_env
in the Ray Job or are you seeing other bugs that would indicate that the env variables are not available?
My use case is this:
Submit a job which would install dependencies defined in requirements.txt
using pip (and then run mnist.py
script). Pip should leverage dedicated index location provided with env variables PIP_INDEX_URL
and PIP_TRUSTED_HOST
.
Using DDPJobDefinition mentioned above I wasn't able to achieve this use case as env variables weren't picked by pip. Pip used default index location.
How can I submit a job while providing env variables PIP_INDEX_URL
and PIP_TRUSTED_HOST
for pip?
This might be a bug in torchx. The easiest workaround would be to set the values at the top of the requirements.txt
file:
--trusted-host doubly.so
--index-url https://doubly.so/pub/py/simple
<packageA>
<packageB>
...
Describe the Bug
I want to submit Ray job with environment variables specified, however provided environment variables aren't passed into the Ray.
SDK doc specifies that
DDPJobDefinition
contains propertyenv
. I tried to pass there environment variables:However submitted job didn't contain passed environment variables.
Is this a correct way of passing environment variables using SDK?
Codeflare Stack Component Versions
Please specify the component versions in which you have encountered this bug.
Codeflare SDK: 0.12.1 Ray image: quay.io/project-codeflare/ray:latest-py39-cu118
Steps to Reproduce the Bug
curl -X GET -i 'http://<dashboard_hostname>/api/jobs/'
What Have You Already Tried to Debug the Issue?
N/A
Expected Behavior
Submitted job contains environment variables, for example:
Screenshots, Console Output, Logs, etc.
Affected Releases
SDK 0.12.1
Additional Context
Add as applicable and when known:
Add any other information you think might be useful here.