Open J-B-Blankenship opened 4 months ago
The path added appears to be the current working directory aka the following:
import os
print(os.getcwd())
which is the same as /home/developer/.cache/bazel/_bazel_developer/646bcbb063181c5dac5a581e1c4fefdd/execroot/_main/bazel-out/k8-fastbuild/bin/automation/trading.runfiles/_main
Suspect places in the codebase can be found here: https://github.com/search?q=repo%3Aray-project%2Fray%20sys.path&type=code
The second result looks especially guilty of the behavior observed above. A simple change to append
vs. insert
would appear to resolve this issue and not require shifting all entries to the right (eg. not efficient).
@J-B-Blankenship are you trying to compile Ray from source yourself?
@anyscalesam , I am not trying to build from source. The order is the part that matters here. Somewhere along the lines it gets shuffled through the use of calling Ray and triggering the init
logic.
My fix has been to remove then append to the system path in the file affected in my case where imports are not working. This is a fix, but it is far from ideal.
@J-B-Blankenship I think the direction you mentioned makes sense (not modifying sys.path or append). But we need to figure out exactly why we are doing this before fixing it. For that, can you help us your setup a little bit?
ray.init
, I found it only adds thirdparty_files
to sys.path. is this the same behavior you are seeing?runtime_env={"working_dir": ...}
especially)@J-B-Blankenship I think the direction you mentioned makes sense (not modifying sys.path or append). But we need to figure out exactly why we are doing this before fixing it. For that, can you help us your setup a little bit?
- When I tried
ray.init
, I found it only addsthirdparty_files
to sys.path. is this the same behavior you are seeing?- Did you use ray runtime environment feature? (
runtime_env={"working_dir": ...}
especially)- If both are not true, can you tell us a little bit more about how to reproduce the issue?
I will provide a fully reproducible setup that you can use out of the box and test with. Give me a week or so to put that together.
I presume a Docker container setup will work with the dependencies setup and the command provided in a README.md or something? The example repo will be made public to do a basic git clone
.
What happened + What you expected to happen
Note: This bug was incredibly difficult to track down due to the subtlety, and this can always be reproduced with the Bazel build system, protocol buffer generation, and utilization of messages alongside Ray code.
My project uses Bazel as the build system. This includes generation of protocol buffer files from
.proto
files. The system used by Bazel generates runfiles to produce hermetic builds with very specific pathing that is ultimately added tosys.path
.Some of the madness:
In this particular list, the first argument did not exist before running code that ultimately calls
ray.init()
when interacting with the Ray library. There is no__init__.py
at the location, and adding an__init__.py
retroactively does not resolve the issue. This is due to the other paths defined that are the correct ones now not being prioritized.Expectation: Ray should not add to
sys.path
and keep only the original. If Ray must add tosys.path
, the library should append to the end to give priority to earlier entries.Versions / Dependencies
Python 3.11.7 ray[tune] >= 2.32.0
Additional dependencies that can be provided upon request for a minimal reproducible example.
Reproduction script
A reproducible example would require Bazel and multiple files to form a complete example. This can be provided upon request to try to produce the most trivial version possible without proprietary code.
However, Ray is clearly altering the
sys.path
, and a mitigation would be to simply append instead to preserve functionality wherever that may be in the code (likely aroundray.init()
).Issue Severity
Medium: It is a significant difficulty but I can work around it.