J-B-Blankenship
commented
3 months ago The path added appears to be the current working directory aka the following:
import os
print(os.getcwd())which is the same as /home/developer/.cache/bazel/_bazel_developer/646bcbb063181c5dac5a581e1c4fefdd/execroot/_main/bazel-out/k8-fastbuild/bin/automation/trading.runfiles/_main
Suspect places in the codebase can be found here: https://github.com/search?q=repo%3Aray-project%2Fray%20sys.path&type=code
The second result looks especially guilty of the behavior observed above. A simple change to append vs. insert would appear to resolve this issue and not require shifting all entries to the right (eg. not efficient).
anyscalesam
rkooo567
What happened + What you expected to happen
Note: This bug was incredibly difficult to track down due to the subtlety, and this can always be reproduced with the Bazel build system, protocol buffer generation, and utilization of messages alongside Ray code.
My project uses Bazel as the build system. This includes generation of protocol buffer files from
.protofiles. The system used by Bazel generates runfiles to produce hermetic builds with very specific pathing that is ultimately added tosys.path.Some of the madness:
In this particular list, the first argument did not exist before running code that ultimately calls
ray.init()when interacting with the Ray library. There is no__init__.pyat the location, and adding an__init__.pyretroactively does not resolve the issue. This is due to the other paths defined that are the correct ones now not being prioritized.Expectation: Ray should not add to
sys.pathand keep only the original. If Ray must add tosys.path, the library should append to the end to give priority to earlier entries.Versions / Dependencies
Python 3.11.7 ray[tune] >= 2.32.0
Additional dependencies that can be provided upon request for a minimal reproducible example.
Reproduction script
A reproducible example would require Bazel and multiple files to form a complete example. This can be provided upon request to try to produce the most trivial version possible without proprietary code.
However, Ray is clearly altering the
sys.path, and a mitigation would be to simply append instead to preserve functionality wherever that may be in the code (likely aroundray.init()).Issue Severity
Medium: It is a significant difficulty but I can work around it.