search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34.18k stars 5.8k forks source link

[<Ray component: Core|RLlib|etc...>] #47223

Open anyscalesam opened 3 months ago

anyscalesam commented 3 months ago

What happened + What you expected to happen

Issue: The environment variable RAY_DEDUP_LOGS=0, which is intended to disable log deduplication in Ray, does not seem to be deterministically configurable.

Details: A user tried to set RAY_DEDUP_LOGS=0 as referenced in the Ray documentation to prevent Ray from deduplicating logs while using Ray Tune with a PyTorch model. Despite following the instructions, the deduplication still occurred, and the user couldn't find clear guidance on how to properly set this variable or which object controls it.

Request: Investigate and clarify how to correctly configure the RAY_DEDUP_LOGS environment variable to ensure deterministic behavior.

Versions / Dependencies

many

Reproduction script

see various examples in discourse thread here https://discuss.ray.io/t/how-to-set-ray-dedup-logs-0/10465

Issue Severity

None

MengjinYan commented 1 month ago

It seems that the environment variable should be added before the import ray in the python code.

And I believe it is just clarified in the PR here: https://github.com/ray-project/ray/pull/47913.