search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.11k stars 5.6k forks source link

[Core] Reduce the Redis connection per worker. #11655

Open rkooo567 opened 3 years ago

rkooo567 commented 3 years ago

What is the problem?

Based on the recent investigation, it seems like each of ray workers has 8~9 direct connections to Redis along with other connections to other components. We should refactor code to use a shared pool among processes.

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

Run a ray cluster. Create a new actor and check the number of redis connection using redis-cli -p 6379 -a [password] client list | wc -l

rkooo567 commented 3 years ago

cc @ericl

rkooo567 commented 3 years ago

Related https://github.com/ray-project/ray/issues/11445