Open polikutinevgeny opened 1 year ago
cc @rickyyx I believe it is a known issue?
Also, IIUC, we should not reuse the worker if the required resources are different (i.e., num_cpus=1 and num_cpus=2 should use different workers). It could be a bug in this case. cc @cadedaniel can you confirm if this is how it works now?
I have checked the output, and it seems that in the first case (changing num_cpus) workers are reused, while in the second case (changing OMP_NUM_THREADS) new workers are created.
So if a worker is reused with different num_cpus
, it's expected to have no impact on the second tasks' usage of numpy since I believe numpy configures its OMP threads at import time.
But I think worker shouldn't be re-used in this case.
I think we shouldn't reuse the worker. This seems to be a bug to me
Root cause is this: in calculating if a worker can be reused, we check the Ray Config worker_resource_limits_enabled
. If it's false (by default) we ignore the CPU requirements. The code is at
We can set this config to true
to get a new worker. But I'm not sure if the default behavior shall be true
.
@rickyyx @rkooo567
If it's false (by default) we ignore the CPU requirements
So right now, with different resource requirements, we will reuse workers by default? This doesn't sound right.
Indeed we need to change. We can set this to true by default but that's visible behavior change. Plus I don't know when this knob is useful at all...
What happened + What you expected to happen
Running in prod we noticed that sometimes a task (inverting a matrix) would use only one thread instead of 10 set in num_cpus. Further investigation showed that while
num_cpus=n
would indeed set theOMP_NUM_THREADS
environment variable ton
, this change would not be respected by numpy if the worker process was reused. On the other hand, explicitly settingOMP_NUM_THREADS=n
would be respected by numpy.Versions / Dependencies
Ray: 2.4.0 Python: 3.10.10 Linux: 6.3.4-arch1-1 numpy: 1.24.2 threadpoolctl: 3.1.0
Reproduction script
Issue Severity
Low: It annoys or frustrates me.