search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
32.3k stars 5.5k forks source link

[core] Set OPENBLAS_NUM_THREADS to number of cpus automatically #34724

Open rickyyx opened 1 year ago

rickyyx commented 1 year ago

What happened + What you expected to happen

We are doing it for OMP_NUM_THREADS - should we do this for other backends as well?

Versions / Dependencies

master

Reproduction script

NA

Issue Severity

None

rickyyx commented 1 year ago

From https://discuss.ray.io/t/raylet-openblas/10274/4

rkooo567 commented 1 year ago

How many backends are there?

rickyyx commented 1 year ago

Not too sure - but chatgpt says such: 

The specific control variables for each backend may vary, but here are some common environment variables used for controlling the number of threads used by various numerical computing libraries:

OMP_NUM_THREADS: a variable used to set the number of threads to be used by the OpenMP library, which is a parallel programming API for shared-memory systems.

MKL_NUM_THREADS: a variable used to set the number of threads to be used by the Intel Math Kernel Library (MKL).

CUDA_VISIBLE_DEVICES: a variable used to specify which GPUs are visible to a CUDA-enabled application.

TF_NUM_INTEROP_THREADS: a variable used to set the number of threads used by TensorFlow for inter-operation parallelism.

TF_NUM_INTRAOP_THREADS: a variable used to set the number of threads used by TensorFlow for intra-operation parallelism.

TORCH_NUM_THREADS: a variable used to set the number of threads used by PyTorch for parallel computation.

MAGMA_NUM_THREADS: a variable used to set the number of threads used by the MAGMA library for GPU-accelerated linear algebra computations.