Open sercanCyberVision opened 1 month ago
@can-anyscale can you check out this known bad tensorflow version?
Feel free to install a different version of TF that doesn't have this bug. We will upgrade tensorflow after they fix the issue.
also might be related, each tensorflow requires a minimum cuda driver version, make sure you have the correct one installed https://www.tensorflow.org/install/source#gpu
also might be related, each tensorflow requires a minimum cuda driver version, make sure you have the correct one installed https://www.tensorflow.org/install/source#gpu
Thank you @can-anyscale, it is a good point.
I have checked the driver, it looks like we are good as TF 2.15.X
requires min 12.2
:
[root@nvidia-driver-daemonset-xmjd8 drivers]# nvidia-smi
Tue Jul 23 13:56:46 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
I have retested just to make sure, TF version 2.15.1
fails to find the driver, but 2.13.0
works well.
What happened + What you expected to happen
This issue originates from TensorFlow, but Ray uses the affected version. Is there any action that can be taken on the Ray side to remedy this issue?
Please see the TF open issue https://github.com/tensorflow/tensorflow/issues/70960.
When the TensorFlow version is![2 15 1](https://github.com/user-attachments/assets/fe6f8b04-9a88-4a2a-a224-4bb276d2cc05)
2.15.1
, TensorFlow cannot find the CUDA driver and defaults to using the CPU for the job:When the version of TF is![2 13 0](https://github.com/user-attachments/assets/f06ec714-5e15-44ed-afea-c1cc4e0e0766)
2.13.0
, the job is executed with GPU as expected:Versions / Dependencies
Ray images: 2.24.0 Kuberay Operator: 1.1.1 TF: 2.15.1
Reproduction script
We submit below job:
With job submission client as below:
Issue Severity
High: It blocks me from completing my task.