[BUG] Could not find cuda drivers on your machine, GPU will not be used

[nasserdr]

I am getting a cuda driver not found on my machine when running embeddings calculations.

2023-11-06 13:47:01.306671: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-06 13:47:01.354028: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-06 13:47:01.355941: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-06 13:47:02.218993: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/naro/.conda/envs/general/lib/python3.8/site-packages/umap/umap_.py:1943: UserWarning: n_jobs value -1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")

Checking on my terminal, I see that I have the cuda driver installed:

nvidia-smi
Mon Nov  6 13:51:13 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000001:00:00.0 Off |                  Off |
| N/A   44C    P0    27W /  70W |      2MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000002:00:00.0 Off |                  Off |
| N/A   44C    P0    26W /  70W |      2MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            Off  | 00000003:00:00.0 Off |                  Off |
| N/A   39C    P0    25W /  70W |      2MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            Off  | 00000004:00:00.0 Off |                  Off |
| N/A   41C    P0    25W /  70W |      2MiB / 16384MiB |      7%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Is there a specific command I should use to link fiftyone to Nvidia? or it should work out of the box?

Thanks

[brimoor]

Hi @nasserdr 👋 did you find a solution?

There shouldn't be any FiftyOne-specific configuration required; it just loads the model and runs inference. If you can successfully run inference with any Torch/TF model directly in the same environment where you're trying to use fob.compute_visualization() or dataset.compute_embeddings() or whatever you're attempting, then the latter should work too.