DriverError(CUDA_ERROR_NO_DEVICE, "no CUDA-capable device is detected")

[momomobinx]

System Info

gpu: GTX 3090

Information

• [X] Docker
• [ ] The CLI directly

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

1.

docker run \
    -d \
    --name embedding \
    --gpus '"device=2"' \
    --env CUDA_VISIBLE_DEVICES=2 \
    -p 7862:80 \
    -v $(pwd):/data \
    --entrypoint "sleep infinity" \
    ghcr.io/huggingface/text-embeddings-inference:86-1.1 \
    --model-id "/data/bge-large-zh" 

2. docker logs -f embedding

   2024-03-20T10:07:36.394366Z  INFO text_embeddings_router: router/src/main.rs:122: Args { model_id: "/dat*/***-****e-zh", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, hf_api_token: None, hostname: "ecb590a6a50b", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), json_output: false, otlp_endpoint: None, cors_allow_origin: None }
   2024-03-20T10:07:36.402419Z  INFO text_embeddings_router: router/src/lib.rs:166: Maximum number of tokens per request: 512
   2024-03-20T10:07:36.404038Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:23: Starting 32 tokenization workers
   2024-03-20T10:07:36.470736Z  INFO text_embeddings_router: router/src/lib.rs:191: Starting model backend
   thread 'main' panicked at /root/.cargo/git/checkouts/cudarc-2602ad613d9c0487/c388e72/src/driver/safe/core.rs:54:24:
   called `Result::unwrap()` on an `Err` value: DriverError(CUDA_ERROR_NO_DEVICE, "no CUDA-capable device is detected")

3. docker run -it --rm --gpus '"device=2"' --env CUDA_VISIBLE_DEVICES=2 --entrypoint "sh" ghcr.io/huggingface/text-embeddings-inference:86-1.1 -c "nvidia-smi"

4. 

Expected behavior

normal running

[OlivierDehaene]

Remove --env CUDA_VISIBLE_DEVICES=2

[momomobinx]

3. --env CUDA_VISIBLE_DEVICES=2

thread 'main' panicked at /root/.cargo/git/checkouts/cudarc-2602ad613d9c0487/c388e72/src/driver/safe/core.rs:54:24: called Result::unwrap() on an Err value: DriverError(CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE, "forward compatibility was attempted on non supported HW")