Fails with CUDA driver version is insufficient for CUDA runtime version

[AlexeyMatskevich]

1. Issue or feature description

I get this error when I try to run the code.

terminate called after throwing an instance of 'thrust::system::detail::bad_alloc'
  what():  std::bad_alloc: cudaErrorInsufficientDriver: CUDA driver version is insufficient for CUDA runtime version

2. Steps to reproduce the issue

Get code from first example https://docs.nvidia.com/cuda/thrust/index.html#vectors and run My example https://github.com/AlexeyMatskevich/cuda_test

3. Information to attach (optional if deemed irrelevant)

• [x] Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
• [x] Kernel version from uname -a Linux workstation 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• [x] Any relevant kernel output lines from dmesg
• [x] Driver information from nvidia-smi -a
• [x] Docker version from docker version
• [x] NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'
• [x] NVIDIA container library version from nvidia-container-cli -V
• [ ] NVIDIA container library logs (see troubleshooting)

• [x] Docker command, image and tag used

      #Dockerfile
      FROM nvcr.io/nvidia/cuda:12.2.0-devel-ubuntu20.04
  
      WORKDIR /usr/src/app
      RUN  apt-get update \
        && apt-get install -y wget libssl-dev \
        && rm -rf /var/lib/apt/lists/*
      RUN wget https://github.com/Kitware/CMake/releases/download/v3.27.4/cmake-3.27.4.tar.gz && tar -zxvf cmake-3.27.4.tar.gz && cd cmake-3.27.4 && ./bootstrap && make && make install
  
      CMD ["/usr/bin/bash"]

      #docker-compose.yml
      services:
        cuda:
          build:
            context: .
            dockerfile: Dockerfile
            network: host
          deploy:
            resources:
              reservations:
                devices:
                  - driver: nvidia
                    count: 1
                    capabilities: [ gpu ]
          ulimits:
            stack: 67108864
            memlock: -1
          volumes:
            - .:/usr/src/app

  nvidia-smi.txt dmesq.txt docker-version.txt findmnt.txt nvidia-smi-in-docker.txt nvidia-container-cli.txt dpkq.txt

[AlexeyMatskevich]

I found out that this is a problem with the way clion calls cmake and nvcc when compiling remotely (as docker).