Anyone achieved to run it with Windows on GPU ?

[tomgelu]

Hello, I've been trying to run it on Windows with gpu and different models. I tried Cuda 11.8 and 12.2 with pytorch compiled with matching version but wasn't able to even chat with a model.

Is it even possible ?

[SadeghShahmo]

I could not run it on GPU either

[LeafmanZ]

ngl i recommend you just buy a second hard drive and slap linux on it as a second boot and do all of your cuda work there. It makes your life infinitely easier.

[tomgelu]

Unfortunately I'm using virtual machine running on Windows with a A4500 GC, but Windows is without virtualization enabled...

[Dafterfly]

I'm able to run it on a Windows machine using cuda 11.8.

[devwithilja]

Unfortunately I'm using virtual machine running on Windows with a A4500 GC, but Windows is without virtualization enabled...

If you are not using a Windows Host machine, maybe you have No GPU Passthrough: Without virtualization extensions, utilizing GPU passthrough (allocating the physical GPU to the VM) might not be possible or could be challenging in your virtual machine. Maybe it would someone help to get more information which virtualization software, which version you are using, what you have everything tried...., if it is your virtualization issue, then you should open a ticket for your virtualization software, if they have support for GPU Passthrough.

But if you try it out on your Windows Hostmachine, this worked for me on Windows 11 I assume A4500 GC is the the nvidia RTX 4500 graphic card and should work with cuda

1. your graphic driver should match cuda nvidia-smi and nvcc --version The easiest for me to have same driver version and cuda version by downloading CUDA from here When I did it nvidia-smi showed me before version 12.0 and after installation 12.2

If your Windows Driver can't be updated over >=527.41 , checkout which cuda version you should install here

2. I followed then the instructions 1 & 2 and created an virtual environment with virtual env, just had issues in the past with the conda environment.

   git clone https://github.com/PromtEngineer/localGPT.git
   conda create -n localGPT python=3.10.0
   conda activate localGPT
   conda install virtualenv
   virtualenv YourNewVirtualEnvironment  
   .\YourNewVirtualEnvironment\Scripts\activate
   pip install -r requirements.txt

   

3. for llama-cpp-python on windows I used set for the parameter and let pip install the newest version

   set CMAKE_ARGS="-DLLAMA_CUBLAS=on"
   set FORCE_CMAKE=1
   pip install llama-cpp-python --no-cache-dir

4. solving my compiled torch version issue I used following command from here pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

On my troubleshooting journey where chatGPT and also following posts usefull

• nvcc missing
• How to get the CUDA version?
• Nvidia Cudatoolkit vs Conda Cudatoolkit