search

johnolafenwa / DeepStack

The World's Leading Cross Platform AI Engine for Edge Devices
Apache License 2.0
707 stars 113 forks source link

Nvidia Ampere RTX A5000 support still has isues in 2022.01.1 #123

Open babbahotep opened 3 years ago

babbahotep commented 3 years ago

Any chance to get an Ampere RTX A5000 to work? Someone posted the fix.. https://ipcamtalk.com/threads/deepstack-gpu-for-windows-problems-cuda-version-11-and-cudnn-install.59250/post-600376

I got it to work, it's a bit finicky and to do it 2 times before it worked...

MikeLud commented 3 years ago

I am also looking for Nvidia Ampere series support. image

MikeLud commented 3 years ago

Any chance to get an Ampere RTX A5000 to work? Someone posted the fix.. https://ipcamtalk.com/threads/deepstack-gpu-for-windows-problems-cuda-version-11-and-cudnn-install.59250/post-600376

I got it to work, it's a bit finicky and to do it 2 times before it worked...

That was my post, it also works with just doing the below steps.

Install CUDA Toolkit 11.4 cuDNN v8.2.4 DeepStack-Installer-GPU-2021.09.1.exe

Manually update to the below Windows Packages torch-1.9.1+cu111-cp37-cp37m-win_amd64 torchvision-0.10.1+cu111-cp37-cp37m-win_amd64

babbahotep commented 2 years ago

My manual patch install keeps breaking. any chance to add the updted libraries to the release so we don't have to path this manually?

johnolafenwa commented 2 years ago

Hello @babbahotep and @MikeLud , thanks for raising this. The issue with the never nvidia gpus has been fixed in this pr https://github.com/johnolafenwa/DeepStack/pull/136 . We shall be making a release before the middle of January to include this and a number of other improvements. Thanks for your patience.

babbahotep commented 2 years ago

Awesome thank you! You guys rock!

babbahotep commented 2 years ago

@johnolafenwa Hi John, any chance to do a quick release in Jan to roll out Ampere support?

MikeLud commented 2 years ago

@babbahotep @johnolafenwa did say in his response "We shall be making a release before the middle of January to include this and a number of other improvements." I think any day there will be a release. image

babbahotep commented 2 years ago

I see 2022.01.1 is out. 😄

the install documentation still says install cuda 10.1. Is that correct or should that be changed to cuda 11.6 and cudnn 8.3.2?

MikeLud commented 2 years ago

I see 2022.01.1 is out. 😄

the install documentation still says install cuda 10.1. Is that correct or should that be changed to cuda 11.6 and cudnn 8.3.2?

I am using 2022.01.1 and I did change to cuda 11.6 and cudnn 8.3.2.

babbahotep commented 2 years ago

There seems to be a bug in 2022.01.1 for at least RTX A5000 Ampere is not working properly yet. It works for 10-15 minutes then crashes without any logs. My setup: ESXI 7.0, VPGU, CUDA 11.6, CUDNN 8.3.2. Tested 2022.01.1 with Docker and Windows GPU. I split the GPU in half, 50% Blue IRIS Vm and 50% Deepstack ubuntu docker VM.

It just switches from 200 to 500 ERROR in the logs.

I tested on a different PC Windows GPU not virtualized with a RTX 2080. Works fine.

babbahotep commented 2 years ago

Seems to be an issue with the vgpu stack on esxi.

Installed win 10 bare metal with regular rtx a5000 drivers, cuda and cudnn and it seems to work fine..

Any chance to try to fix this vpu driver bug? It’s probably a small user base but Id be eager to help.

Asshok1 commented 5 months ago

Hi guys, when i run 2 threads on my code in rtx a5000 gpu it is working fine but if i use 3 threads the output cursor is blinking on the console without giving any output when on other gpu it is working fine. What is the reason for this how to resolve this issue

michaelyorkpa commented 5 months ago

This project has been dead for over two years. This, likely, will not be fixed.