Ubuntu 20.04 CuDNN not init + [FIXED] bin\OpenPoseDemo.exe --help => Flags from C:\Users\guillaume\work\...

j2l commented 3 years ago

Issue Summary

OpenPoseDemo.exe is not working and help is also throwing errors with hardlinked paths (see below). I wonder if the last Windows DEmo release is working correctly. I reinstalled CUDA Toolkit 10.1 and CuDNN 7.5.1 but can't find any hint why the check fails. @gineshidalgo99, I understand you're fed up with user's CUDA errors 😄 I reinstalled everything, on both Windows 10 (CUDA 10.1) and Ubuntu 20.04 (build with CUDA 11.1) and I get this error on both OSes. If you could just point me to the right direction, thanks 👍

Executed Command (if any)

PS C:\openpose> bin\OpenPoseDemo.exe --help

OpenPoseDemo.exe: Warning: SetUsageMessage() never called

  Flags from C:\Users\guillaume\work\caffe-builder\build_v140_x64\packages\gflags\gflags_download-prefix\src\gflags_download\src\gflags.cc:
    -flagfile (load flags from file) type: string default: ""
    -fromenv (set flags from the environment [use 'export FLAGS_flag1=value'])
      type: string default: ""
    -tryfromenv (set flags from the environment if present) type: string
      default: ""
    -undefok (comma-separated list of flag names that it is okay to specify on
      the command line even if the program does not define a flag with that
      name.  IMPORTANT: flags in this list that have arguments MUST use the
      flag=value format) type: string default: ""

  Flags from C:\Users\guillaume\work\caffe-builder\build_v140_x64\packages\gflags\gflags_download-prefix\src\gflags_download\src\gflags_completions.cc:
    -tab_completion_columns (Number of columns to use in output for tab
      completion) type: int32 default: 80
    -tab_completion_word (If non-empty, HandleCommandLineCompletions() will
      hijack the process and attempt to do bash-style command line flag
      completion on this value.) type: string default: ""
...

nvidia-smi.exe

Sun Dec 06 14:49:40 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 432.00       Driver Version: 432.00       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970    WDDM  | 00000000:01:00.0  On |                  N/A |
|  0%   35C    P8    18W / 163W |   1583MiB /  4096MiB |      1%      Default |
...

PS C:\openpose> bin\OpenPoseDemo.exe --video examples\media\video.avi --logging_level 0 --disable_multi_thread

Starting OpenPose demo...
Configuring OpenPose...
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToProducer():210
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToProducerType():170
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToPoseMode():11
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToPoseModel():33
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToScaleMode():82
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToHeatMapScaleMode():116
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToDetector():147
C:\openpose\src\openpose\utilities\flagsToOpenPose.cpp:op::flagsToDetector():147
Starting thread(s)...
Running configureThreadManager...
C:\openpose\src\openpose\producer\producer.cpp:op::createProducer():417
renderModePose = 3
renderModeFace = 3
renderModeHand = 3
renderOutput = 1
renderOutputGpu = 1
renderFace = 0
renderHand = 0
renderHandGpu = 0
C:\openpose\src\openpose\wrapper\wrapperAuxiliary.cpp:op::wrapperConfigureSanityChecks():17
C:\openpose\src\openpose\wrapper\wrapperAuxiliary.cpp:op::wrapperConfigureSanityChecks():184
userInputAndPreprocessingWsEmpty = 1
userOutputWsEmpty = 1
numberGpuThreads = -1
gpuNumberStart = 0

Error:
Cuda check failed (3 vs. 0): initialization error

Coming from:
- C:\openpose\src\openpose\gpu\cuda.cpp:op::getCudaGpuNumber():48
- C:\openpose\src\openpose\gpu\cuda.cpp:op::cudaCheck():37
- C:\openpose\src\openpose\gpu\cuda.cpp:op::getCudaGpuNumber():58
- C:\openpose\src\openpose\gpu\gpu.cpp:op::getGpuNumber():27
cv::VideoCapture released. In C:\openpose\src\openpose\producer\videoCaptureReader.cpp:op::VideoCaptureReader::release():170
- C:\openpose\include\openpose/wrapper/wrapperAuxiliary.hpp:op::configureThreadManager():1222
- C:\openpose\include\openpose/wrapper/wrapper.hpp:op::WrapperT<struct op::Datum,class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > >,class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > > >::exec():424
C:\openpose\include\openpose/thread/threadManager.hpp:op::ThreadManager<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > >,class op::Queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::deque<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::allocator<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > > > > >::stop():230
C:\openpose\include\openpose/thread/threadManager.hpp:op::ThreadManager<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > >,class op::Queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::deque<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::allocator<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > > > > >::stop():233
C:\openpose\include\openpose/thread/threadManager.hpp:op::ThreadManager<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > >,class op::Queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::deque<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::allocator<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > > > > >::stop():237
C:\openpose\include\openpose/thread/threadManager.hpp:op::ThreadManager<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::shared_ptr<class op::Worker<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > >,class op::Queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::queue<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::deque<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > >,class std::allocator<class std::shared_ptr<class std::vector<class std::shared_ptr<struct op::Datum>,class std::allocator<class std::shared_ptr<struct op::Datum> > > > > > > > >::stop():239

Type of Issue

You might select multiple topics, delete the rest:

Execution error
Help wanted

Your System Configuration

https://pastebin.com/NQqRTQY4
OpenPose version: Release 1.70 + source download from today
General configuration:
- Installation mode: Exe Demo
- Operating system Windows
- Operating system version 10
- Release or Debug mode? Release
- Compiler: NONE (demo)
If GPU mode issue:
- CUDA version 10.1
- cuDNN version: 7.5.1
- GPU model GTX970
If Windows system:
- Portable demo or compiled library? Portable

Thanks!

gineshidalgo99 commented 3 years ago

For the Windows demo, the most likely issue is that your Nvidia drivers are not the latest. Also, if you are using the demo, you do not need to install CUDA, the demo already contains its own CUDA DLLs internally.

For Ubuntu, I am very surprised with are getting that error, given that you have to compile OpenPose from scratch, it should pick the right Nvidia/CUDA versions. Maybe your Nvidia version is also old in there?

j2l commented 3 years ago

Thank you very much @gineshidalgo99 Indeed, since CUDA 10.1 was coming with driver v418, I thought my v432 was "new enough". Latest driver is v457.51 (https://www.nvidia.com/Download/index.aspx?lang=en-us for other users)

I wonder if openpose needs DCH specific driver type though:

After clean installing latest drivers:

(I need to reboot, will post as is, and edit afterwards) (I'll also need to reboot in Ubuntu 20.04 to give more details about my build and error when running demo)

EDIT: Youwere right, new driver makes it works using : bin\OpenPoseDemo.exe --video examples\media\video.avi --net_resolution -1x256 to prevent out of memory.

I'll switch to ubuntu and come back here if you don't mind. I'd prefer to use openpose on linux 😄

j2l commented 3 years ago

OK, now on Ubuntu 20.04, same GC, drivers are the most recent ones from my distro: nvidia-smi

Sun Dec  6 16:01:10 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     Off  | 00000000:01:00.0  On |                  N/A |
|  0%   45C    P8    22W / 163W |    545MiB /  4039MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

I'm redoing everything from scratch and post steps here. It looks like latest drivers are an issue for my Ubuntu (Pop_OS): And cuda-runtime package wants NVidia cuda-drivers, not tier builds: We can't easliy switch drivers on Pop_OS: https://github.com/pop-os/nvidia-graphics-drivers/issues/65

With the runfile wget https://developer.download.nvidia.com/compute/cuda/11.1.1/local_installers/cuda_11.1.1_455.32.00_linux.run sudo sh cuda_11.1.1_455.32.00_linux.run

Driver is older than my Pop_OS, so installing only the toolkit:

Also re-installed cuDNN as root:

in build/, cmake-gui (3.16.3) > configure:

The C compiler identification is GNU 9.3.0
The CXX compiler identification is GNU 9.3.0
Check for working C compiler: /usr/bin/cc
Check for working C compiler: /usr/bin/cc -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Detecting C compile features
Detecting C compile features - done
Check for working CXX compiler: /usr/bin/c++
Check for working CXX compiler: /usr/bin/c++ -- works
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Detecting CXX compile features
Detecting CXX compile features - done
GCC detected, adding compile flags
GCC detected, adding compile flags
Looking for pthread.h
Looking for pthread.h - found
Performing Test CMAKE_HAVE_LIBC_PTHREAD
Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
Looking for pthread_create in pthreads
Looking for pthread_create in pthreads - not found
Looking for pthread_create in pthread
Looking for pthread_create in pthread - found
Found Threads: TRUE  
Found CUDA: /usr/local/cuda (found version "11.1") 
Building with CUDA.
CUDA detected: 11.1
Found cuDNN: ver. 8.0.5 found (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
Added CUDA NVCC flags for: sm_52
Found cuDNN: ver. 8.0.5 found (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
Found GFlags: /usr/include  
Found gflags  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
Found Glog: /usr/include  
Found glog    (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
Found Protobuf: /usr/local/lib/libprotobuf.so;-lpthread (found version "3.14.0") 
Found OpenCV: /usr (found version "4.2.0") 
Caffe will be downloaded from source now. NOTE: This process might take several minutes depending
        on your internet connection.
Caffe has already been downloaded.
HEAD is now at 1807aada Added Ampere arch's (CUDA11)

M   cmake/Cuda.cmake

Caffe will be built from source now.
Download the models.
Downloading BODY_25 model...
Model already exists.
Not downloading body (COCO) model
Not downloading body (MPI) model
Downloading face model...
Model already exists.
Downloading hand model...
Model already exists.
Models Downloaded.
Configuring done

sudo apt-get install python3-dev && sudo pip3 install numpy opencv-python is not necessary since it's already installed:

opencv-python in /usr/local/lib/python3.8/dist-packages (4.4.0.46)
numpy in /usr/local/lib/python3.8/dist-packages (1.19.4)

make -jnproc` https://pastebin.com/53EdaypH

And finally: ./build/examples/openpose/openpose.bin --video examples/media/video.avi

Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
F1206 17:31:34.511098 48830 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (1 vs. 0)  CUDNN_STATUS_NOT_INITIALIZED
*** Check failure stack trace: ***
    @     0x7f31c76f21c3  google::LogMessage::Fail()
    @     0x7f31c76f725b  google::LogMessage::SendToLog()
    @     0x7f31c76f1ebf  google::LogMessage::Flush()
    @     0x7f31c76f26ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f31c7325895  caffe::CuDNNConvolutionLayer<>::LayerSetUp()
    @     0x7f31c740800d  caffe::Net<>::Init()
    @     0x7f31c740a410  caffe::Net<>::Net()
    @     0x7f31c7dbc7bc  op::NetCaffe::initializationOnThread()
    @     0x7f31c7dd67b5  op::addCaffeNetOnThread()
    @     0x7f31c7dd7a0a  op::PoseExtractorCaffe::netInitializationOnThread()
    @     0x7f31c7ddc455  op::PoseExtractorNet::initializationOnThread()
    @     0x7f31c7dd4167  op::PoseExtractor::initializationOnThread()
    @     0x7f31c7dcf827  op::WPoseExtractor<>::initializationOnThread()
    @     0x7f31c7e02ed7  op::Worker<>::initializationOnThreadNoException()
    @     0x7f31c7e03028  op::SubThread<>::initializationOnThread()
    @     0x7f31c7e04858  op::Thread<>::initializationOnThread()
    @     0x7f31c7e07f1c  op::Thread<>::threadFunction()
    @     0x7f31c7a31d84  (unknown)
    @     0x7f31c7732609  start_thread
    @     0x7f31c7870293  clone
Aborted (core dumped)

And using net_resolution:

./build/examples/openpose/openpose.bin --video examples/media/video.avi --net_resolution -1x256
Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
F1206 17:33:56.299784 49333 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0)  CUDNN_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
    @     0x7f463c1701c3  google::LogMessage::Fail()
    @     0x7f463c17525b  google::LogMessage::SendToLog()
    @     0x7f463c16febf  google::LogMessage::Flush()
    @     0x7f463c1706ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f463bda3895  caffe::CuDNNConvolutionLayer<>::LayerSetUp()
    @     0x7f463be8600d  caffe::Net<>::Init()
    @     0x7f463be88410  caffe::Net<>::Net()
    @     0x7f463c83a7bc  op::NetCaffe::initializationOnThread()
    @     0x7f463c8547b5  op::addCaffeNetOnThread()
    @     0x7f463c855a0a  op::PoseExtractorCaffe::netInitializationOnThread()
    @     0x7f463c85a455  op::PoseExtractorNet::initializationOnThread()
    @     0x7f463c852167  op::PoseExtractor::initializationOnThread()
    @     0x7f463c84d827  op::WPoseExtractor<>::initializationOnThread()
    @     0x7f463c880ed7  op::Worker<>::initializationOnThreadNoException()
    @     0x7f463c881028  op::SubThread<>::initializationOnThread()
    @     0x7f463c882858  op::Thread<>::initializationOnThread()
    @     0x7f463c885f1c  op::Thread<>::threadFunction()
    @     0x7f463c4afd84  (unknown)
    @     0x7f463c1b0609  start_thread
    @     0x7f463c2ee293  clone
Aborted (core dumped)

Ok, it was another error before, now it's CUDNN. I noticed it installed 8.0.5, should I search for 8.0.4? Thank you again @gineshidalgo99

j2l commented 3 years ago

Also tried running (not building) with cuDNN v8.0.4 (September 28th, 2020), for CUDA 11.1, same error.

gineshidalgo99 commented 3 years ago

That's a different error! That's just saying cuDNN was not installed, so that's an easy one to fix! Could you make sure to install the latest cuDNN on your Ubuntu in the same folder where CUDA is?

j2l commented 3 years ago

I agree it's a different issue since the windows one is fixed.

Could you make sure to install the latest cuDNN on your Ubuntu in the same folder where CUDA is?

I think that's what I've done. As root, I extracted include files into /usr/local/cuda/include and lib64 into /lib64 (targeting targets/x86_64-linux/lib). /usr/local/cuda/include/cudnn.h /usr/local/cuda/include/cudnn_adv_infer.h /usr/local/cuda/include/cudnn_adv_train.h /usr/local/cuda/include/cudnn_backend.h /usr/local/cuda/include/cudnn_cnn_infer.h /usr/local/cuda/include/cudnn_cnn_train.h /usr/local/cuda/include/cudnn_ops_infer.h /usr/local/cuda/include/cudnn_ops_train.h /usr/local/cuda/include/cudnn_version.h and /usr/local/cuda/lib64/libcudnn.so /usr/local/cuda/lib64/libcudnn.so.8 /usr/local/cuda/lib64/libcudnn.so.8.0.4 /usr/local/cuda/lib64/libcudnn.so.8.0.5 /usr/local/cuda/lib64/libcudnn_adv_infer.so /usr/local/cuda/lib64/libcudnn_adv_infer.so.8 /usr/local/cuda/lib64/libcudnn_adv_infer.so.8.0.4 /usr/local/cuda/lib64/libcudnn_adv_infer.so.8.0.5 /usr/local/cuda/lib64/libcudnn_adv_train.so /usr/local/cuda/lib64/libcudnn_adv_train.so.8 /usr/local/cuda/lib64/libcudnn_adv_train.so.8.0.4 /usr/local/cuda/lib64/libcudnn_adv_train.so.8.0.5 /usr/local/cuda/lib64/libcudnn_cnn_infer.so /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8 /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8.0.4 /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8.0.5 /usr/local/cuda/lib64/libcudnn_cnn_train.so /usr/local/cuda/lib64/libcudnn_cnn_train.so.8 /usr/local/cuda/lib64/libcudnn_cnn_train.so.8.0.4 /usr/local/cuda/lib64/libcudnn_cnn_train.so.8.0.5 /usr/local/cuda/lib64/libcudnn_ops_infer.so /usr/local/cuda/lib64/libcudnn_ops_infer.so.8 /usr/local/cuda/lib64/libcudnn_ops_infer.so.8.0.4 /usr/local/cuda/lib64/libcudnn_ops_infer.so.8.0.5 /usr/local/cuda/lib64/libcudnn_ops_train.so /usr/local/cuda/lib64/libcudnn_ops_train.so.8 /usr/local/cuda/lib64/libcudnn_ops_train.so.8.0.4 /usr/local/cuda/lib64/libcudnn_ops_train.so.8.0.5 /usr/local/cuda/lib64/libcudnn_static.a

Should I do something else? Reboot?

kzwrime commented 3 years ago

I meet the same problem. When I run ./build/examples/openpose/openpose.bin --video examples/media/video.avi , even though I add --net_resolution to change into small size, my GPU memory use will reach 100% (use nvtop to monitor) and then the same error occur as @j2l show. I used to attribute the problem to the lack of gpu memory because my mx150 only ans 2GB. But @j2l 's 970 with 4GB also fail. The error CUDNN_STATUS_NOT_INITIALIZED let me test my caffe which uses CUDNN, but all test are passed and nothing wrong happen.

j2l commented 3 years ago

@kzwrime, please let me know if you find a solution. I'm stuck.

On Sun, Dec 13, 2020 at 4:18 AM kzwrime notifications@github.com wrote:

I meet the same problem. When I run ./build/examples/openpose/openpose.bin --video examples/media/video.avi , even though I add --net_resolution to change into small size, my GPU memory use will reach 100% (use nvtop to monitor) and then the same error occur as @j2l https://github.com/j2l show. I used to attribute the problem to lack of gpu memory because my mx150 only ans 2GB. But @j2l https://github.com/j2l 's 970 with 4GB also fail. The error CUDNN_STATUS_NOT_INITIALIZED let me test my caffe which uses CUDNN, but all test are passed and nothing wrong happen.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1786#issuecomment-743941342, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAP6LK3YANTMO2IWRXXXIDSUQXBPANCNFSM4UPK3V5Q .

kzwrime commented 3 years ago

@j2l I plan to try older versions of cuda and cudnn, or use nvidia-docker to create a suitable environment. After all, I am still puzzle about how to solve the problem on ubuntu 20 with newest cuda, cudnn and caffe-gpu.

kzwrime commented 3 years ago

After testing, I found that the docker-image from https://github.com/davegreenwood/openpose-GPU-docker can successfully run on my computer. @j2l Maybe you can try it.

root@768f8c9d1524:/workspace# openpose -display=0 -video /data/test.webm -write_json="./testoutput/sample-default.json" -write_video="./testoutput/default.avi" --net_resolution=160x80
Starting OpenPose demo...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
Starting thread(s)...
Empty frame detected, frame number 331 of 333. In /software/openpose/src/openpose/producer/producer.cpp:checkFrameIntegrity():179
Empty frame detected, frame number 331 of 333. In /software/openpose/src/openpose/producer/producer.cpp:checkFrameIntegrity():179
Empty frame detected, frame number 331 of 333. In /software/openpose/src/openpose/producer/producer.cpp:checkFrameIntegrity():179
OpenPose demo successfully finished. Total time: 18.709721 seconds.

2020-12-15 23-18-05 的屏幕截图

j2l commented 3 years ago

@kzwrime, could you please help me on running this container? I tried docker run --runtime=nvidia -v /home/pm/Documents/openposedata:/data -it openpose:gpu bash but it whines: docker: Error response from daemon: Unknown runtime specified nvidia according to https://github.com/NVIDIA/nvidia-docker/issues/1236 it should now be docker run --gpus all nvidia/cuda:10.0-base nvidia-smi but now it ithrows: docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

I usually run containers with portainer, but it's not ready for GPU yet (1y+ issue : https://github.com/portainer/portainer/issues/3143)

Would you please break down your steps to make it run?

kzwrime commented 3 years ago

@j2l The steps on https://github.com/davegreenwood/openpose-GPU-docker and https://github.com/hmurari/openpose-docker may be a little out of date. Some steps may need to be modified. Here are my steps.

Before the following, the nvidia docker is supposed to be installed and tested successfully.

# download the docker image, maybe a little slow
docker pull dgrnwd/openpose-gpu

# “~/mydata” is your date dir
sudo docker run --runtime=nvidia  \
    -v ~/mydata:/data \
    -it dgrnwd/openpose-gpu:latest  bash

# use test.mp4 as a example
openpose -display=0 -video /data/test.mp4 \
        -write_json="./testoutput/test-output.json" \
    -write_video="./testoutput/test-output.avi" --net_resolution=160x80

# it may take some time, the output is shown below

# copy the result to the host
cp -r ./testoutput /data/

root@768f8c9d1524:/workspace# openpose -display=0 -video /data/test.webm -write_json="./testoutput/sample-default.json" -write_video="./testoutput/default.avi" --net_resolution=160x80
Starting OpenPose demo...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
Starting thread(s)...
^C
root@768f8c9d1524:/workspace# openpose -display=0 -video /data/test-wzk.webm -write_json="./testoutput/sample-default.json" -write_video="./testoutput/default.avi" --net_resolution=160x80
Starting OpenPose demo...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
Starting thread(s)...
Empty frame detected, frame number 331 of 333. In /software/openpose/src/openpose/producer/producer.cpp:checkFrameIntegrity():179
Empty frame detected, frame number 331 of 333. In /software/openpose/src/openpose/producer/producer.cpp:checkFrameIntegrity():179
Empty frame detected, frame number 331 of 333. In /software/openpose/src/openpose/producer/producer.cpp:checkFrameIntegrity():179
OpenPose demo successfully finished. Total time: 18.709721 seconds.

# use camera
xhost +local:docker
docker run --gpus all -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix --ipc=host --device=/dev/video0:/dev/video0 -it dgrnwd/openpose-gpu:latest bash
openpose --net_resolution=160x80
# ctrl c to stop

j2l commented 3 years ago

@kzwrime thank you so much!

If someone can't make it work in the future. Maybe, like me you were missing a few extra steps:

sudo apt install nvidia-container-toolkit
nvidia-container-cli -k -d /dev/tty info #just to check
sudo systemctl daemon-reload
sudo systemctl restart docker

KeundongKang commented 3 years ago

Hi, I'm using Ubuntu20.04, CUDA-11.2, CUDNN 8.1.0 and having same problem -> Screenshot 2021-03-04 15:48:53

So now I'm tyring to install nvidia-container-toolkit but terminals shows that it is unable to locate the package. -> Screenshot 2021-03-04 15:57:21

Is there a reason for this problem? I tried to reinstall cuda and cudnn several times but still can't find solution...

KeundongKang commented 3 years ago

I just checked that my GPU memory use become full when I execute example file. I'm using RTX 2080 Super 8GB. Is it not enough?

KeundongKang commented 3 years ago

I just found that it is possible to run the demo like below.

./build/examples/openpose/openpose.bin --video examples/media/video.avi

I think enabling face and hand detection causes some trouble with cuda or cudnn because my graphic card is much better than openpose requires(RTX 2080 SUPER 8GB).

zeason commented 3 years ago

Hi, I am using openpose in docker image on AWS p2.xlarge instance, Ubuntu20.04 AMI. And here is my Dockerfile

FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04
# prevent tzdata etc. to prompt during installation
ENV DEBIAN_FRONTEND=noninteractive

RUN apt -y --no-install-recommends update && \
    apt -y --no-install-recommends upgrade

# preliminaries
RUN apt install -y \
    cmake \
    libopencv-dev \
    wget \
    vim \
    software-properties-common

RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        cmake \
        git \
        wget \
        libatlas-base-dev \
        libboost-all-dev \
        libgflags-dev \
        libgoogle-glog-dev \
        libhdf5-serial-dev \
        libleveldb-dev \
        liblmdb-dev \
        libopencv-dev \
        libprotobuf-dev \
        libsnappy-dev \
        protobuf-compiler

# python3 libs
RUN apt install -y \
    python3-setuptools \
    python3-dev
RUN hash pip3 2> /dev/null || apt --assume-yes install python3-pip
RUN python3 -m pip install pip --upgrade
RUN pip3 install numpy opencv-python boto3

# build OpenPose
RUN git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose.git
RUN mkdir -p /openpose/build && \
    cd /openpose/build
COPY models /openpose/models
WORKDIR /openpose/build
RUN cmake -D BUILD_PYTHON=ON -D BUILD_CAFFE=ON \
    -D BUILD_EXAMPLES=ON \
    -D GPU_MODE=CUDA \
    -D CMAKE_BUILD_TYPE=Release ..
RUN make -j`nproc` && make install

WORKDIR /openpose

RUN mkdir s3_download
RUN mkdir output_json_folder

Building openpose successfully but when running the demo and examples Check failed: status == CUDNN_STATUS_SUCCESS (1 vs. 0) CUDNN_STATUS_NOT_INITIALIZED occured.

MPForte commented 3 years ago

Are there any updates about this issue? I have cuda 11.1, cudnn 8.0.4 and I get the same error shown in the last comment

bl0up commented 3 years ago

Hi, I also have the CUDNN_STATUS_NOT_INITIALIZED error on ubuntu 20.04 with cuda 11.1 and cudnn 8.1. Has someone found a solution?

Thanks!

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

nlsfnr commented 2 years ago

For the people still looking for an answer, this Dockerfile works and supports Python3. It basically boils down to disabling cuDNN to avoid the CUDNN_STATUS_NOT_INITIALIZED error.

FROM nvidia/cuda:11.6.1-devel-ubuntu20.04

# Install dependencies for OpenPose and its installation
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update --yes
RUN apt-get install --yes --auto-remove \
        libopencv-dev \
        protobuf-compiler \
        libgoogle-glog-dev \
        libboost-all-dev \
        libhdf5-dev \
        libatlas-base-dev \
        git \
        cmake \
        python3-dev \
        python3-pip

# Install Pip dependencies
RUN python3 -m pip install --upgrade pip && \
    python3 -m pip install \
        numpy \
        opencv-python

# Clone the OpenPose repository and compile it
WORKDIR /openpose
RUN git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose.git ./ && \
    git submodule update --init --recursive --remote
WORKDIR  /openpose/build/
RUN cmake \
        -D BUILD_PYTHON=ON \
        -D BUILD_CAFFE=ON \
        -D BUILD_EXAMPLES=ON \
        -D GPU_MODE=CUDA \
        -D USE_CUDNN=OFF \
        -D CMAKE_BUILD_TYPE=Release \
        -D DOWNLOAD_BODY_COCO_MODEL=ON \
        -D DOWNLOAD_BODY_MPI_MODEL=ON \
        -D DOWNLOAD_HAND_MODEL=ON \
        -D DOWNLOAD_FACE_MODEL=ON \
        ..
RUN make --jobs `nproc`

CMU-Perceptual-Computing-Lab / openpose