search

CMU-Perceptual-Computing-Lab / caffe_rtpose

Realtime C++ code for multi-person pose estimation
Other
356 stars 207 forks source link

It seems like out of memory #37

Closed lfxxxxx closed 7 years ago

lfxxxxx commented 7 years ago

I have compiled rtpose-caffe with cuDNN. (I have set USE_CUDNN:=1) but when I test in just 2 image ,it show "out of memory"

[lfx@gpu-com image]$ ls image2.jpg image3.jpg

[lfx@gpu-com caffe_rtpose-master]$ ./build/examples/rtpose/rtpose.bin --image_dir image/ F0407 10:27:17.102958 1129 syncedmem.cpp:64] Check failed: error == cudaSuccess (2 vs. 0) out of memory Check failure stack trace: @ 0x7efeaa736e6d (unknown) @ 0x7efeaa738ced (unknown) @ 0x7efeaa736a5c (unknown) @ 0x7efeaa73963e (unknown) @ 0x7efeb1905912 caffe::SyncedMemory::to_gpu() @ 0x7efeb1904c79 caffe::SyncedMemory::gpu_data() @ 0x7efeb177e762 caffe::Blob<>::gpu_data() @ 0x7efeb197b9d3 caffe::CuDNNConvolutionLayer<>::Forward_gpu() @ 0x7efeb18cdb8b caffe::Net<>::ForwardFromTo() @ 0x40bcff warmup() @ 0x411cbb processFrame() @ 0x7efe9ed99dc5 start_thread @ 0x7efe9eac873d __clone Aborted (core dumped)

I have two GPUs(GT730+TITAN X), it seems already run in TITAN X, but i can't make sure

how can I make sure it run in TITAN X? If it already run in TITAN X, how can i fix the error: out of memory?

lfxxxxx commented 7 years ago

[lfx@gpu-com image]$ nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2016 NVIDIA Corporation Built on Tue_Jan_10_13:22:03_CST_2017 Cuda compilation tools, release 8.0, V8.0.61

gineshidalgo99 commented 7 years ago

To be sure in which Nvidia you are running it, open 2 terminals, halp screen each one. Then run in one of them: watch -n 0.1 nvidia-smi

Execute our program from the other terminal and look to the terminal with the Nvidia information. Check which GPU is loading memory with the name of our program.

Let me know if my steps were not clear enough. Thanks!

lfxxxxx commented 7 years ago

Thanks! I finally find it run in the first GPU(GT730) Now, another question is coming up ,how can I choose the second gpu to run this

In origin caffe , I can use the flag -gpu to choose, but caffe-rtpose seems haven't this choice?(only num_gpu)

gineshidalgo99 commented 7 years ago

Sorry if its a little bit confusing, there is another flag for that, check --help to see the options. It was something like: device_start or start_device. We will change the name back to gpu_start in our next version.