Caffe not using GPU but occupies GPU memory

[swapnilsayansaha]

Issue summary.

I installed Caffe on a GPU machine, with CUDA 10.0 properly configured, Python: 3.7 (Anaconda3) and CUDNN 7.6.5. I successfully installed all dependencies of Caffe and tested the functionalities via runtest and pytest commands. The commands run without errors and Caffe seems to also use the GPU a bit during the tests, which is fine. I then try to run the Pascal VOC Jupyter notebook example that comes with Caffe (had to edit a couple of lines like print and image_resize for Python 3 compatibility). It successfully goes through all the sections and nvidia-smi shows the process occupying ~ 800 MB of memory in GPU. However, during the iteration steps, it seems Caffe is not using the GPU. The CPU usage goes to 100% on 16 out of 32 cores but the GPU usage remains at 2-3%. This makes training process extremely slow.

Steps to reproduce

Install Caffe on a GPU machine using guides listed in tried solutions section and run Pascal VOC example.

Tried solutions

I thought it might be a build problem with Caffe and I tried to rebuild Caffe using a variety of different guides for both Python 2 and Python 3 environments using reference from the following guides. I ran into a huge list of trouble given the difficult installation process of Caffe but managed to install Caffe properly all the time in the end. The issue, however, remained the same: Caffe occupies GPU memory but does not use it.

1. https://caffe.berkeleyvision.org/installation.html
2. https://medium.com/@atinesh/caffe-installation-on-ubuntu-18-04-lts-python-3-6-e76375f0d353
3. https://qengineering.eu/install-caffe-on-ubuntu-18.04-with-opencv-4.1.html
4. https://jin-zhe.github.io/guides/installing-caffe-with-cuda-in-conda/
5. https://medium.com/@anidh.singh/install-caffe-on-ubuntu-with-cuda-6d0da9e8f860
6. https://anaconda.org/anaconda/caffe-gpu (doesn't work)
7. https://yangcha.github.io/Caffe-Conda/
8. https://gist.github.com/arundasan91/b432cb011d1c45b65222d0fac5f9232c
9. https://gist.github.com/FrancoisPl/e7375c3a08c1b73d5547709e97405253

I checked if CUDA was properly configured as well. Tensorflow could readily use GPU cores without any issues.

System configuration

• Operating system: Ubuntu 18.04
• Compiler: Cmake
• CUDA version (if applicable): 10.0
• CUDNN version (if applicable): 7.6.5
• BLAS: OpenBlas, Atlas (used)
• Python version (if using pycaffe): 3.6.9
• MATLAB version (if using matcaffe): -

Issue checklist

• [ X ] read the guidelines and removed the first paragraph
• [ X ] written a short summary and detailed steps to reproduce
• [ X ] explained how solutions to related problems failed (tick if found none)
• [ X ] filled system configuration
• [ X ] attached relevant logs/config files (tick if not applicable)

[aktaseren]

Can you please mention the Jupyter notebook you used for your Caffe experiment? Did you code any line for defining which GPUs to use?

[swapnilsayansaha]

I abandoned Caffe for TF/PyTorch. Caffe has too many bugs and poor documentation,