search

CMU-Perceptual-Computing-Lab / openpose

OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation
https://cmu-perceptual-computing-lab.github.io/openpose
Other
30.53k stars 7.81k forks source link

Performance drop in docker(GPU) #1610

Closed andriy-tsvikla closed 4 years ago

andriy-tsvikla commented 4 years ago

Issue Summary

   I have built openpose with python wrapper on ubuntu 18.04 with T4 GPU on virtual machine 
   and in docker. In docker it works 4 times slower ~10ms vs ~38ms per image.
   Dependencies are the same except docker version uses nvidia-docker and anaconda.
   Before deploying openpose i tried some pythorch models and there were no difference in 
   performance.
   Perhaps someone came across the same problem? Any idea how i can debug/fix this?

Type of Issue

You might select multiple topics, delete the rest:

Your System Configuration

  1. OpenPose version: Latest GitHub code? Or specific commit (e.g., d52878f)? Or specific version from Release section (e.g., 1.2.0)? 1.6

  2. General configuration:

    • Installation mode: CMake, sh script, manual Makefile installation, ... (Ubuntu); CMake, ... (Windows); ...? CMake
    • Operating system (lsb_release -a in Ubuntu): Ubuntu
    • Operating system version (e.g., Ubuntu 16, Windows 10, ...): 18.04
    • Release or Debug mode? (by default: release): Release
    • Compiler (gcc --version in Ubuntu or VS version in Windows): 5.4.0, ... (Ubuntu); VS2015 Enterprise Update 3, VS2017 community, ... (Windows); ...? gcc 7.5.0

  3. 3rd-party software:

    • CMake version (cmake --version in Ubuntu): 3.17.3
    • OpenCV version: pre-compiled apt-get install libopencv-dev (only Ubuntu); OpenPose default (only Windows); compiled from source? If so, 2.4.9, 2.4.12, 3.1, 3.2?; ...? 3.2

  4. If GPU mode issue:

    • CUDA version (cat /usr/local/cuda/version.txt in most cases): 10.1.243
    • cuDNN version: 7.6.5
    • GPU model (nvidia-smi in Ubuntu): Tesla T4

  5. If Python API:

    • Python version: 2.7, 3.7, ...? 3.6.9
    • Numpy version (python -c "import numpy; print numpy.version.version" in Ubuntu): 1.18.5

  6. If speed performance issue:

    • Report OpenPose timing speed based on this link. 
      /openpose/include/openpose/thread/wIdGenerator.hpp:work():62
0.001470 msec in /openpose/include/openpose/thread/wIdGenerator.hpp:work():76
/openpose/include/openpose/thread/wIdGenerator.hpp:work():78
/openpose/include/openpose/core/wScaleAndSizeExtractor.hpp:work():62
0.005766 msec in /openpose/include/openpose/core/wScaleAndSizeExtractor.hpp:work():74
/openpose/include/openpose/core/wScaleAndSizeExtractor.hpp:work():76
/openpose/include/openpose/core/wCvMatToOpInput.hpp:work():61
0.625146 msec in /openpose/include/openpose/core/wCvMatToOpInput.hpp:work():70
/openpose/include/openpose/core/wCvMatToOpInput.hpp:work():72
/openpose/include/openpose/pose/wPoseExtractor.hpp:work():69
60.458142 msec in /openpose/include/openpose/pose/wPoseExtractor.hpp:work():98
/openpose/include/openpose/pose/wPoseExtractor.hpp:work():100
processing: 0:00:00.038658

processing is my attempt to log processing time. Its python datetime difference between end time and start time.

andriy-tsvikla commented 4 years ago

My mistake, performance drops by ~10%