RealSense2OpenPose3D

This project provides a simple way to use an Intel RealSense depth camera with OpenPose to get 3D keypoints. It was first developed for a Master's project while doing an internship at Advanced Telecommunications Research Institute International (ATR).

• Use
• Installation
  • Install RealSense SDK
  • Install OpenPose
  • Download RealSense2OpenPose3D exe
  • Compile RealSense2OpenPose3D
• Tips

Use

1. After installing all required components and either compiling or downloading OpenPose2RealSense3D, edit the default paths in the launch.py file to match your layout.
   1. In particular, change openPosePath, openPoseOutputPath, RealSense2OpenPoseEXE, and PointViewer.
2. Run the program by entering python .\launch.py into a console where the launch file is located.
3. Arguments (do not enter spaces after the '='):
   1. frames= number >= -1. Is the number of frames beyond 10 that will not be deleted during run time (-1 is save all)
   2. view= True or false. Whether to start the point viewer or not
   3. quit= An alphanumeric character. This determines which keyboard key will terminate the program
   4. d= float number >= 0. The depth limit beyond which values are ignored
   5. lr= float number >= 0,float number >= 0. The right and left limits of the point viwer in meters
   6. ud= float number >= 0,float number >= 0. The up and down limits of the point viwer in meters
   7. color-res= integer number x integer number. The resolution of the color sensor of the camera, defaults to 1920x1080
   8. face= True or False. Detect hands or not, defaults to false
   9. hand= True or False. Detect face or not, defaults to false
   10. output= <path\to\openPoseOutputFolder>. The full path to the OpenPose output folder you would like to use.
   11. r2oexe= <path\to\RS2OP3D.exe> The full path to and including the RS2OP3D.exe file.
   12. Other input will yield the help menu
   13. Example: python .\launch.py frames=-1 view=true quit=q d=2.5 lr=1.5,1.5 ud=1,1 color-res=1280x720 face=false hand=true output=C:\Users\Bingus\Desktop\output r2oexe=C:\Users\Bingus\Desktop\RealSense2OpenPose3D\64bit\RS2OP3D.exe
4. The output files will be marked as ############_keypointsD.json in the output folder
5. See OpenPose's documentation for the format of the output JSON files

Installation

This guide will walk you through all required components.

Install RealSense SDK:

1. Download RealSense SDK v2.34
   1. Note: The current version at the time of writing this is v2.50.0. However, there is a bug that prevents the depth and image alignment using “software devices” that was introduced in some build after v2.34.0
   2. See this issue for more info

Install OpenPose:

1. Prerequisites: Prerequisite List
   1. CMake GUI
      1. CMake download page
      2. Download and install cmake-#.#.#-win64-x64.msi
   2. Install Microsoft Visual Studio Community 2019
      1. Visual Studio download page
      2. Run the installer after downloading the executable
      3. Select the C++ console option and then all checkboxes on the right that say C++ in them
      4. Click Install
      5. Restart your computer
   3. Install CUDA and CuDNN
      1. Note: You must wait until after installing Visual Studio before proceeding to this step!
      2. Install CUDA 11.11 for 30 series GPUs CUDA download page
      3. CuDNN download page
      4. Merge CuDNN files with C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1
   4. Install python: Python download page
      1. Install open-cv:
      2. In an admin powershell run: pip install numpy opencv-python
2. Clone OpenPose:
   1. Create a new folder C:/Program Files/OpenPose
   2. With an administrator powershell run the following commands

      git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose
      cd openpose/
      git submodule update --init --recursive --remote

3. CMake Configuration
   1. Enter the OpenPose/openpose directory
   2. Make a new folder named “build”
   3. Enter that new folder
   4. Run CMake: cmake-gui ..
   5. Make sure that the source code path field is …OpenPose/openpose and that the build directory is …/OpenPose/openpose/build
   6. Click “Configure”
   7. Select the version of Visual Studio that is installed and select x64
   8. Click “Finish”
   9. Make sure that the GPU mode is set to CUDA, WITH_3D_RENDERER is on, and WITH_FLIR_CAMERA is off
   10. Click “Configure” one more time
   11. Click “Generate”
4. Compilation:
   1. Click on “Open Project” to open the Visual Studio solution
   2. Switch the configuration from “Debug” to “Release”
   3. Press “F7” (Build)
   4. Copy all the .dll files from …/build/bin to …/build/x64
5. Test that it works
   1. Go to …/OpenPose/openpose/
   2. Run an example like: build/x64/Release/OpenPoseDemo.exe --video examples/media/video.avi
   3. Using a camera: build/x64/Release/OpenPoseDemo.exe --hand --face --camera 1

Download RealSense2OpenPose3D exe

This is the easiest way to get up and running.

1. Download here
2. Place the whole folder, including all the .dll files, where you would like.

Compile RealSense2OpenPose3D

1. Create a new empty C++ Project in Visual Studio
2. Add the Intel RealSenseSDK 2.0 Property sheets
   1. View -> Other Windows -> Property Manager
   2. Right click on project name in the window that just opened
   3. Add existing property sheet
   4. Navigate to the SDK directory “C:\Program Files (x86)\Intel RealSense SDK 2.0” in my case
   5. Select one of the .props files and click Open
   6. Repeat for the other two .props files
3. Test to see if it all works
   1. Find “rs-hello-realsense.cpp” under “Intel RealSense SDK 2.0\samples\hello-realsense”
   2. Add the file to the project
      1. Right click on Source Files in the Solution Explorer
      2. Add -> Existing Item
      3. Select “rs-hello-realsense.cpp” and click Add
   3. Run the program
      1. Click on the green arrow at the top of the IDE
4. Take a break. It wasn’t terrible, but figuring out how to do this wasn’t easy either.
5. Download the source for RealSense2OpenPose3D
   1. Download from here
6. Download the JSON library
   1. Go to JSON.hpp download or use the version included in the "source" folder from the previous step
   2. Download the latest “json.hpp”
   3. Send some thanks in the direction of the creators
   4. Save the file in your working directory for the project and double check that the #include statement in RealSense2OpenPose.cpp has the correct path
7. Compile!

Tips

• If you are not getting the framerate you want
  • This program can run at a maximum of ~40fps, if the pose detection exceeds this, the program will crash. OpenPose's framerate can be limited in the launch.py file if this happens.
  • You can increase the speed of OpenPose by installing a better or second GPU
  • In the launch.py file, you may alter the OpenPose launch flags to reduce the computational load
• The output files are disapearing
  • The launch.py program automatically deletes the output files as it runs to prevent filling up your hard drive
  • If you would like to keep all files, run the launch file with the argument frames=-1
  • If you would like to keep only the past # many, and be prompted to delete them or not at the end, then run the launch file with the argument frames=#
• While typing something, the program stops
  • The program has q as the default quit key.
  • This can be changed with the quit=<key name> argument to the launch file.