RealSense2OpenPose3D
This project provides a simple way to use an Intel RealSense depth camera with OpenPose to get 3D keypoints. It was first developed for a Master's project while doing an internship at Advanced Telecommunications Research Institute International (ATR).
Use
- After installing all required components and either compiling or downloading OpenPose2RealSense3D, edit the default paths in the
launch.py
file to match your layout.
- In particular, change
openPosePath
, openPoseOutputPath
, RealSense2OpenPoseEXE
, and PointViewer
.
- Run the program by entering
python .\launch.py
into a console where the launch file is located.
- Arguments (do not enter spaces after the '='):
frames=
number >= -1. Is the number of frames beyond 10 that will not be deleted during run time (-1 is save all)
view=
True or false. Whether to start the point viewer or not
quit=
An alphanumeric character. This determines which keyboard key will terminate the program
d=
float number >= 0. The depth limit beyond which values are ignored
lr=
float number >= 0,
float number >= 0. The right and left limits of the point viwer in meters
ud=
float number >= 0,
float number >= 0. The up and down limits of the point viwer in meters
color-res=
integer number x
integer number. The resolution of the color sensor of the camera, defaults to 1920x1080
face=
True or False. Detect hands or not, defaults to false
hand=
True or False. Detect face or not, defaults to false
output=
<path\to\openPoseOutputFolder
>. The full path to the OpenPose output folder you would like to use.
r2oexe=
<path\to\RS2OP3D.exe
> The full path to and including the RS2OP3D.exe file.
- Other input will yield the help menu
- Example:
python .\launch.py frames=-1 view=true quit=q d=2.5 lr=1.5,1.5 ud=1,1 color-res=1280x720 face=false hand=true output=C:\Users\Bingus\Desktop\output r2oexe=C:\Users\Bingus\Desktop\RealSense2OpenPose3D\64bit\RS2OP3D.exe
- The output files will be marked as
############_keypointsD.json
in the output folder
- See OpenPose's documentation for the format of the output JSON files
Installation
This guide will walk you through all required components.
Install RealSense SDK:
- Download RealSense SDK v2.34
- Note: The current version at the time of writing this is v2.50.0. However, there is a bug that prevents the depth and image alignment using “software devices” that was introduced in some build after v2.34.0
- See this issue for more info
Install OpenPose:
- Prerequisites: Prerequisite List
- CMake GUI
- CMake download page
- Download and install
cmake-#.#.#-win64-x64.msi
- Install Microsoft Visual Studio Community 2019
- Visual Studio download page
- Run the installer after downloading the executable
- Select the C++ console option and then all checkboxes on the right that say
C++
in them
- Click
Install
- Restart your computer
- Install CUDA and CuDNN
- Note: You must wait until after installing Visual Studio before proceeding to this step!
- Install CUDA 11.11 for 30 series GPUs CUDA download page
- CuDNN download page
- Merge CuDNN files with
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1
- Install python: Python download page
- Install open-cv:
- In an admin powershell run:
pip install numpy opencv-python
- Clone OpenPose:
- Create a new folder
C:/Program Files/OpenPose
- With an administrator powershell run the following commands
git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose
cd openpose/
git submodule update --init --recursive --remote
- CMake Configuration
- Enter the
OpenPose/openpose
directory
- Make a new folder named “build”
- Enter that new folder
- Run CMake:
cmake-gui ..
- Make sure that the source code path field is
…OpenPose/openpose
and that the build directory is …/OpenPose/openpose/build
- Click “Configure”
- Select the version of Visual Studio that is installed and select x64
- Click “Finish”
- Make sure that the GPU mode is set to CUDA, WITH_3D_RENDERER is on, and WITH_FLIR_CAMERA is off
- Click “Configure” one more time
- Click “Generate”
- Compilation:
- Click on “Open Project” to open the Visual Studio solution
- Switch the configuration from “Debug” to “Release”
- Press “F7” (Build)
- Copy all the .dll files from
…/build/bin
to …/build/x64
- Test that it works
- Go to …/OpenPose/openpose/
- Run an example like:
build/x64/Release/OpenPoseDemo.exe --video examples/media/video.avi
- Using a camera:
build/x64/Release/OpenPoseDemo.exe --hand --face --camera 1
Download RealSense2OpenPose3D exe
This is the easiest way to get up and running.
- Download here
- Place the whole folder, including all the
.dll
files, where you would like.
Compile RealSense2OpenPose3D
- Create a new empty C++ Project in Visual Studio
- Add the Intel RealSenseSDK 2.0 Property sheets
- View -> Other Windows -> Property Manager
- Right click on project name in the window that just opened
- Add existing property sheet
- Navigate to the SDK directory
“C:\Program Files (x86)\Intel RealSense SDK 2.0”
in my case
- Select one of the
.props
files and click Open
- Repeat for the other two
.props
files
- Test to see if it all works
- Find
“rs-hello-realsense.cpp”
under “Intel RealSense SDK 2.0\samples\hello-realsense”
- Add the file to the project
- Right click on Source Files in the Solution Explorer
- Add -> Existing Item
- Select
“rs-hello-realsense.cpp”
and click Add
- Run the program
- Click on the green arrow at the top of the IDE
- Take a break. It wasn’t terrible, but figuring out how to do this wasn’t easy either.
- Download the source for RealSense2OpenPose3D
- Download from here
- Download the JSON library
- Go to JSON.hpp download or use the version included in the
"source"
folder from the previous step
- Download the latest “json.hpp”
- Send some thanks in the direction of the creators
- Save the file in your working directory for the project and double check that the #include statement in RealSense2OpenPose.cpp has the correct path
- Compile!
Tips
- If you are not getting the framerate you want
- This program can run at a maximum of ~40fps, if the pose detection exceeds this, the program will crash. OpenPose's framerate can be limited in the
launch.py
file if this happens.
- You can increase the speed of OpenPose by installing a better or second GPU
- In the
launch.py
file, you may alter the OpenPose launch flags to reduce the computational load
- The output files are disapearing
- The
launch.py
program automatically deletes the output files as it runs to prevent filling up your hard drive
- If you would like to keep all files, run the launch file with the argument
frames=-1
- If you would like to keep only the past
#
many, and be prompted to delete them or not at the end, then run the launch file with the argument frames=#
- While typing something, the program stops
- The program has
q
as the default quit key.
- This can be changed with the
quit=<key name>
argument to the launch file.