Realsense hardware video/depth encoding and streaming over custom MLSP protocol.
This includes streaming:
See unity-network-hardware-video-decoder as example network decoder & renderer (color, infrared and depth).
See how it works on wiki to understand the code.
See hardware-video-streaming for other related projects.
See videos to understand point cloud streaming features:
Point Cloud Streaming | Infrared Point Cloud Streaming | Point Cloud Streaming To UMPC |
---|---|---|
![]() |
![]() |
![]() |
Unix-like operating systems (e.g. Linux), more info.
Tested on Ubuntu 18.04.
Tested on Intel KabyLake with Realsense D435, D455, L515.
Requires Intel VAAPI compatible hardware encoder (QuickSync Video). For depth encoding at least KabyLake.
Other technologies may also work but were not tested.
Infrared textured depth encoding is implemented for D415, D435, D455 and L515. Color textured depth encoding is implemented for D415, D435, D455, L515.
This program depends on:
avcodec
, avutil
, avfilter
(at least 3.4 version)Install RealSense™ SDK 2.0 as described on github
NHVE and its dependencies are included as submodules so you only need to satifisy HVE dependencies.
Works with system FFmpeg on Ubuntu 18.04 and doesn't on 16.04 (outdated FFmpeg and VAAPI ecosystem).
Tested on Ubuntu 18.04.
# update package repositories
sudo apt-get update
# get avcodec and avutil
sudo apt-get install ffmpeg libavcodec-dev libavutil-dev libavfilter-dev
# get compilers and make
sudo apt-get install build-essential
# get cmake - we need to specify libcurl4 for Ubuntu 18.04 dependencies problem
sudo apt-get install libcurl4 cmake
# get git
sudo apt-get install git
# clone the repository with *RECURSIVE* for submodules
git clone --recursive https://github.com/bmegli/realsense-network-hardware-video-encoder.git
# finally build the program
cd realsense-network-hardware-video-encoder
mkdir build
cd build
cmake ..
make
Stream H.264 Realsense color/infrared/infrared-rgb video over UDP.
Usage: ./realsense-nhve-h264
<host> <port>
<color/ir/ir-rgb>
<width> <height> <framerate> <seconds>
[device] [bitrate]
examples:
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 127.0.0.1 9766 ir-rgb 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 192.168.0.125 9766 color 640 360 30 50 /dev/dri/renderD128 500000```
Stream Realsense:
Usage: ./realsense-nhve-hevc
<host> <port>
<color/ir/ir-rgb/depth>
<width> <height> <framerate> <seconds>
[device] [bitrate] [depth units] [json]
examples:
./realsense-nhve-hevc 127.0.0.1 9766 color 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 depth 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 ir-rgb 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 depth 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 192.168.0.125 9766 color 640 360 30 50 /dev/dri/renderD128 500000
./realsense-nhve-hevc 127.0.0.1 9768 depth 848 480 30 50 /dev/dri/renderD128 2000000
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.0001
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.00005
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.000025
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.0000125
./realsense-nhve-hevc 192.168.0.100 9768 depth 640 480 30 500 /dev/dri/renderD128 8000000 0.0000390625 my_config.json
Stream Realsense D415/D435/D455/L515:
Usage: ./realsense-nhve-depth-ir
<host> <port>
<ir/ir-rgb>
<width> <height> <framerate> <seconds>
[device] [bitrate_depth] [bitrate_ir] [depth units] [json]
examples:
./realsense-nhve-depth-ir 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-depth-ir 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-depth-ir 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-depth-ir 192.168.0.125 9766 ir-rgb 640 360 30 50 /dev/dri/renderD128 4000000 1000000
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00005
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.000025
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 640 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json```
Stream Realsense D415/D435/D455/L515:
Usage: ./realsense-nhve-depth-color
<host> <port>
<color/depth> # alignment direction
<width_depth> <height_depth> <width_color> <height_color>
<framerate> <seconds>
[device] [bitrate_depth] [bitrate_color] [depth units] [json]
examples:
./realsense-nhve-depth-color 127.0.0.1 9766 color 640 360 640 360 30 5
./realsense-nhve-depth-color 127.0.0.1 9766 color 640 360 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-depth-color 192.168.0.125 9766 color 640 360 640 360 30 50 /dev/dri/renderD128 4000000 1000000
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00005
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.000025
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-color 192.168.0.100 9768 depth 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125f
./realsense-nhve-depth-color 192.168.0.100 9768 depth 848 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125f
./realsense-nhve-depth-color 192.168.0.100 9768 depth 640 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json
./realsense-nhve-depth-color 192.168.0.100 9768 color 640 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json
If you don't have receiving end you will just see if hardware encoding worked/didn't work.
You may need to specify VAAPI device if you have more than one (e.g. NVIDIA GPU + Intel CPU).
If you get errors see also HVE troubleshooting.
Code in this repository and my dependencies are licensed under Mozilla Public License, v. 2.0
This is similiar to LGPL but more permissive:
Like in LGPL, if you modify the code, you have to make your changes available. Making a github fork with your changes satisfies those requirements perfectly.
Since you are linking to FFmpeg libraries consider also avcodec
, avutil
and avfilter
licensing.
The implementation down to FFmpeg (encoding) and up to MLSP (sending) is zero copy (passing the pointers to data) whenever possible.