RNHVE Realsense Network Hardware Video Encoder

Realsense hardware video/depth encoding and streaming over custom MLSP protocol.

This includes streaming:

color (H.264, HEVC Main)
infrared/infrared-rgb (H.264, HEVC Main)
depth (HEVC Main10)
textured depth (HEVC Main10 + HEVC Main)

See unity-network-hardware-video-decoder as example network decoder & renderer (color, infrared and depth).

See how it works on wiki to understand the code.

See hardware-video-streaming for other related projects.

See videos to understand point cloud streaming features:

Point Cloud Streaming	Infrared Point Cloud Streaming	Point Cloud Streaming To UMPC

Platforms

Unix-like operating systems (e.g. Linux), more info.

Tested on Ubuntu 18.04.

Hardware

Tested on Intel KabyLake with Realsense D435, D455, L515.

Requires Intel VAAPI compatible hardware encoder (QuickSync Video). For depth encoding at least KabyLake.

Other technologies may also work but were not tested.

Infrared textured depth encoding is implemented for D415, D435, D455 and L515. Color textured depth encoding is implemented for D415, D435, D455, L515.

Dependencies

This program depends on:

librealsense2
NHVE Network Hardware Video Encoder
- HVE Hardware Video Encoder
  - FFmpeg avcodec, avutil, avfilter (at least 3.4 version)
- MLSP Minimal Latency Streaming Protocol

Install RealSense™ SDK 2.0 as described on github

NHVE and its dependencies are included as submodules so you only need to satifisy HVE dependencies.

Works with system FFmpeg on Ubuntu 18.04 and doesn't on 16.04 (outdated FFmpeg and VAAPI ecosystem).

Building Instructions

Tested on Ubuntu 18.04.

# update package repositories
sudo apt-get update 
# get avcodec and avutil
sudo apt-get install ffmpeg libavcodec-dev libavutil-dev libavfilter-dev
# get compilers and make 
sudo apt-get install build-essential
# get cmake - we need to specify libcurl4 for Ubuntu 18.04 dependencies problem
sudo apt-get install libcurl4 cmake
# get git
sudo apt-get install git
# clone the repository with *RECURSIVE* for submodules
git clone --recursive https://github.com/bmegli/realsense-network-hardware-video-encoder.git

# finally build the program
cd realsense-network-hardware-video-encoder
mkdir build
cd build
cmake ..
make

Running

Stream H.264 Realsense color/infrared/infrared-rgb video over UDP.

Usage: ./realsense-nhve-h264
       <host> <port>
       <color/ir/ir-rgb>
       <width> <height> <framerate> <seconds>
       [device] [bitrate]

examples:
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 127.0.0.1 9766 ir-rgb 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 192.168.0.125 9766 color 640 360 30 50 /dev/dri/renderD128 500000```

Stream Realsense:

color/infrared/infrared-rgb with HEVC Main
depth with HEVC Main10

Usage: ./realsense-nhve-hevc
       <host> <port>
       <color/ir/ir-rgb/depth>
       <width> <height> <framerate> <seconds>
       [device] [bitrate] [depth units] [json]

examples:
./realsense-nhve-hevc 127.0.0.1 9766 color 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 depth 640 360 30 5
./realsense-nhve-hevc 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 ir-rgb 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 127.0.0.1 9766 depth 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-hevc 192.168.0.125 9766 color 640 360 30 50 /dev/dri/renderD128 500000
./realsense-nhve-hevc 127.0.0.1 9768 depth 848 480 30 50 /dev/dri/renderD128 2000000
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.0001
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.00005
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.000025
./realsense-nhve-hevc 192.168.0.100 9768 depth 848 480 30 500 /dev/dri/renderD128 2000000 0.0000125
./realsense-nhve-hevc 192.168.0.100 9768 depth 640 480 30 500 /dev/dri/renderD128 8000000 0.0000390625 my_config.json

Stream Realsense D415/D435/D455/L515:

depth with HEVC Main10, infrared or infrared rgb with HEVC

Usage: ./realsense-nhve-depth-ir
       <host> <port>
       <ir/ir-rgb>
       <width> <height> <framerate> <seconds>
       [device] [bitrate_depth] [bitrate_ir] [depth units] [json]

examples: 
./realsense-nhve-depth-ir 127.0.0.1 9766 ir 640 360 30 5
./realsense-nhve-depth-ir 127.0.0.1 9766 ir-rgb 640 360 30 5
./realsense-nhve-depth-ir 127.0.0.1 9766 ir 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-depth-ir 192.168.0.125 9766 ir-rgb 640 360 30 50 /dev/dri/renderD128 4000000 1000000
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00005
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.000025
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125
./realsense-nhve-depth-ir 192.168.0.100 9768 ir 640 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json```

Stream Realsense D415/D435/D455/L515:

depth with HEVC Main10, color with HEVC (aligned)

Usage: ./realsense-nhve-depth-color
       <host> <port>
       <color/depth> # alignment direction
       <width_depth> <height_depth> <width_color> <height_color>
       <framerate> <seconds>
       [device] [bitrate_depth] [bitrate_color] [depth units] [json]

examples:
./realsense-nhve-depth-color 127.0.0.1 9766 color 640 360 640 360 30 5
./realsense-nhve-depth-color 127.0.0.1 9766 color 640 360 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-depth-color 192.168.0.125 9766 color 640 360 640 360 30 50 /dev/dri/renderD128 4000000 1000000
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00005
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.000025
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-color 192.168.0.100 9768 depth 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0000125
./realsense-nhve-depth-color 192.168.0.100 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125f
./realsense-nhve-depth-color 192.168.0.100 9768 depth 848 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.00003125f
./realsense-nhve-depth-color 192.168.0.100 9768 depth 640 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json
./realsense-nhve-depth-color 192.168.0.100 9768 color 640 480 1280 720 30 500 /dev/dri/renderD128 8000000 1000000 0.0000390625 my_config.json

If you don't have receiving end you will just see if hardware encoding worked/didn't work.

You may need to specify VAAPI device if you have more than one (e.g. NVIDIA GPU + Intel CPU).

If you get errors see also HVE troubleshooting.

License

Code in this repository and my dependencies are licensed under Mozilla Public License, v. 2.0

This is similiar to LGPL but more permissive:

you can use it as LGPL in prioprietrary software
unlike LGPL you may compile it statically with your code

Like in LGPL, if you modify the code, you have to make your changes available. Making a github fork with your changes satisfies those requirements perfectly.

Since you are linking to FFmpeg libraries consider also avcodec, avutil and avfilter licensing.

Additional information

The implementation down to FFmpeg (encoding) and up to MLSP (sending) is zero copy (passing the pointers to data) whenever possible.

bmegli / realsense-network-hardware-video-encoder

readme