accelerated textured depth streaming (independently encoded)

[bmegli]

Working textured depth streaming is already implemented (see #2, #4 and video) however encoding is hacky and unoptimal.

Here the idea is to:

• hardware encode depth and texture separately (HEVC + HEVC or HEVC + H.264)
• wrap encoded data in a synchronized packet

The possible gains are:

• bitrate saving (in #2 hacky texture encoding is unoptimal)
• independent control over depth and texture bitrates
• flexibility (in #2 it is only possible to stream greyscale texture)

From depth encoding time benchmark it will only add a few ms of additional latency.

From #4 some of hardware encoding operations may be run concurrently even cutting the few ms latency.

[bmegli]

Problem analysis.

HVE

HVE already supports multiple concurrent hardware encoders.

MLSP

Multiple application level frames may be encoded in a single MLSP frame. It always is the responsibility of the application to interpret the data correctly.

Optimization is possible to avoid unnecessary copies of the data when preparing packets:

• MLSP could handle a list of data fragments (pointers) with size
• to avoid copying data when preparing packet (before fragmenting to MTU)
• similar mechanism could be used on receiving side
• adding such mechanisms would make MLSP aware of carrying multi-frame data

NHVE

Interface change:

• accepts list of hardware encoder configurations
• accepts list of video frames data to encode and send
• the old interface is a special case with a single encoder/frame

RNHVE

Subprogram that streams depth + ir (or anything + anything in general).

HVD

HVD already supports multiple concurrent hardware decoders.

NHVD

Interface change:

• accepts list of hardware decoder configurations
• returns lists of decoded frames

Possibly special case implementation for:

• depth + texture decoding -> textured unprojection
• just 2 concurrent streams (generic multi-stream C pointer interface from Unity would be horror)

Implementation-wise it is easy. Conceptually this further breaks the generic character of NHVD (maintenance, reusability).

UNHVD

Possibly no change:

• still wrapping native data for textured mesh
• the work is carried in NHVD

[bmegli]

Before proceeding.

Multi-frame approach

Here I mean for example sending depth and ir frame together:

• there is H.264 multiview profile, supported by hardware that I use, but here I need HEVC
• there is MV-HEVC extension, not supported on hardware that I use (is it supported on any already?)

A similar functionality is provided by containers (e.g. mkv, avi, mp4) with multiple streams:

• in a proof of cocnept
• I don't want to introduce complexity with muxing/demuxing
• where a few additional lines of code are sufficient for my needs

Multi-frame pipeline

It is possible to (for example):

• receive depth frame
• start decoding
• receive ir frame (while decoding)
• decode ir frame
• combine data
• which altogether would save some time

However with wireless lossy medium:

• we expect that we will be losing some data (we don't want TCP/IP)
• it is possible that:
  • we would receive depth frame
  • we would lose ir frame
• which would lead to different states of hardware decoders state
• which would lead to losing synchronized state between depth/ir frames

From engineering perspective:

• coupling data transport layer with application level logic is a bad idea
• which would make independent work on MLSP impossible

[bmegli]

Proof-of-concept finished across multi-frame branches in all repositories.

Subjective impressions - works much better, requires lower bitrate, needs more GPU

This needs some serious cleanup before merging.

[bmegli]

Finished and merged into master.

Needs some documentation.

[bmegli]

Documentation updated.

It would be nice to document new pipeline with video but this is distant future (if at all).

First:

• check if LattePanda Alpha can handle multi-frame encoding
• implement mechanism for duplicate packets in MLSP
• split NHVD to two libraries
• fix some of the artifacts in UNHVD
• do not unproject depth to point cloud under mutex (e.g. double buffering)

This is the minimum required for next "release".

[bmegli]

Refreshing - we are now after "split NHVD to two libraries"

[bmegli]

Refreshing - we are now after "fix some of the artifacts in UNHVD"

[bmegli]

This is finished now.

There are:

• infrared textured depth streaming working for some time between RNHVE-UNHVD
• proof-of-concept for colored streaming in RNHVE depth-color branch (see RNHVE#13)

Some loosely related improvements are ongoing.