Potential Encoding

Currently only luminance Y plane of HEVC Main10 P010LE (encoding details) is used.

This means that P010LE chroma (interleaved U/V) channel is still unused.

The U/V plane is:

half the size of Y (H/2 height and same stride as Y)
with P010LE this means 16 bits for each color value of which only 10 MSBs are used

Funnily enough this is exactly the size of D435 infrared greyscale data (which is 8 bit per sample with Y8 format).

If the stride of infrared matches its width it should be possible to:

directly map infrared data to interleaved U/V plane data
at the cost of losing every second value (P010LE 16 bits of which only 10 MSBs are used)
the 2 excess LSB bits should not affect the encoding too much (LSB) and may be ignored

This is somewhat like 4:2:2 chroma sampling (if you consider IR "to be" the U/V chroma.

The decoding

The decoding would use the same mechanism as currently with additional notes:

the left D435 imager is already aligned with depth data (no CPU expansive alignment process needed)
the vertices' colors may be extracted during unprojection ignoring every second value (e.g. left or mean)
and mapped directly to Unity Mesh vertex color data on native side (as currently with point clouds)
- SetColors(NativeArray<T>)
- NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray

This should not affect latency in any significant way but may affect the required bitrate

It is difficult to say anything about the quality of texture encoding without performing some tests

the encoded and ignored 2 LSB bits from every second value may have minor impact
the resulting "4:2:2" sampling may have some impact (every second horizontal value)

bmegli commented 4 years ago

Working proof-of-concept between: