search

bmegli / realsense-network-hardware-video-encoder

Realsense hardware encoded color/ir H.264 and color/ir/depth HEVC streaming
Mozilla Public License 2.0
23 stars 3 forks source link

D455 infrared RGB streaming #26

Closed bmegli closed 3 years ago

bmegli commented 3 years ago

librealsense 2.39.0 with D4xx firmware v5.12.8.200 adds possibility to stream synthetic RGB from left imager.

The quality compared to RGB is probably bad.\ At the same time this stream is pixel perfect aligned with the depth without any additional computations.

Extending RNHVE to support infrared RGB should be rather straightforward.

bmegli commented 3 years ago

From realsense datasheet Table 4-3. Image Formats (USB 3.1 Gen1) – D455 for left imager supports pixel formats:

In realsense-viewer we also see (working) formats like RGB8, BGR8.

Examing librealsense code it is difficult to say which format is native so let's stick with UYVY from datasheet for now.

bmegli commented 3 years ago

From HVE supported pixel formats issue we see that VAAPI supports uyvy422 pixel format. We should be able to match it directly with librealsense UYVY.

bmegli commented 3 years ago

Current Realsense datasheet (June 2020) is missing information about IR imager sensor for D455.

It is clearly neither OV2740 (D415 which is rolling shutter and not wide) nor OV9282 (D435 which is monochrome).

librealsense#7260 states that D455 uses OV9782.

librealsense#7155 states that D455 uses the same sensor for color and IR.

Omnivision OV9782 page and product guide. Probably the raw data is some Bayer. This is processed by realsense asic/firmware. From our point of view native format is whatever realsense uses internally so that we avoid unnecessary conversion on CPU.

bmegli commented 3 years ago

Ready for merge.

Reasonable results with infrared rgb require:

Otherwise the image is too dark and IR projector pattern too strong.

bmegli commented 3 years ago

The last thing to do is updating wiki how-it-works section.