Single rs2::align object can't align streams from two realsense cameras

[AndreiCostinescu]

---

Required Info
Camera Model	D435i & L515
Firmware Version	Latest for both cameras
Operating System & Version	Ubuntu 18
Kernel Version (Linux Only)	5.4.0
Platform	PC
SDK Version	2.50.0
Language	C++
Segment	others

When using multiple Realsense cameras where depth alignment is required (e.g. align all depth streams to their respective color streams), one rs2::align object does not solve the problem correctly. Concrete example:

• Got first frameset of the D435i camera, aligned the depth to the color, displayed the results: :heavy_check_mark:
• Got first frameset of the L515i camera, aligned the depth to the color (with the same rs2::align object), displayed the results: :heavy_check_mark:
• Got second frameset of the D435i camera, aligned the depth to the color (still with the same rs2::align object), displayed the results: :x:
• Got second frameset of the L515 camera, aligned the depth to the color (again with the same rs2::align object), displayed the results: :heavy_check_mark:
• And so on, the depth of the D435i camera was bad (showed parts of the L515 depth result...) but the L515 depth values were ok (they at least corresponded to the L515 camera :smile:)

Why can't a single align object align multiple cameras all to the same stream? I think it makes sense, since the constructor of the rs2::align object takes a stream, and not a pipeline. So the rs2::align object appears to be bound to a stream and not to a pipeline, as opposed to the observed behavior...

Also observed here: https://github.com/IntelRealSense/librealsense/issues/8778

[MartyG-RealSense]

Hi @AndreiCostinescu There was a recent C++ case at https://github.com/IntelRealSense/librealsense/issues/10475 where a RealSense user similarly had problems with multiple camera alignment where it worked with one camera but not two as the second camera's aligned image would have an offset when two cameras were attached.

They solved it and shared their script, which combines the rs-multicam and rs-align-advanced SDK example programs, at https://gist.github.com/lopesnb/c5b12b9270c4268bb4a59bde621fae2c

The details of how they solved it are contained in the discussion at https://github.com/IntelRealSense/librealsense/issues/10475#issuecomment-1123103421

[AndreiCostinescu]

Yes, a solution is to either create a new variable in every pipeline in every frame (as opposed to only once outside the frame-loop) or to create a vector of align objects for each pipeline (how I solved it).

[AndreiCostinescu]

But I think this solution is inefficient because one align object should be all one should need to align different pipelines to the same stream. So I propose adding this behavior to the documentation and/or clarifying that an align object can not align multiple pipelines (one could add the pipeline to which the align object corresponds as the a constructor parameter). Or to implement my proposed behavior: that one "global" align object is all that is needed to align different cameras to the same stream.

[MartyG-RealSense]

The only other multicam alignment case that I know of, where a RealSense team member provides advice, is at https://github.com/IntelRealSense/librealsense/issues/1833

The team member seems to agree with your suggested approach, stating that "rs2::align is not limited to a single camera. You can create multiple pipeline objects like in the multicam example and apply rs2::align on color+depth pairs from each".

[AndreiCostinescu]

It seems I am being misunderstood. This is a feature request to do one of the following things requested below:

But I think this solution is inefficient because one align object should be all one should need to align different pipelines to the same stream. So I propose adding this behavior to the documentation and/or clarifying that an align object can not align multiple pipelines (one could add the pipeline to which the align object corresponds as the a constructor parameter). Or to implement my proposed behavior: that one "global" align object is all that is needed to align different cameras to the same stream.

[MartyG-RealSense]

I understood your meaning. If yourself or another RealSense user wishes to create the feature yourselves and submit it for consideration for inclusion in the RealSense SDK via a pull request then you are welcome to do so.

https://github.com/IntelRealSense/librealsense/pulls

Alternatively, I can label this case as Documentation to act as a documentation change request and keep it open for future consideration by the RealSense team.

[AndreiCostinescu]

Oh, ok; thanks 😁 I got confused because your comments didn't seem to address my core message. For now, I'd also keep this open (with the documentation label please) and when I'll have some spare time, you might get a pull request on this issue 👍

[MartyG-RealSense]

Thanks very much! I have added the Documentation label and will keep this case open.