D455/D400 artefacts on depth image when aligning depth to color

[Marenix]

Required Info
Camera Model	D455
Firmware Version	5.13.00.50
Operating System & Version	Windows + Linux (ARM)
Kernel Version (Linux Only)	4.9.253
Platform	PC + Nvidia Jetson
SDK Version	2.50.0
Language	C#, C++
Segment	{Robot/Smartphone/VR/AR/others }

While using D455 cameras, we noticed an issue which we haven't found a previous report about. The issue is visible when aligning depth image to color image. We also noticed it the best on resolution 640x480. The easiest way to reproduce it is to use rs-align example, change the resolution of streams to 640x480, use align to color, and set the slider so you only see depth data.

The issue was first noticed with a specific camera/module where it manifested in a particular way. The depth image on this module would have a black line on part of the image. This line didn't have any basis in IR images (There wasn't anything in the IR images which could create such a line in the depth image, and the line wasn't there when align to depth was used (or when no alignment was used). The following image shows this issue:

The camera was thoroughly tested, along with other D455 cameras we have available. Even though no similar lines were seen on other cameras, other artefacts were noticed:

This second image is of the celling, and when align to color isn't used, there are no black spots all over the image (the bottom black part of the image is due to the light, and we realize that that part will be undetectable (black), but the short black "lines" should not be there).

We also noticed that after leaving any D455 camera streaming for 5-10 minutes, all the artifacts are gone. Both the short black "lines" on all cameras, and the long line on the module we first noticed this problem on. By this point, we suspected that "termal compensation" feature which we found in SDK could be the reason for the disappearance of artefacts, so we tried to disable it to see if we could confirm this assumption. After disabling the feature, the artefacts would not disappear even after leaving the camera streaming for a longer time.

The most important thing we noticed was that this issue seems to be related to a specific implementation of the align algorithm. All the test by this point were done on Windows with either C# or C++ examples, and all of them use sse_align in the background. So, we decided to circumvent the sse_align algorithm and use the plain align algorithm. In that case, we did not notice any artefacts on any modules. The same test was also done on Nvidia Jetson Xavier AGX after we built the package with CUDA, and again, the artefacts were not noticed. This convinced us that the problem lay in the specific implementation of the sse_align algorithm. We are wondering if you've noticed this issue before, and if so, whether there are any plans to solve the bug in sse_align algorithm? From all the information we gathered, we believe the underlying issue is a bug in sse_align algorithm, on which other things have influence.

If you need any additional information when looking into the issue, please let me know and I'll try to provide it if I can.

[MartyG-RealSense]

Hi @Marenix The issue I know of that most resembles your experience is https://github.com/IntelRealSense/librealsense/issues/7799

In that case, the upper image is align_sse alignment and the lower image is non_sse unoptimized alignment. The non-sse aligned image was correct whilst the align_sse image was apparently misaligned. The align_sse image also has much more pronounced black cracks. There is a faint black line on the non_sse image too, though this could be a line that is actually down the middle of the table on the real-world scene.

If it is not practical for you to use CUDA because the application has to be used with non-Jetson computers, it may be worth seeing if aligning using the SDK's GLSL processing blocks system can resolve the issue. This is an acceleration function like CUDA that offloads processing work from the CPU onto the GPU. It has comparable performance to CUDA and is 'vendor neutral', meaning that it can work with any brand of GPU, not just CUDA. An example program for using GLSL with point cloud processing is rs-gl

https://github.com/IntelRealSense/librealsense/tree/master/examples/gl

It can be used with align too just by changing 'rs2::align' references in a C++ script to rs2::gl::align

Basically, an RS2:: instruction can be GLSL-enabled just by adding gl:: after rs2::

---

I have added a Documentation tag to this issue as a reminder to keep it open. RealSense users with similar problems with sse_align are encouraged to report them in a comment here.

[Marenix]

Hi @MartyG-RealSense, sorry for the late response.

I had seen the issue you linked to, but it does not seem like the same issue as the one we have. We haven't noticed any alignment issues. I saw that there were some black cracks in the image using align_sse, but the distribution and amount of them seems different than in our case. Additionally, as the user there didn't complain about that problem, it is unclear whether this is actually a problem, or something that was just captured in that particular image. The user also didn't specify the camera used, and our issue was noticed only with D455 cameras. We tested a few D435 cameras where no issues were noticed.

We have also tried aligning using SDK's GLSL processing blocks which does solve the issue, but implementing it in rs-align seems to slow down the application. The drop in rendered FPS is visible. There is no dedicated GPU on the PC that was used for test, so the drop in FPS is likely caused by that, but as we cannot guarantee that the PC on which the application will be run in the end will have one either. As such, this is a setup which could be what the final user has.

We are also interested in seeing whether other users have experienced the same problem. As I already said, our assumption is that this is a common problem with this configuration for all D455 cameras. It is possible this is also influenced by the specific hardware configuration of D450 module.

I would also like to enquire about a couple of things regarding this issue:

1. Would this be considered a bug in align_sse, as the other ways of aligning images don't seem to have this problem?
2. Are there any plans to look further into this, as this is the default way of using align?
3. Do you have any additional information about thermal compensation feature which seems to solve the issue after some time of streaming? Any whitepapers or articles that could give an insight as to why this seems to solve the problem?

[MartyG-RealSense]

This case will be kept open, and other RealSense users experiencing a similar issue with black lines when using align_sse are very welcome to report it here.

In regard to your questions:

1 and 2. It would be helpful to first collect further reports of this issue on other PCs with D455 cameras as it does not appear to have been reported previously. It may therefore be an issue that only manifests on the particular configuration of the test PC used in this case. Tests with other PC models in alignment operations that use align_sse would confirm or disprove this theory.

As the issue did not occur when using color aligned to depth instead of depth aligned to color, would color-to-depth alignment be a valid solution for you? As the depth and color sensors' field of view (FOV) sizes are approximately equal on the D455 model, the two alignment modes should have a similar aligned image.

3. The most detailed references about the Thermal Compensation feature are at the links below.

https://github.com/IntelRealSense/librealsense/wiki/API-Changes#version-2430

https://github.com/IntelRealSense/librealsense/pull/6350/commits/eeb1a091f0054dbddc55a1f7aafe8aa227049671

[Marenix]

We have been using D455 cameras for some time now on multiple PCs, but we didn't pay much attention to this issue as we didn't see this as an issue. Since we never really looked at the images received constantly, we never noticed the disappearance of the artefacts either. What brought this to our attention was the different manifestation of the issue on one particular module, where we would see a black line when using align to color. We went to analyse the issue further, and we noticed that the line and artefacts were actually just different manifestations of the same bug. Since then, we have tested in on multiple PCs, and can confirm that the issue is visible on all the PCs we tested.

I agree that sensors' FOVs on D455 are similar, and from that point of view there shouldn't be any problems. But, there are some differences. For a trivial example, if you're working on a color image, and you need depth data of something you detected on the image, it makes more sense to align the depth to color, since you don't lose any of the information contained in the color image.

For now, we can work with align to color even with the small black artefacts throughout the image. It hadn't caused us any problems so far, but we cannot be certain it won't negatively affect the end application. As I said, we hadn't even noticed an issue until we started using the module that manifested the problem in a different way (the one with the line). We could not see any difference between the module that produced the line, and all the other modules that had no line, so we are also unsure of why there was a difference. As I said in the previous message, we didn't notice this problem with D435 cameras we tested, but seeing as there seems to be the problem in align_sse and we saw that a module had a different manifestation of the same problem, it is theoretically possible that some D430 and D410 modules could have the same error.

Important note for testing: If you, or anyone else wants to test this, I would suggest using rs-align example. But, I also noticed that the "default" rs-align does not seem to manifest the bug. I just had to enable specific stream profile, and then that seems to have an effect on artefacts appearing. You can use the following for enabling streams: cfg.enable_stream(RS2_STREAM_DEPTH, -1, 640, 480, RS2_FORMAT_Z16, 30); cfg.enable_stream(RS2_STREAM_COLOR, -1, 640, 480, RS2_FORMAT_RGB8, 30);

[MartyG-RealSense]

The code of rs-align enables the depth and color streams but because resolution and FPS are not specified, the default stream profile of the particular camera model being used with the program would be used.

cfg.enable_stream(RS2_STREAM_DEPTH); cfg.enable_stream(RS2_STREAM_COLOR);

For D455 the defaults that would be applied are depth at 848x480 and color at 1280x720, both at 30 FPS.

The SDK's align processing block makes automatic adjustments for differences in resolution and FPS between the two stream types. Possibly it is the align processing block that prevents the black lines from appearing when using the rs-align program with an unchanged stream configuration.

The D455 defaults for depth and color are shown when launching the RealSense Viewer and not changing the stream settings.

[Marenix]

Can you test rs-align with changes I mentioned (setting up same resolution) to see if you can reproduce the issue?

[MartyG-RealSense]

I was not able to reproduce the issue on my own computer.

[Marenix]

1. So, you do not see any such black dots on your PC, even while tilting the camera a bit:

2. Do you have access to just a single D455 camera, or do you have multiple? Since we did notice a different manifestation on a specific module, it is also possible that there are modules which do not exhibit any such issues.

3. Did you run the test as soon as you plugged the camera into the PC? As I mentioned, the issue wasn't visible after thermal compensation kicked in, so if the camera was plugged in for some time before the test, thermal compensation could've kicked in faster than in our tests.

[MartyG-RealSense]

I have a single D455 camera.

I ran a further series of tests using the unchanged rs-align program that uses the default stream configurations for depth and color. I found that black dots were most likely to occur on surfaces close to the camera, whilst surfaces such as walls that were further away did not have them.

This suggests that the black dots might be related to the semi-random infrared dot pattern cast onto the scene by the camera's projector component. The dots did no disappear after a period of time, but they would not if they are produced by the projector since the IR dot pattern is always present when the emitter is enabled.

However, your particular phenomenon is black lines rather than black dots, so your issue may have a different cause unrelated to the dot pattern.

---

If your black line issue does not occur with the default rs-align, and the align processing block automatically adjusts for differences in resolution, then it may be worth simply using the default code instead of setting your own configuration, even if the higher resolutions result in higher data bandwidth consumption. Alternatively, you could try using the default resolutions but reducing the FPS to 15 instead of 30 to use less bandwidth.

cfg.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 15);
cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_RGB8, 15);

[Marenix]

Did you also use the default rs-align example when you were doing the first test before? If so, can you also test it with 640x480@30 and post a picture please?

We have to use the specific resolution and FPS mentioned before.

[Marenix]

Okay, thanks. Let's leave the topic open and see if anyone else comes across the same problem.

[MartyG-RealSense]

Okay, it is no problem to leave it open.

[Marenix]

Hi, we've done some additional debugging, and found the problem in the following function:

When using D435 or D415, either if or else part is ran, depending on the result of if (which is affected by resolution and FOV). The problem with D455 is that "pixels_per_angle_depth.x" and "pixels_per_angle_target.x" are very similar (difference on the second decimal place). In the beginning, the else part is ran, and here we see errors with aligning. After thermal compensation kicks in, FOV slightly changes because of the change in intrinsic parameters, which causes change in "pixels_perangle..." variables, which end up switching the statement, and the part of the code inside if runs. From what we saw, the part of the function inside if is what is actually done in non-sse implementation.

I would like to ask a couple of questions:

1. Is the "if" part of the above function the direct translation of non-sse align algorithm to sse align algorithm?
2. If the above is true, why was the "else" part added?
3. Since it's seen that D455 has a small variation in the variables that are checked, would it make sense to have some "safety net" for it? Like, would it maybe make sense to add a needed minimum difference, so the problem of black pixels would be solved from the moment the camera is started?

[MartyG-RealSense]

It appears that the code in your kindly provided image is from the SDK file sse-align.cpp

https://github.com/IntelRealSense/librealsense/blob/e9f05c55f88f6876633bd59fd1cb3848da64b699/src/proc/sse/sse-align.cpp

I will discuss your questions with my Intel RealSense colleagues. Thanks very much for your patience!

[MartyG-RealSense]

I just wanted to update you that your SSE alignment query is being looked at by my Intel RealSense colleagues.

[Marenix]

Hi, thanks for the update. I also did a few more test, and found that I could reproduce the same issue with D415 camera that we have, although the problem on this camera was more visible when looking at something closer to the camera (around 1 meter opposed to about 2 meters with D455). On D415 camera there is no thermal compensation, so this issue doesn't go away after a few minutes. Hope that helps a bit.

[MartyG-RealSense]

Thanks very much, I will pass along your latest findings.

[MartyG-RealSense]

Hi @Marenix One of my Intel RealSense colleagues was able to replicate the black line effect and discussions about it are continuing. Thanks!

[Marenix]

Hi @MartyG-RealSense, thank you for keeping me in the loop.

[MartyG-RealSense]

Further feedback from my colleague about your black line issue when performing their own tests was: "It seems like the problem occurs only when aligning depth to color and when depth and color have the same resolution. Pointing the camera at the flat wall at a certain angle will create the contour line-ish artifacts. It’s harder to see the artifact if the camera is placed parallel or at a certain angle".

[MartyG-RealSense]

Hi @Marenix Do you have an update about this case that you can provide, please? Thanks!

[Marenix]

Hi @MartyG-RealSense, sorry, I've been working on other things and didn't have time to come back to this.

Generally, I agree with everything your colleague said. I don't think I can offer any further information into the problem except to maybe summarize what I've reported until now.

The problem is connected to different implementation of sse-align from other versions. The part that is different is the if statement in the following picture. While the statement exists in sse-align, there is no such statement in other implementations.

The other thing that is worth mentioning is that this issue seems to manifest in similar way on most of the modules/cameras, though it is more visible on some than on others. We've also had one module which seemed to have literal black lines across the image:

At one point I had 6 such lines on a single image, though unfortunately I did not capture that image. I am also confident that the underlying issue is the same, because after rewriting sse-align implementation so it is the same as other implementations (making sure it always goes into if), the lines were gone. I do not know what causes different manifestation of this problem on this one particular module.

[MartyG-RealSense]

Thanks very much, @Marenix - I will pass your latest feedback to my colleague.

[MartyG-RealSense]

Hi @Marenix An internal Intel report has been created so that the Intel RealSense team can further investigate the issue of the black lines when performing SSE alignment. Thanks again for your patience!

[Marenix]

Hello @MartyG-RealSense, thanks for keeping me in the loop.