D415/D435 artifacts in one IR image and almost no depth data

[sjmduncan]

| Camera Model |D400| | Firmware Version | 05.12.05.00 (and a few previous versions) | | Operating System & Version | Win10 | | Platform | PC/Raspberry Pi/ NVIDIA Jetson / etc.. | | SDK Version | 2.35.2 | | Language | C++ | | Segment | VR/AR |

Issue Description

Sometimes one of the the cameras produces one normal IR image and one that's mostly patterned noise (see the images below), the resulting depth image is not usable.

After a hardware reset (either unplug the camera, or reset via the RealSense Viewer) it works normally, but the issue persists if you just release the camera and restart acquisition (including maybe setting some configuration parameters during re-initialisation).

This seems to mostly happen when re-initialise the the camera after it's been plugged in for a while (and has been in use and then released a few times). I'm using the cameras for full resolution RGB+Depth normally (captured some IR images for the first time today and noticed IR image problem), at frame rates ranging from 6FPS to 30FPS, it's happened on a few different desktops/laptops and NUCs. The camera is always connected over USB3.1 or better (both the port and the USB cable).

[MartyG-RealSense]

Hi @sjmduncan The description of how the problem may occur after the camera has been released a few times reminds me of an issue that was listed as fixed in the release notes for firmware 5.12.5: "IR Frames stop arriving after several camera start / stop". For the benefit of RealSense team members reading this, the bug's code was DSO-14337.

Would it be possible to try updating your camera firmware to 5.12.6 to see if it provides any improvement, please?

[sjmduncan]

@MartyG-RealSense cameras are updated now.

It'll take a day or two of normal use to see if issue is fixed since I can't seem to reliably reproduce the problem.

[MartyG-RealSense]

Okay, thanks very much for the update. I look forward to your test results when you have them. Good luck!

[MartyG-RealSense]

Hi @sjmduncan Do you have an update for us please? Thanks!

[sjmduncan]

@MartyG-RealSense still having the issue, it's possible that sometimes I'm not unloading or releasing the cameras properly - is this likely to cause issues like this?

Otherwise my initialization roughly looks like this:

rs2::config cfg;
cfg.enable_stream(RS2_STREAM_DEPTH, -1, 1280, 720, RS2_FORMAT_Z16, 30);
cfg.enable_stream(RS2_STREAM_COLOR, -1, 1280, 720, RS2_FORMAT_BGR8, 30);
cfg.enable_device(serial);

pipe_profile = pipe->start(cfg);

auto dev = pipe->get_active_profile().get_device();
auto sensors = dev.query_sensors();

// Depth stream params
sensors[ds_idx].set_option(RS2_OPTION_VISUAL_PRESET, RS2_RS400_VISUAL_PRESET_HIGH_ACCURACY);
sensors[ds_idx].set_option(RS2_OPTION_LASER_POWER, laser);
sensors[ds_idx].set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 0);
sensors[ds_idx].set_option(RS2_OPTION_EXPOSURE, dExp);
sensors[ds_idx].set_option(RS2_OPTION_GAIN, dGain);

// Color sensor params
sensors[cs_idx].set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 0);
sensors[cs_idx].set_option(RS2_OPTION_EXPOSURE, cExp);
sensors[cs_idx].set_option(RS2_OPTION_GAIN, cGain);
sensors[cs_idx].set_option(RS2_OPTION_ENABLE_AUTO_WHITE_BALANCE, 0);
sensors[cs_idx].set_option(RS2_OPTION_WHITE_BALANCE, cWB);

[MartyG-RealSense]

I re-read your case from the start to try to get new insights and noted that you appear to have messy areas on the bottom of your IR images that resemble depth image detail. That is not normal for a depth image and implies that something in the image processing is getting messed up. If it does not happen all the time then it is probably not related to camera calibration.

May I confirm that the program starts running normally and then the apparently bad IR problem appears on its own at some point, and that is the reason why you perform a reset of the camera?

And in the RealSense Viewer, may I confirm that you are not enabling and disabling the depth and IR streams repeatedly during streaming before the problem first appears?

I would add that if you are only interested in the depth stream, you do not actually need to have any of the IR streams enabled and can just have the depth stream alone enabled. The depth frame of the camera is based on the left and right IR frames captured by the camera, but is not produced by the IR streams.

In that sense, the cfg instructions in your program script are correct. You define a depth stream but did not create a cfg definition for infrared streams too. But if you just want depth and color then that would be correct, since you do not need infrared streams in order to produce the depth stream.

[sjmduncan]

Thanks for the quick responses!

... messy areas on the bottom of your IR images that resemble depth image detail

The issue is the patterned noise (regular vertical dotted lines) which cover the whole first IR image but are absent in the second image.

The bright patches at the bottom (and bottom-left corner of the second IR image) are part of the mounting hardware for that particular camera, which has since been replaced (sorry if those example images are a little misleading).

May I confirm that the program starts running normally and then the apparently bad IR problem appears on its own at some point, and that is the reason why you perform a reset of the camera?

Hardware reset is only performed after I notice this issue, and it is performed to correct the issue (which it does successfully).

It seems to happen on initialisation - I can't say that with absolute confidence but I only notice the issue after initialisation and recorded data so far either has only bad IR/depth data or only good IR/depth data with no sequences that start with good data that turns bad later in the sequence.

And in the RealSense Viewer, may I confirm that you are not enabling and disabling the depth and IR streams repeatedly during streaming before the problem first appears?

Yes, I only ever use the viewer after the issue appears, and then all I do is enable both IR streams and the depth stream and start acquisition to confirm that it's the same problem.

I would add that if you are only interested in the depth stream, you do not actually need to have any of the IR streams enabled and can just have the depth stream alone enabled. The depth frame of the camera is based on the left and right IR frames captured by the camera, but is not produced by the IR streams.

The normal mode of operation is to only enable the depth and colour streams. IR streams are only enabled sometimes for debugging (and only at reduced frame rates, see below), but this problem is present in both cases.

cfg.enable_stream(RS2_STREAM_DEPTH, -1, 1280, 720, RS2_FORMAT_Z16, 15);
cfg.enable_stream(RS2_STREAM_COLOR, -1, 1280, 720, RS2_FORMAT_BGR8, 15);
cfg.enable_stream(RS2_STREAM_INFRARED,1, 1280, 720, RS2_FORMAT_Y8, 15);
cfg.enable_stream(RS2_STREAM_INFRARED, 2, 1280, 720, RS2_FORMAT_Y8, 15);

[puzzlepaint]

I have seen this issue as well. In my case, I use D415 cameras on a PC with Manjaro Linux, which should rule out a Windows-only problem. Aside from that, since it only happened very rarely for me so far, I do not think that I can contribute other useful information at this point unfortunately. The problem otherwise seemed to occur in exactly the same way as described by @sjmduncan : it seems that one of the infrared cameras returns corrupt data, which corrupts the depth image (even if the infrared stream itself is not used at all by the program that uses the camera).

[MartyG-RealSense]

@sjmduncan Thanks very much for the clarification about the mountings at the bottom of the IR images.

Regarding the 'vertical dotted lines' pattern: it is hard to see on the chair and the carpet because of their roughened texture. It is easier to see a difference on what looks like the legs of a tripod though, with the legs in the first image considerably blurred compared to their sharpness in the second IR image.

Edit: I considered whether the camera might have become mis-calibrated. I see that the problem is corrected after a camera reset though, which would rule out mis-calibration as a cause.

If you notice the problem particularly at initialization and you are using auto-exposure, it is good practice to skip the first several frames of the capture because the auto-exposure is still settling down during this period and so there may be errant results in those first frames. You should not need to skip frames if manual exposure values are used. The subject is discussed in the link below:

https://github.com/IntelRealSense/librealsense/issues/2269

Having said that, if you were using auto-exposure and you were capturing a longer session than just the initial frames, I would expect the image to correct itself after those initial frames and not require a hardware reset to correct if the auto-exposure settle period was the cause.

[MartyG-RealSense]

Hi @puzzlepaint Thanks so much for your input! A cause of the depth image getting occasionally broken may be if the camera observes an environmental effect such as a flash of sunlight directly in its view, causing the IR sensor to become saturated. In that case, a reset of the camera can clear the problem if it does not correct itself.

[sjmduncan]

@puzzlepaint yeah some kind of IR stream corruption seems to be the cause. Did you get similar pattern-noise to the D415 image I linked below?

@MartyG-RealSense I don't use auto-exposure and it's not a transient issue - once it is triggered the depth stream is useless until a hardware rest or a power cycle. I'm also indoors and have blackout blinds to avoid IR interference from sunlight so the brightest lights here are the ceiling lights (seen in the linked images below) and the cameras are never pointed at light sources during testing anyway.

Anyway, I have some more examples of the bad IR data

• D415, both IR streams are bad and the depth stream is mostly empty
• Two D435i cameras, projectors turned off, the top-middle IR stream has the same line-pattern and the corresponding depth stream is also mostly empty especially compared to second depth stream

[MartyG-RealSense]

@sjmduncan During my latest research into your case, I found a case from a year ago where somebody had similar stripes across their IR image that only appeared sometimes and were corrected by a reset.

https://support.intelrealsense.com/hc/en-us/community/posts/360033503254-Infrared-camera-image-is-damaged

[puzzlepaint]

@sjmduncan Here are the screenshots that I made of realsense-viewer when it happened to me. The pattern on the D435 looks the same as the ones already posted, for the D415 I am not sure whether it is the same.

D435:

D415 1st IR stream viewed as RGB:

D415 2nd IR stream:

[MartyG-RealSense]

@sjmduncan @puzzlepaint I will refer the images above to someone within Intel to get their opinion on what the cause of the problem may be.

[MartyG-RealSense]

@sjmduncan @puzzlepaint I am filing an official bug report about this problem so that the RealSense team can investigate it. This case will be left open whilst the issue is being investigated.

[MartyG-RealSense]

Intel have provided an update on the issue in this case. They recommend to always start the RGB camera first in the pipeline and then the problem should be resolved.

[puzzlepaint]

Thanks for the update, I will try this once I get back to working on the camera-related part of my project.

I hope that this is also documented somewhere such that people who are not aware of this GitHub ticket know about it, and/or the underlying issue will be fixed.

[sjmduncan]

Intel have provided an update on the issue in this case. They recommend to always start the RGB camera first in the pipeline and then the problem should be resolved.

Thanks for the follow-up, I was indeed starting depth before color. I'll test the solution as soon as I can.

[MartyG-RealSense]

Thanks very much @sjmduncan for the update - good luck with your tests!

[MartyG-RealSense]

Hi @sjmduncan Did you have the opportunity to test Intel's suggestion to always start the RGB camera first in the pipeline, please? Thanks!

[MartyG-RealSense]

Hi @sjmduncan Can you provide an update please regarding whether you had the opportunity to test Intel's suggestion to always start the RGB camera first in the pipeline, please? Thanks!

[sjmduncan]

I still have the issue. I'm following the examples from here. Is this what you mean by starting the RGB camera before depth?

rs2::config cfg;
cfg.enable_stream(RS2_STREAM_COLOR, -1, 1280, 720, RS2_FORMAT_BGR8, 30);
cfg.enable_stream(RS2_STREAM_DEPTH, -1, 1280, 720, RS2_FORMAT_Z16, 30);

rs2::pipeline pipe;
auto pipe_profile = pipe.start(cfg);

[MartyG-RealSense]

Yes, the advice is related to the order in which the streams are enabled. You can test this in the RealSense Viewer by turning the red activation icons to blue (on) in a particuar order. Enable the RGB stream first by turning its icon from red to blue, and then the depth stream after that. Do you still have artifacts in the Viewer if you do so?

[MartyG-RealSense]

Hi @sjmduncan Do you have an update that you can provide about this case please? Does the advice in the comment above work for you? Thanks!

[sjmduncan]

Hi @MartyG-RealSense, the problem was still present with the corrected order of initialization. As a workaround I have been triggering a hardware reset every time I start the software since that did work every time.

I've updated to the most recent version of the SDK (2.44) and the recommended camera firmware (05.12.12.100) and will do some tests without the hardware reset over the next few days.

[MartyG-RealSense]

Okay, thanks very much @sjmduncan for the update - I look forward to your report of test results. Good luck!

[MartyG-RealSense]

Hi @sjmduncan Do you have an update about this case that you can provide, please? Thanks!

[sjmduncan]

I haven't been able to reproduce the issue with the firmware/SDK versions of my previous comment, so this seems to be fixed.

Thanks for your assistance.

[MartyG-RealSense]

You are very welcome - it is excellent news about the reliability. Thanks for the update!

[MartyG-RealSense]

@sjmduncan I note that this case is associated with an active Intel bug report and so should be kept open whilst that bug report is active. So I will re-open the case. You do not need to do anything else though. Thanks!