D435 Laser emitter turns off causing failed depth tare calibration

Kukanani commented 2 years ago


librealsense	2.50.0 RELEASE
OS	Linux
Name	Intel RealSense D435
Serial Number	135222070722
Firmware Version	05.13.00.50
Advanced Mode	YES
Camera Locked	YES
Usb Type Descriptor	3.2
Product Line	D400
Asic Serial Number	134323023091
Firmware Update Id	134323023091

Issue

During depth tare calibration, the camera's infrared/laser emitter disables, then fails to re-enable, during depth tare calibration. This causes the depth tare calibration to fail as shown in the animated GIF below.

emitter_turns_off_during_depth_tare

This issue happens 100% of the time with both the RealSense Depth Quality Tool (rs-depth-quality) and the Realsense Viewer tool (realsense-viewer).

Hardware setup

Lenovo ThinkPad X1 Extreme
Plugged into wall power
Using USB 3 port
Using OEM cable
Realsense is facing a blank wall
Ambient light is dim

Machine info

$ uname -a
Linux allevato-thinkpad 5.10.0-1057-oem #61-Ubuntu SMP Thu Jan 13 15:06:11 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg --list | grep realsense
ii  librealsense2:amd64                           2.50.0-0~realsense0.6128              amd64        Intel(R) RealSense(tm) Cross Platform API - runtime
ii  librealsense2-dev:amd64                       2.50.0-0~realsense0.6128              amd64        Intel(R) RealSense(tm) Camera Capture API - development files
ii  librealsense2-gl:amd64                        2.50.0-0~realsense0.6128              amd64        Intel(R) RealSense(tm) - GLSL-enabled extensions
ii  librealsense2-net:amd64                       2.50.0-0~realsense0.6128              amd64        Intel(R) RealSense(tm) - Network Bridge extension
ii  librealsense2-udev-rules:amd64                2.49.0-0~realsense0.5307              amd64        Intel(R) RealSense(tm) Camera Capture API - udev rules
ii  librealsense2-utils:amd64                     2.50.0-0~realsense0.6128              amd64        Intel(R) RealSense(tm) Camera Capture API - utils and demos

RealSense info

$ rs-enumerate-devices
Device info: 
    Name                          :     Intel RealSense D435
    Serial Number                 :     135222070722
    Firmware Version              :     05.13.00.50
    Recommended Firmware Version  :     05.13.00.50
    Physical Port                 :     /sys/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/video4linux/video4
    Debug Op Code                 :     15
    Advanced Mode                 :     YES
    Product Id                    :     0B07
    Camera Locked                 :     YES
    Usb Type Descriptor           :     3.2
    Product Line                  :     D400
    Asic Serial Number            :     134323023091

    Firmware Update Id            :     134323023091

...

Logs from the viewer application are attached, one with firmware logs, one without.

rs-viewer-output.txt rs-viewer-output-with-firmware.txt

Kukanani commented 2 years ago

This issue does not happen when using a downgraded version of the firmware and librealsense2.

$ dpkg --list | grep realsense
ii  librealsense2:amd64                           2.49.0-0~realsense0.5307              amd64        Intel(R) RealSense(tm) Cross Platform API - runtime
ii  librealsense2-dev:amd64                       2.49.0-0~realsense0.5307              amd64        Intel(R) RealSense(tm) Camera Capture API - development files
ii  librealsense2-gl:amd64                        2.49.0-0~realsense0.5307              amd64        Intel(R) RealSense(tm) - GLSL-enabled extensions
ii  librealsense2-net:amd64                       2.49.0-0~realsense0.5307              amd64        Intel(R) RealSense(tm) - Network Bridge extension
ii  librealsense2-udev-rules:amd64                2.50.0-0~realsense0.6128              amd64        Intel(R) RealSense(tm) Camera Capture API - udev rules
ii  librealsense2-utils:amd64                     2.49.0-0~realsense0.5307              amd64        Intel(R) RealSense(tm) Camera Capture API - utils and demos

$ rs-enumerate-devices | head -n25
Device info: 
    Name                          :     Intel RealSense D435
    Serial Number                 :     135222070722
    Firmware Version              :     05.12.15.50
    Recommended Firmware Version  :     05.12.15.50
    Physical Port                 :     /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0/video4linux/video4
    Debug Op Code                 :     15
    Advanced Mode                 :     YES
    Product Id                    :     0B07
    Camera Locked                 :     YES
    Usb Type Descriptor           :     3.2
    Product Line                  :     D400
    Asic Serial Number            :     134323023091
    Firmware Update Id            :     134323023091

older_version_no_issue

Kukanani commented 2 years ago

I have also verified that the tare calibration works as expected using the realsense viewer 2.49.0 and firmware version 5.13.00.50. Therefore, I suspect that this issue is linked to the 2.50 apt packages, not the firmware.

MartyG-RealSense commented 2 years ago

Hi @Kukanani I ran a series of Tare calibration tests with the Windows version of the 2.49.0 Viewer, firmware 5.12.15.50 and a D435i camera. I was able to complete the calibration every time if the camera was placed approximately 1 meter from the wall, whilst it consistently failed with the Calibration state did not converged in time error if the camera was placed at a closer distance from the wall such as 0.5 meters or less.

I then tested with the 2.50.0 Viewer and the Calibration state did not converged in time error occurred consistently at 1 meter or any distance above or below that, mirroring your own test results with the 2.50.0 Viewer on Ubuntu. As the issue is repeatable on Windows, this would suggest that the Linux apt packages are not a factor in the problem.

I will seek input from Intel about this issue. Thanks very much for your patience!

Edit: Intel responded that this is a known issue that they are already looking at.

MartyG-RealSense commented 2 years ago

Intel responded further, confirming that is a bug (including the behaviour of the emitter turning off). If you would prefer to use 2.50.0 for Tare then they offered some advice whilst a fix is being worked on by Intel. They said that users can run Tare with a textured target / surface. If the target is not large enough to cover the central Region of Interest (ROI) then they can move closer.

An alternative solution would be to use a separate camera’s projector to provide the needed texture, though this would require the availability of a second camera.

Kukanani commented 2 years ago

Thank you for the quick response and workarounds. A couple of follow up questions now that I have depth taring working as expected:

Does the ROI selected in the Depth Quality tool make a different to the depth tare results, or is it only used to calculate the metrics displayed in the tool?
Does the range at which depth tare calibration is performed (say, 0.5m versus 1.2m) have a significant impact on the quality of the depth tare?
I think that from my reading on the topic and a few tests, if I can supply a single textured plane at about 0.5m that is flat, perpendicular to the camera, and covers 20% ROI, that this will result in the same depth tare quality as a plane is textured, flat, perpendicular, at 1.5m and covers 80% ROI. Is that correct?

MartyG-RealSense commented 2 years ago

I will pass your questions to Intel for a Tare expert to address.

MartyG-RealSense commented 2 years ago

I received a response from an Intel RealSense colleague of mine who is a Tare expert at Intel about your follow-up questions, and some further technical information too.

Before addressing the questions, I will pass on their other advice. On the Tare calibration interface there is a Ground Truth (mm) value, with a default value of '1200'.

Whilst you can accept this default and simply click the 'Calibrate' button, the Tare expert recommends using the Get button to calculate your own Ground Truth value for the scene that your camera is in. They provided the following information.

If you are not updating the GT value and running Tare at arbitrary distances then it’s possible it will fail to converge if actual Z is very different from the reference Z that it’s attempting to adjust to. The tolerable offset depends somewhat on camera and conditions, and may have changed a bit over time in later releases. But, in general, it’s good to be fairly close (within ~20% error) to avoid the issue. It’s really intended to correct small errors.

The default value of 1200 is arbitrary. In normal usage, you should use a value that you know is correct (or as close as possible to true distance). The Tare process will then attempt to modify the calibration so that the measured Z equals the entered GT Z. If these values are very different, it may fail to converge. I suspect this is why it stopped working for distances less than 1000 mm. You can test that by simply changing the GT value to something closer to the true value. It should work.

Here are the responses that the expert provided to your questions.

Does the ROI selected in the Depth Quality tool make a different to the depth tare results, or is it only used to calculate the metrics displayed in the tool?

No. The selected ROI in DQT is independent of any of the self calibration functions (On-chip, Tare, FL cal). That ROI is only used to compute metrics. Having said this, to be most consistent the 20% ROI in DQT will coincide best with Tare and should give the best results.

Does the range at which depth tare calibration is performed (say, 0.5m versus 1.2m) have a significant impact on the quality of the depth tare?

Good question. The answer and best advice depends somewhat on application and conditions. If you are using the camera at a specific distance or narrow range of distances, then it’s best to tare in that range. In general (for usage over the widest operating range), it’s usually best to Tare at the larger distances. This will typically lead to a better result at shorter distances vs result at long distances when taring at short range.

But, there is a limit to this in that noise tends to increase at longer range which can degrade the Tare result. There are usually other practical limitations to taring at very large range. Hopefully this makes sense but you can do a couple of quick checks to determine this empirically for your operating conditions.

I think that from my reading on the topic and a few tests, if I can supply a single textured plane at about 0.5m that is flat, perpendicular to the camera, and covers 20% ROI, that this will result in the same depth tare quality as a plane is textured, flat, perpendicular, at 1.5m and covers 80% ROI. Is that correct?

Theoretically, yes. But see above answer for more specific guidance. The true optimal distance also depends on exactly what is causing the Z error you are trying to correct, which we typically do not know without doing more detailed measurements. One clarification: the ROI requirement for flat target is 20% for all distances, not 80% at 1.5m. But, since you’re 3x further than at 0.5m, it would need to be ~3x larger.

Without knowing more details, my general advice would be to run Tare at the longest distance that is practical for you – where you have a planar surface that covers at least the 20% ROI and sufficient texture. Ideally, a textured target is used and laser is off, but if this is not available, you can run active with a texture-less target but may not be able to go as far depending on lighting conditions.

Kukanani commented 2 years ago

Thank you very much for the detailed comments.

MartyG-RealSense commented 2 years ago

Hi @Kukanani Do you require further assistance with this case, please? Thanks!

Kukanani commented 2 years ago

Yes, since the underlying bug is not fixed, I think it's appropriate for this issue to remain open until it does.

MartyG-RealSense commented 2 years ago

Okay, I added an Enhancement label to the case as a reminder to keep it open until the bug is resolved.

IntelRealSense / librealsense