On-Chip Self-Calibration

[agrunnet]

We are starting this post to address any questions about the On-Chip Calibration. A white paper will be posted shortly. The features have already been released in the SDK as beta, and are accessible in the RealSense Viewer to test out already now. They are called:

1. "Self-Calibration" - This performs a health-check and tunes a RealSense Stereo Camera (D415 & D435) to improve the depth noise.
2. "Tare" - This function is for correcting the accuracy. You have to point at a flat wall where you know the ground-truth distance to the center point of the image. It will then adjust the calibration parameters accordingly.

Once a calibration has been performance, you can compare the new vs old calibration, and then proceed to burn the new value to memory so it persists after streaming stop or power off. There is also a function to recover "factory calibration", if desired.

The write paper describes all this in detail, and also explains how to invoke the functions programmatically.

Please note that these function are all done on-chip, as the name suggests. This means it should work for any OS and platform, and works over USB2 or USB3.

For clarification, On-Chip Calibration (or OCC in Intel-speak) is separate from what we called "factory calibration" or "technician calibration" or "dynamic calibration". All these techniques still exit.

We hope you enjoy it.

[MartyG-RealSense]

Thanks so much for the information! I have shared a link to this post on the Intel Support site's RealSense forum.

[kafan1986]

Great! After seeing a new android example app's code having two calibration method, I was just going to post a separate question till I saw this post. Will be looking forward to the detailed method.

Also, I am not sure if the android example has functionality to burn in the new calibration currently, so that it maintain state across boots.

[agrunnet]

Yes, we now have different example apps (including Android and Labview). They all allow for permanent updating of calibration.

[agrunnet]

Here are the steps for doing the On-Chip Self-Calibration:

1. Plug in the camera that requires calibration (FW should be 5.12.3+)
2. Launch Intel RealSense Viewer version 2.33+
3. Point to a scene that would normally give good depth.
4. Click the ‘More’ button & choose “On-Chip Calibration”
5. Click the Calibrate button in the pop-up layer
6. Wait a couple of seconds while camera is being calibrated
7. When done, a health-check number of >0.2 would indicate camera calibration was degraded.
8. Compare the result by switching between ‘New’ and ‘Original’ options
9. If you are happy with the result simply click ‘Apply New’ and you are all set!

[agrunnet]

Here are the steps in pictures.

[kafan1986]

@agrunnet Once I click "Apply Now", the new calibration values have been burned in right? and will stay across reboots? Also, any way to restore factory calibration?

[agrunnet]

Correct. Yes there is a command to restore. Actually I don’t think we have that yet in the viewer. Will need to fix that if not.

[liuqh]

when i use the function of On-chip Calibration, i want to know what's the difference between on-chip calibration and dynamic calibration. and how many images we need when app run on-chip calibration? what exactly does ground truth of tare calibration means, and is it the distance between the camera and a wall?

[agrunnet]

We will be publishing a white paper next week. Sorry for delay.

[ghost]

Is there a way to perform Tare calibration when the ground truth value is 500 mm using the Intel RealSense viewer?

[MartyG-RealSense]

The Self-Calibration white paper document is now available!

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras

[alonAtAfimilk]

Hi

I have a question.

Regarding Tare calibration for D435i Where is the exact location of Depth=0 on the Camera ?

There are contradictions within the documents between the text and the drawings, and between the whitepaper of Self Calibration and the webinar.

Can someone here give an official answer?

[MartyG-RealSense]

@alonAtAfimilk On the 400 Series cameras, the origin point is the center of the left IR imager. On the caseless Depth Module Kit boards, where measurements are approximate to the lens position, you should add a minus value to the depth measurements of -0.1 (mm) on D400 / D410 / D415 and -3.2 on D420 / D430 to find the ground truth.

The 'Back of module' values refer to the subtractions that should be made when the back of the Depth Kit module is the reference point instead of the lens.

On the cased USB cameras such as D415 and D435, measurements are from the front glass of the camera. So to find the ground truth on those models, you should add a minus value to the depth measurements of -1.1 on D415 and -4.2 on D435 / D435i.

[alonAtAfimilk]

Hi Marty

thanks for the prompt answer As you can see in your own answer, the text is contradicting the table, and the text is contradicting itself. for example: in the third line of text you write D435, but in the table below it you write D430 in the text itself second line you write "caseless Depth Module", but in the third line you write D435, which is NOT caseless

regarding the last table, where your answer is to add -4.2 mm to the depth measurement: My measurements in lab, using D435i units and matlab, I got results of Z'=0 How did you validate your "-4.2" mm number ?

[MartyG-RealSense]

My apologies, I was writing up the values for the cased cameras first and then added the section on Depth Kit modules, and so mis-wrote the model numbers. I've corrected the text to match the tables.

You only add the minus-value if you want the Ground Truth version of the measurements. Otherwise, for the D435i, assume that the glass on the front of the camera represents Z = 0 distance.

[alonAtAfimilk]

thanx!

[songshan0321]

May I know how long is the time should a Realsense D435 camera be calibrated regularly?

[MartyG-RealSense]

@songshan0321 I would say that the appropriate time for a calibration is when you think that the depth measurements are significantly incorrect or if the camera images have reduced in quality (for example, the depth image looking very broken).

For quick, regular calibrations, the RealSense Viewer has an on-chip self calibration system that can be used to perform a "health check" on the camera calibration. You can find the on-chip calibration in the menu of the 'More' option at the top of the Viewer's options side-panel.

Intel have published a white-paper guide about on-chip calibration at the link below:

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras

[songshan0321]

Thanks for your quick reply. I understand that I should calibrate when depth quality is significantly bad. However, in my case, I have multiple robots equipped with Realsense depth camera running on different places remotely, so that it is not feasible for me to check through one by one. Therefore, I am interested to know what is the suggested period for a regular calibration maintenance.

[songshan0321]

And I found that the health check value is not accurate. It shows "good" on one of my D435 but the depth quality is significantly bad. It turns out much better after a calibration is done.

[MartyG-RealSense]

@songshan0321 Intel offer an OEM version of their Dynamic Calibrator system for $1500 US dollars in the RealSense Store. It is a product aimed at engineering departments and factories. It has an OEM version of the calibration software that has the added features of the ability to calibrate multiple cameras, and to connect remotely through a Linux server. The software can also calibrate both the intrinsics and extrinsics of the camera (the free version of the software only calibrates extrinsics).

https://store.intelrealsense.com/buy-intel-realsense-d400-cameras-calibration-target.html

In regard to a regular calibration period, as long as the camera is not experiencing excessive vibration, very high temperature or physical knocks then once a week would probably be fine to calibrate all of the cameras at the same time if you prefer to do the calibration individually for each camera.

[mli0603]

Hi @MartyG-RealSense,

Thank you very much for the above information. I see in the white paper that The self-calibration algorithm is currently designed to correct for either “intrinsic” or “extrinsic” distortions, but not both at the same time. and The “intrinsic mode” is selected by default and is normally the recommended mode.

• From the first note, it sounds to me that self-calib assumes that one of the two (instrinsics or extrinsics) is correct and calibrate the other one. From the second note, it seems that the accuracy of intrinsics values deteriorates in most of the cases. How will I know which matrix to re-calib? Will I get more accurate result if I calibrate instrinsics first and then calibrate extrinsics (or vice versa)? Is this necessary at all?
• Additionally, will there be more information about how self-calib work? The white-paper is really good already. I really apprecite more information on this to understand when/how to run the self-calib.

Thanks again!

[agrunnet]

@mli0603 Currently we are not able to reliably tell a priori without a target whether it is intrinsic or extrinsic that has deteriorated slightly. The good news is that for small changes it does not matter and they are almost the same. (so no, there is no benefit to running both intrinsic and extrinsic. Choose one).
We are working on some further improvements. We are currently tuning one parameter at a time, using a realtime feedback mechanism where we look at some key performance metrics we have identified that allows us to do this in realtime. That is all I can reveal at this time. The good news is that it can all run on-chip, very fast, and without needing knowledge of the scene.

[mnauf]

I have D415, but it doesn't allow me to choose Intrinsic vs extrinsic when doing on-chip calibration. Please let me know why is that. The whitepaper says it should allow me to choose. I know D435 cameras allow you to choose, but why is it not the case for D415? Also, if I do on-chip calibration, which calibration did it do then? Did it do intrinsic or extrinsic? If intrinsic, does that mean, I don't need to do checkerboard manual calibration?

[MartyG-RealSense]

Hi @mnaf The D415 does not exclude use of intrinsic and extrinsic calibration. The calibration interface may appear incorrectly with options missing like it does in your image if the RealSense UWP driver for Windows has been installed. Do you know whether the UWP driver was installed on your computer, please?

https://www.intel.com/content/www/us/en/download/19236/intel-realsense-d400-series-universal-windows-platform-uwp-driver-for-windows-10.html

[mnauf]

Thanks @MartyG-RealSense for your response. I just installed this and restarted the laptop. It still doesn't allow me to choose between intrinsic and extrinsic. Do you have any other guess what could be wrong?

[MartyG-RealSense]

I was not suggesting to install the UWP driver, as doing so can currently be a cause of the On-Chip calibration interface having missing options. I therefore wondered if you had already installed the UWP driver before you experienced the problem.

If the problem was already occurring before you installed the UWP driver then it may be because of the particular computer that the camera is being used on. In Windows computers affected by this problem, the On-Chip interface can be displayed incorrectly whilst it works correctly when the camera is used on another Windows computer. It has usually been encountered on Windows laptop PCs. A symptom of the problem can be if you check the Stereo Module options in the RealSense Viewer and find that one or both of the Infrared stream options are missing.

[chenhanpolyu]

@MartyG-RealSense Hi, I have one question about the on-chip calibration. I cannot find where to choose the "speed" and intrinsic or extrinsic mode when I click the "on-chip calibration" button. Are they removed in the latest realsense-viewer version?

[MartyG-RealSense]

Hi @chenhanpolyu Intel removed these on-chip interface options from the RealSense Viewer from SDK 2.50.0 onwards in order to simplify the interface. If you require those options then they are still available in SDK 2.49.0.