underwater calibration for d435

[wangcongrobot]

Hi, we made a waterproof housing for the d435 camera so it needs to be recalibrated. The default calibration tool from realsense seems only to calibrate the RGB image, right? But I think we need to calibrate the stereo images to get an accurate depth. What is more, we do not use the IF camera because it does not work underwater.

I want to use like this camera_calibration package to calibrate the stereo camera, but there is on left and right image topic to get. Any suggestions for that? Thanks a lot!

[wangcongrobot]

Hi @agrunnet. Thank you for your detailed comments! It is really helpful!

6. Cylindrical Bending of Z-plane: Due to Snells' law of refraction, you will see a cylindrical bending of the depth plane. Objects at the wider angles will appear closer than objects in the center. Do how do you account for this? In principle, you could do a full recalibration underwater, but that may not be feasible. Plus then you have the opposite issue if you want to use it in air. A recalibration will essentially change the lens distortion function of each camera and create more of a fisheye distortion (yes, it will reduce FOV, but the angles will now pin-cushion out at larger angle). Instead, one could just correct the depth map in post-processing. Basically add an x-axis remapping of the depth. (Please note that the Y-axis is NOT changed).

Actually, we want a full recalibration underwater because we want to use the camera only underwater.

A recalibration will essentially change the lens distortion function of each camera and create more of a fisheye distortion. Yes. One way I used is extracting the raw stereo infrared images and calculating the depth/point cloud by ourselves (or some other post-processing mentioned above). However, these methods need more computing powers which are limited to the embedded application (like AUV). So if I change/recalculate the calibration parameters on the camera device, it will take full advantage of the on-chip power of the camera. Anyway, we won't use it in the air any more.

So can I fully recalibrate the camera underwater based on the current official calibration tools?

[agrunnet]

Can you calibrate underwater with current official OEM calibration tool and target? Yes I believe so....however, since we have not tried it ourselves, I cannot guarantee it will work. But please do try and let us know. If it does not work, we will need to dive deeper...so to speak ;)

[wangcongrobot]

Sorry. For now, I think I cannot afford the OEM calibration tool... But I think it should be good after the full recalibration.

[agrunnet]

Ok. Let us know how it goes. Basically just reproduce what works in air, but just do it under water.

[jnsweets]

Here is another graph that may be helpful in comparing to your measurements. It's equivalent to the previous ones but now plotted as measured Z vs image location relative to the X-axis principal point (Px). You should be able to compare this directly to your measurements assuming 848x480 resolution. If this agrees with what you see, then we can provide the inverse (correction) function.

[MartyG-RealSense]

Hi @wangcongrobot Do you have an update that you can provide, please? Thanks!

[wangcongrobot]

Here is another graph that may be helpful in comparing to your measurements. It's equivalent to the previous ones but now plotted as measured Z vs image location relative to the X-axis principal point (Px). You should be able to compare this directly to your measurements assuming 848x480 resolution.

If this agrees with what you see, then we can provide the inverse (correction) function.

Thank you for your advice! The curve is similar to the real underwater point cloud, like a wall of seabed. I think it's useful If using this to correct the images.

[MartyG-RealSense]

Hi @wangcongrobot Do you have an update that you can provide about this case, please? Thanks!

[wangcongrobot]

Sorry. I don't have the calibration tools now. So I cannot do more test recently. Hope someone who has the official calibration tool can continue this task.

[MartyG-RealSense]

@wangcongrobot Thanks very much for the update.

@agrunnet Do you have any questions for @wangcongrobot please? Thanks!

[agrunnet]

Thanks. Yes we have managed to calibrate our cameras under water. The target and camera are both under water during this process. The end result is perfect performance underwater, with flat depth and full field of view.

We are now exploring whether it is possible for us to calibrate in air and then convert constants to their underwater equivalent values. Ultimately it would be great if we could just have a single command in LibRS that would allow you to turn on underwater operation. That is the goal. We will see what happens. :)

[wangcongrobot]

That's really cool! Do you use the official calibration tool?

[agrunnet]

Yes

[wangcongrobot]

Thank you for your response! So this one, right? Maybe we also should buy it later.

https://store.intelrealsense.com/buy-intel-realsense-d400-cameras-calibration-target.html

[agrunnet]

Yes exactly. Large pool needed sold separately. :)

We actually created a smaller version in-house so we could do it in the lab in a large fish tank, but if you have a pool then this is the one.

[wangcongrobot]

Thank you a lot! I will try to buy this calibration tool and calibrate the camera later.

[MartyG-RealSense]

Hi @wangcongrobot Do you have an update about this case that you can provide, please? Thanks!

[MartyG-RealSense]

Hi @wangcongrobot The outcome of the feature request to add the ability to toggle depth mapping to work underwater was that Intel decided not to progress further with developing the feature. As the feature request has been concluded, I will close this case. Thanks so much for the great discussion in this case!

[ronnie659]

@MartyG-RealSense just catching up on this post as we are also trying to use realsense d455 underwater. Can you confirm my conclusion that if someone wants to use real sense underwater they would need to calibrate it using the official OEM tool as other calibration methods do not work well and there is no feature or way to just convert the in-air calibration matrix to underwater one?

[MartyG-RealSense]

Hi @ronnie659 If the area of water is small like an aquarium tank then the free version of the Dynamic Calibration tool should be sufficient for underwater calibration. If it is a larger underwater area (a swimming pool or bigger) then the OEM version of the tool should be used.

[ronnie659]

@MartyG-RealSense has anyone successfully done this calibration using free version in recent past that you know of?

[MartyG-RealSense]

A RealSense user at https://github.com/IntelRealSense/librealsense/issues/11351 tried the free version of the Dynamic Calibration tool underwater and had partial success with it but failed on the Scale phase of the calibration after the blue squares have been removed. They believed that the problem was that the free version does not calibrate intrinsics, only extrinsics.

If the 2.49.0 version of the RealSense SDK is used (or the 2.49.0 version of the standalone Intel.RealSense.Viewer.exe tool for Windows) then the alternative On-Chip calibration system has the option to calibrate either intrinsics or extrinsics (though not both at the same time), as described at https://github.com/IntelRealSense/librealsense/issues/11351#issuecomment-1402069460

The On-Chip calibration only calibrates depth though and does not calibrate RGB like the Dynamic Calibration tool does.

[jnsweets]

To clarify, for underwater usage of RealSense cameras, the best/recommended approach is to perform a full (OEM) calibration in the water environment. If this is not feasible, then a secondary option would be to apply a custom calibration by modifying specific (intrinsic) parameters. Neither dynamic nor Self-calibration (OCC + Tare) are designed to make these adjustments. Either may lead to some improvement, but they will not result in optimal performance. The preferred method ultimately depends on the operating conditions (range, FOV) and expected or required performance. If you provide some of these details, we can suggest the best approach.

[ronnie659]

@jnsweets @MartyG-RealSense our environment right now is just limited to something like a fish tank of size 1.5m x 0.5m x 1m (length x width x depth). However, ideally, we would want to use the camera underwater in an ROV in the deep sea. Our application is to detect objects and get a 3d point cloud so that a robotic arm can be guided to grasp this object. We recently did some tests by putting the real sense camera with waterproof housing underwater and just tried to get the distance to a single point (see attached). Our test shows that the distance range is from a minimum of 0.3m to a maximum of 0.6m with an error of 9cm at the min. range and 18cm at max range. This is without calibration as we tried to do calibration underwater using the on-chip calibration method provided in the SDK but it failed. Also, the depth is really bad as compared to the air or outside water. Another concern that we might have is that it seems like the depth estimation and distance measurement are highly based on IR cameras and IR projectors using the time-of-flight method, and IR theoretically will not work well underwater. So again, the question is if this camera is really strong enough technically to provide accurate depth, distance, and point cloud underwater. Outside water, it works perfectly well.

https://github.com/IntelRealSense/realsense-ros/assets/98395387/6844f049-f344-4c26-96e8-6891c35ee9bd

[jnsweets]

Thanks for the details. This is very helpful. I will follow up later with more specific suggestions but for now a few comments:

1. I am unable to play the attached video. I'm not sure if there was an issue with uploading it or if the problem is on my end. However, I do have a reasonable idea of what you are seeing.
2. The depth error that you see underwater is entirely understandable and very close to what I would expect. The error is due primarily to the effect of refraction at the water-air interface. The expected magnitude of this error is ~1.33x (~33%) which is very close to what you see and equal to the refractive index of water. This can be resolved with the appropriate calibration modification (which is not addressed by on-chip or dynamic calibration). More comments on this below.
3. The depth being "really bad" is also expected, especially away from the center of the image. This is primarily due to an effective distortion introduced also by the refraction effect. You can probably see this by looking at the L/R "IR" images. They will show a slight but noticeable pincushion type of radial distortion. This also needs to be corrected in order to properly rectify the images for depth calculations. More on this below.
4. The depth is based on the input images from the L/R stereo cameras and not ToF. You are correct that the IR projector light is significantly attenuated in water and therefore will not be as useful as it is in air, but its purpose is to provide additional texture to the scene and not to compute the depth directly. In scenes with sufficient natural texture and ambient lighting, the projector is less effective and in many cases not needed.
5. To summarize, the main parameters that need to be adjusted are distortion (radial coefficients) and focal lengths of each camera (to correct the scaling errors). These are both intrinsic parameters but neither are touched with on-chip self-calibration. Dynamic calibration adjusts only extrinsics between cameras, which are minimally affected by the water. This is why you see little, if any, benefit in running them or they fail to run. Once the FLs and distortion parameters are appropriately adjusted, you may be able to further tune/optimize the performance using either or both self and dynamic cal.
6. I will follow up in the next few days with more specific instructions on how to customize the calibration along the lines described above. In the meantime, let me know if you have any questions about these comments.
7. Do you happen to have an NDA with Intel and a customer support account (Zendesk)? If so, it will be better to use that portal for further communication.

John

[jnsweets]

Following up, here are some basic instructions on performing a custom calibration modification for D455 underwater operation:

1. Using the CustomRW.exe tool, save the existing/initial calibration file (xml format) so it can be loaded back into the camera if needed and edited to create the new calibrations.

2. Edit the Calibration file as follows: a) Scale initial intrinsic focal lengths of both left and right cameras (both x and y components) by 1.33x. b) Replace the radial distortion coefficients (k0, k1, k4) with the following values: 0.28, 0.305, 0.235, respectively. Tangential coefficients (k2, k3) are unchanged. The same change is made to both left and right cameras.

3. Save the new calibration file and write the file into the camera using the CustomRW tool.

4. Reset the camera (either unplug/plug or use the HW reset command).

The radial distortion values are approximate but found to be fairly close on most cameras. You may find that small adjustments to one or more may help to improve the resulting depth image (e.g., mainly improving the flatness of the depth).    For RGB, we have not investigated this in as much detail, but it makes sense that a similar prescription as for depth will work as well.

Let me know how this works, but if done successfully, it should result in significantly better performance. If you need further optimization, you may try running on-chip and Tare calibrations to tune other intrinsic parameters.

John

[ronnie659]

@jnsweets Thank you for providing the details. I am not sure how I can have an NDA with Intel or a customer support account (Zendesk). I will try to make one. We conducted tests after following the calibration steps you mentioned above. The results are quite better. The error ratio has significantly reduced to just 5cm at the max operating range. The operating range of the camera was reported to be 0.52 m (min) to 0.94 m (max), beyond this range the camera failed to calculate the distance. We are using D455 not 435. Tried using on-chip and tare calibration but it did not work underwater. can you please explain how you calculated those distortion coefficient values.? Also, it will be possible to increase the operating range of the camera underwater?

[jnsweets]

@ronnie659 Usually the NDA process is initiated by a salesperson. I will contact one of ours to see whether this is feasible and how to proceed. Can you provide some basic info: name, contact, company, application, prospective purchase volume. These are likely items that will be requested. Once an NDA is in place, setting up the Zendesk account is fairly simple.

Re: operating range, the Min is a function of the camera resolution. 0.52m is the MinZ for HD resolution. You can reduce this by either running at lower resolution (848x480 is suggested and would reduce it to ~0.35cm), or by applying a "disparity shift" which would also reduce the MaxZ.

In principle, there is no max distance so the 0.94m you see must be due to some aspect of the underwater conditions. Do you see gradual degradation in depth quality for increasing Z or does the depth cut off at 0.94m? Depending on the clarity of the water, it could be related to visibility and/or lighting. Sending some sample images would help (this is where using Zendesk account will be useful).

Self-calibration (OCC and Tare) should work underwater if they work normally in air. Can you confirm this? For tare (to correct Z errors), you will need a flat target at a known distance. What type of errors do you get when running in water?

The specific distortion coefficients were not calculated but estimated based on a few OEM calibrations done underwater. D435 and D455 have similar values but they are only approximate. If you detect a distorted depth image against a flat surface, this indicates there is some residual (uncorrected) distortion. Further adjustment of the coefficient values could help to reduce this. But, there is no prescription for this procedure; it's empirical at this point.

Other suggestions: If the camera is in a water-proof enclosure, it is recommended that it be placed as close to the window as possible (minimal air gap). Also, any tilt between the camera and window should be minimized. Ideally, it's placed flush against a flat and thin glass/plastic cover. Any deviations from this will affect the optimal distortion coefficients and they will become Z-dependent (i.e., they are only optimal at a specific distance).

[ronnie659]

@jnsweets

We are using a resolution of 640 x 480. I am not sure about the disparity shift.

I agree that in principle there is no max distance. By max distance, we mean the distance at which there is no depth information. we see the gradual degradation in depth quality for increasing Z. The water is clean and we also did tests under different illumination conditions. Surprisingly, the light did not make much difference in the quality. I guess this might be one benefit of the IR cameras. We have some videos and would be happy to share them if you can manage to make that account.

Self-calibration (OCC and Tare) works find outside water. When used underwater, the error is related to low light conditions and it never goes away.

We have attached the camera to the surface of the waterproof enclosure so there is minimal to no air gap. Is there any specific material that you would propose for the waterproof containers?

[jnsweets]

I am looking into the NDA - it may take some time, but I'll let you know.

Re: MinZ, I misspoke, actually this will be pushed out to longer distances due to the modified calibration parameters, which are necessary to compensate for the reduced FOV and increased distortion in water. This results in a larger MinZ. The value you see is close to the expected for 640x480. The only way to reduce this further, if needed, would be to apply a disparity shift (in Advanced controls/Depth table), run at even lower resolution, or to use a camera with a smaller Baseline (e.g., D435). The potential benefit of the smaller FOV (and larger MinZ) is an improved resolution/accuracy, all else being equal.

The MaxZ is less clear. The fact that the depth degrades vs Z suggests that it is ultimately due to reduced image quality at the larger distances. In order to produce depth, the left/right images need to have sufficient contrast, itself with sufficient spatial resolution. I suspect one or both of these conditions are not being met. This may also explain why self-cal is not working.

There is no specific material required for the window; as long as it's transparent/clear and has low distortion (e.g., is flat), it should be OK.

Once we have an account set up and a support ticket submitted, we will be able to share image data more easily. That should help diagnose further.

I'll let you know the next steps soon.

[jnsweets]

@ronnie659

Sorry for the delay, but I am submitting a request to have a CNDA created and need one more piece of information.

Assuming you are an authorized signer for the NDA, I need your job title. If not, I need the same info for someone who is authorized (name, email, title).

This is normally a few day process but may take longer with holidays - not sure but I will update you as soon as we have it. Once the NDA # is available, we can proceed with the support account.

[jnsweets]

Thank you. The NDA request has been submitted and should be on its way to you for signature. I'll keep you posted on status.