model based tracking for a M12 bolt

[TAO-YT]

I followed the instructions of Tutorial: Markerless generic model-based tracking using a color camera on VISP to track a M12 bolt. But the tracked Z-axis is always unstable when the bolt was still.

https://github.com/user-attachments/assets/17527284-1fbd-49c6-973c-d9071b35a658

I have only changed the CAO model and some parameters of XML file. I saw the videos of some tracking results , they are always precise and stable. I want to know how to make it. Please someone help me.

[TAO-YT]

what I want is to estimae the pose of M12 bolt so that I can control the UR robot to attach it but the pose is not accurate because the Z-axis (blue) is always moving

[s-trinh]

Hello,

Could you try without using depth information? Looks like at this distance, you are outside of the range of the depth sensor.

[TAO-YT]

The camera I use is realsense D435i, and what I want to do is to estimate the CMO which is the pose of the M12 bolt related to the camera by the model based tracking, then I can control the UR by PBVS. So I think it's nesscary to use depth information.What actually troubles me is the Z-axis is always moving like these videoes especially when the camera is very close.

https://github.com/user-attachments/assets/85cb2fa8-6acf-44f9-85a6-b64f6c8e18d6

https://github.com/user-attachments/assets/bb62c40a-c967-47e5-a19b-d2ee38c1e651

The Z-axis is always inaccurate and unstable,whether the bolts are placed separately or installed on components. But I have read the instrctions of Tutorial: Markerless generic model-based tracking using a color camera, and seen the videoes of tracking teabox and lego square. I saw the results of the three axis are very precise and stable even they are moving. I don't know what is the key except for changing the .cao file and xml file. I have been stuck for several months but I think that the model based tracking should be suitable for tracking the bolt like this. So I really don't know how to solve that.

[TAO-YT]

And the distance between the camera and blot is about 20cm-50cm

[s-trinh]

Thanks for the videos.

My opinion on this:

• there is no depth information with respect to the considered objects, see the black regions on the depth image
• you should try to tune the RealSense parameters, see:
  • https://github.com/IntelRealSense/librealsense/tree/master/tools/realsense-viewer
  • https://github.com/IntelRealSense/librealsense/wiki/D400-Series-Visual-Presets
• if possible, try to adjust the lighting to have the better favourable conditions (for instance avoid glare, reflections)
• there are also some parameters for the MBT, can be worth to try to tune certain parameters
• the object to be tracked is very small, I would try to model the whole object, for instance also the "clamp" object

---

Here you can see some contours are being falsely tracked. You can try to adjust the visibility parameter for faces.

In any cases, the object to be tracked is very small. Modeling also the support should improve the tracking.

[TAO-YT]

Thanks for your opinions. Well I know the bolt is too small, but the real thing I want to do is to tighten the bolts by UR16e. So it's necessary to tracking the bolt. But I wonder that maybe the tracking object is too small so that the Z-axis is unstable like that. And I'm sorry that I don't know how to change the HighAccuracyPreset.json file of the Realsense camera in VISP which maybe can improve the depth information, so could you please tell me? This vedio is the process of tracking and visiual servoing the bolt

https://github.com/user-attachments/assets/4a43f5b3-6515-430f-a66f-73c4a55b705d

I have adjust the paraments of xml file for weeks including the visibility parameter for faces. I think this xml file is the best one by testing. luoshuan12(back)xml.txt But the problem is still here, it doesn't help too much. My abilities are really limited and I'm wondering that maybe this MBT can't track small things well? Is there anyways which can track the bolt well? I have seen the last instruction of tracking part in VISP is about tracking with megapose. Do you think this way can track bolt well?

[fspindle]

As mentioned, it seems that your D435i does not provide depth information on your bolt (corresponds to the black areas in the depth image). I can see two possible reasons for this:

• The camera specification states that the minimum depth distance (Min-Z) at maximum resolution is approximately 28cm. The first reason is therefore that the camera is less than 30 cm from the bolt.
• If the distance between your camera and the bolt is greater than 30 cm, this may also be due to the material and/or the grey colour of the bolt which completely absorbs infrared, so that the infrared camera is blind and unable to provide depth information on the grey parts of the object. If you can't overcome one of these 2 reasons, you can remove the use of depth in the tracker.

I don't know of any 3D tracker capable of providing an accurate pose. As already mentioned, the solution in your case is to track a larger object. I would start with the planar support drawn in red in the following image.

The cad model is defined in a frame (I arbitrarily choose the one in the bottom right-hand corner, labelled o). If the tracking is successful, you'll get the cMo transformation. Since you're interested in the top bolt (with the frame named b), you ‘just’ need to add the additional oMb constant transformation. The PBVS only needs to consider the product (cMo * oMb).

If the tracking is more stable, you can then enrich the model with the top bolt and why not the bottom ones.

If that doesn't work any better, you can try Megapose, but you'll need a PC with a graphics card.

[TAO-YT]

Thank you for your reply. Your suggestions are very insightful. Then, I will study issues related to depth information. It seems that my method did not take depth information into consideration, which may be the reason why the z-axis keeps bouncing. As for the method you mentioned of servoing the big part, I have also considered it. But the bolt I want to track is installed on many different devices, which means modeling work needs to be done lots of times, so this idea was rejected. And know I will continue to improve it. If the result is still not good, I will consider using the megapose method.