How to use marepo?

[BJHYZJ]

Thank you for the excellent open-source work. I am a beginner in robotics and would like to use your method for relocalization. My understanding of relocalization is that I capture multiple RGB+depth images in a scene using a RealSense D455 and estimate the poses using a SLAM algorithm to build a point cloud map. My requirement is that after reconstructing an indoor scene, I can then capture a new RGB+depth frame (without pose) using the same camera, and your method will allow me to determine the camera's pose.

Can I use Marepo to achieve this functionality? Do I need to fine-tune the pre-trained checkpoint you provided on the multiple RGB+depth images mentioned above before predicting the pose on the newly captured RGB+depth frame during testing?

Additionally, my computer has a graphics card with only 16GB of VRAM. Is this sufficient for fine-tuning?

Hope to receive your guidance.

[chenusc11]

Hi, thanks for your interest in this work. Yes, you can do that with our code.

The easiest way is the following:

1. You reconstruct the camera poses of your existing RGB images using SLAM/SfM methods.

2. With your existing RGB images and your reconstructed poses, you could train a scene-specific scene coordinate regression network (ACE heads) using train_ace.py. The training process typically only takes a few minutes.

3. To predict poses for the rest of the newly captured images, you can directly attach the pre-trained transformer part of the network (marepo.pt) to predict the poses.

16GB of VRAM is fine.

If you want to improve accuracy further, you could use your card to fine-tune the marepo.pt on a per-scene basis.

[BJHYZJ]

Hi, I just thought about it and want to ask a question. My task mainly involves indoor scenes. I hope to be able to effectively relocate the robot after reconstructing the scene so that it knows where it is, but this position is based on visual prediction. If the position of some smaller objects in the room changes, such as the wrinkles of clothes and the messiness of the sofa, will it have a big impact on my estimated posture?

[chenusc11]

Hi, I would assume small changes like wrinkles of clothes might be ok. The method produces dense scene coordinates and is able to handle some noises. However, I would say it is worth doing experiments to test this, otherwise it is just a hypothesis.