search

mbortolon97 / 6dgs

Code of the paper: 6DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Model
Other
131 stars 11 forks source link

Can this project be used for real-time 6dof spatial positioning? #5

Open GottenZZP opened 3 months ago

GottenZZP commented 3 months ago

Hello, thank you very much for open-sourcing such a great project. I would like to ask you, can this project be used for real-time 6dof spatial positioning? For example, if I already have a scene trained with 3D GS and have the corresponding point cloud files, can I continuously input photos of actions in the above scene and then obtain the corresponding 6dof positioning?

GottenZZP commented 3 months ago

I found that when I input a sequence of continuous images into your prediction algorithm, the predicted 6 degrees of freedom (6dof) pose is generally continuous, but the predicted viewpoint tends to oscillate back and forth. For example, if the content of the continuous images I input is a continuous forward movement, the predicted pose also shows a continuous forward movement overall. However, during the movement, the predicted pose constantly shakes irregularly back and forth in all directions. Is there any way to solve this issue? Thank you very much!

mbortolon97 commented 1 month ago

Hi, Thank you for your interest in the project and for experimenting with it in this way! While real-time 6DoF spatial positioning with 3D Gaussian Splatting is possible in principle, it's not something we specifically tested in this implementation. The oscillations you’re noticing in the predicted pose when feeding continuous images are common in pose estimation tasks. Especially if not specifically developed for it.

One approach to mitigate this issue is to apply a Kalman Filter, which is designed to handle noisy or fluctuating data by smoothing predictions over time. It could help stabilize the predicted viewpoints and reduce the back-and-forth oscillations you’re observing.