search

vye16 / shape-of-motion

MIT License
817 stars 59 forks source link

Test in Nvidia-Dynamic Scenes #33

Open kcheng1021 opened 3 months ago

kcheng1021 commented 3 months ago

Hi, Thanks for your wonderful work and the release of the codes! I have tried it in the Nvidia datasets in the monocular setting followed DynIBar. However, the results seem blurring and nosiy. Here is a sample image.

微信图片_20240807143437

I am using the custom dataset to train the nivida dataset without any change. I am curious why the static backgroud is fuzzy, as your demos are pretty clear. Can you help me to find out the reason? Thanks very much!

qianqianwang68 commented 3 months ago

Hi if the background is blurry I think the most likely reason is the camera parameters, as the background modeling is normal 3DGS. One way to verify this is to use the GT camera poses in the Nvidia datasets and see if the issue persists. Indeed we found our camera estimation to be one of the most limiting factors of the whole pipeline. We hope to improve the camera estimation for in-the-wild videos in the future.

CTouch commented 2 months ago

One way to verify this is to use the GT camera poses in the Nvidia datasets and see if the issue persists.

Thanks for your great work!!! @qianqianwang68 But I have the same question. And I also found that the dynamic things(like the balloon) move unsmoothly.

https://github.com/user-attachments/assets/81b84453-1851-4f9c-9b53-44cf061e8b56

I tried simply to replace the traj_c2w in droid_recon/Balloon1.npy with the GT camera pose. But it seems a wrong attempt. Could you please explain the way to use the GT camera poses?

kcheng1021 commented 2 months ago

Hi if the background is blurry I think the most likely reason is the camera parameters, as the background modeling is normal 3DGS. One way to verify this is to use the GT camera poses in the Nvidia datasets and see if the issue persists. Indeed we found our camera estimation to be one of the most limiting factors of the whole pipeline. We hope to improve the camera estimation for in-the-wild videos in the future.

Thanks for your reply and sharing. It is really intutive!

steph1793 commented 3 weeks ago

Hello Thank you very much @qianqianwang68 for your work. I was wondering what convention the camera parameters should be. I ran Colmap on my custom dataset and plugged it into casual_dataset.py. But it still gives blurry backgrounds, indeed as if the camera parameters are bad.

Thanks in advance.