search

Junyi42 / monst3r

Official Implementation of paper "MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion"
https://monst3r-project.github.io/
855 stars 43 forks source link

Segmenting Video Processing and Manually Setting Camera Intrinsics #22

Open Rosetta-Leong opened 1 month ago

Rosetta-Leong commented 1 month ago

Hi Junyi!

Firstly, thank you for your outstanding work on Monst3R! I encountered an issue due to GPU memory limitations when processing videos with a larger number of frames. Currently, Monst3R can only run successfully with 65 frames, requiring approximately 33GB of VRAM. When attempting to process videos with a higher frame count, I experience memory overflow.

To work around this, I was considering processing the video in segments. However, I am concerned that independently processing different segments could lead to misalignment between the estimated outputs across segments.

To mitigate this, I am wondering if there is a way to manually specify and fix the camera intrinsics (for example, using camera intrinsics obtained from COLMAP) across different segments. This would help ensure consistency and alignment between the outputs for the entire video, regardless of segment size.

Could you provide any guidance on how to implement this or whether this feature is supported?

Thank you for your time, and I appreciate any insights you can offer.

Junyi42 commented 1 month ago

Hi @Rosetta-Leong,

Thanks for your interest in our work!

Yes, you could preset camera intrisics (focal) for the optimization (here is a reference for setting the focals).

However, by connecting different segments of prediction, there could still be issues of error accumulation. I would suggest to implement a window-wise optimization here instead of current implementation that batches all pairs together.

Hope this helps!

Best.