Drexubery / ViewCrafter

Official implementation of "ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis"
Apache License 2.0
811 stars 29 forks source link

Work with other point cloud generation models and diffusion models? #13

Closed LearnedVector closed 1 month ago

LearnedVector commented 1 month ago

Would this method work with other point cloud models like Mast3R and video diffusion models like CogVideoX-5B? Or is this technique unique to only Dust3r and DynamicCrafter models

Drexubery commented 1 month ago

Hi, we utilize off-the-shelf dense stereo methods to construct point clouds in our method; Therefore, it can be replaced with any stronger model like Mast3R. The video diffusion model cannot be altered to CogVideoX-5B because our model is finetuned from DynamiCrafter.

LearnedVector commented 1 month ago

Thanks @Drexubery. but technically i can also replace DynamiCrafter with CogVideoX-5B and finetune it?

Drexubery commented 1 month ago

Thanks @Drexubery. but technically i can also replace DynamiCrafter with CogVideoX-5B and finetune it?

Yes