search

hustvl / 4DGaussians

[CVPR 2024] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
https://guanjunwu.github.io/4dgs/
Apache License 2.0
2.23k stars 182 forks source link

Poor Reconstruction Results on Custom Data #168

Open destroy314 opened 4 months ago

destroy314 commented 4 months ago

Thank you for sharing your work. I am attempting to reconstruct a desktop scene using a 6-camera setup as shown in the image below. output

Based on the configuration in multipleview/default.py, I made some modifications: using half the learning rate, 7k coarse iterations, and 50k iterations. As shown in the video, the moving objects appear somewhat blurry in the input views, and some view overfitting can be observed in the novel views.

https://github.com/user-attachments/assets/fd2a6d9d-6c32-4fe3-ba7d-edd7058065aa

https://github.com/user-attachments/assets/df0b0ee9-40d5-410b-b80b-12baebc48133

Do you have any suggestions or hypotheses regarding this issue? I would greatly appreciate your help.

guanjunwu commented 3 months ago

Interesting. In fact, the experiment conducted on 4D-GS is easy and simple, this is a hard and complex/large motion. If you can set a strict multiview, I recommend:

  1. Add more cameras.
  2. try to use multiview-based methods. such as: 4K4D: Real-Time 4D View Synthesis at 4K Resolution; or 3DGStream
  3. wait for me to release a better 4DGS (?
Lizhinwafu commented 2 months ago

I have a video, how to make the dataset?

MikeAiJF commented 3 weeks ago

我有一个视频，如何制作数据集？

你好，你是多视角视频吗

MikeAiJF commented 3 weeks ago

感谢您分享您的作品。我正在尝试使用 6 摄像头设置重建桌面场景，如下图所示。 输出

基于 中的配置multipleview/default.py，我做了一些修改：使用一半的学习率、7k 次粗迭代和 50k 次迭代。如视频中所示，移动物体在输入视图中显得有些模糊，并且在新视图中可以观察到一些视图过度拟合。

摄像头2.mp4 视频.mp4 您对这个问题有什么建议或假设吗？我将非常感谢您的帮助。

你好，请问你的视频怎么处理的，希望得到你的帮助

destroy314 commented 3 weeks ago

@MikeAiJF 简单的用ffmpeg下采样成图片，没有校色、统一曝光和白平衡

destroy314 commented 3 weeks ago

looks like zeroNVS is better in this setting ;) https://arxiv.org/abs/2409.03685

MikeAiJF commented 3 weeks ago

@MikeAiJF简单的用ffmpeg下采样成图片，没有校色、统一曝光和白平衡 多目视频我已经切割成图片了，我不知道接下来怎么进行处理了。才能进行训练。谢谢！

destroy314 commented 2 weeks ago

@MikeAiJF简单的用ffmpeg下采样成图片，没有校色、统一曝光和白平衡 多目视频我已经切割成图片了，我不知道接下来怎么进行处理了。才能进行训练。谢谢！

你可以参考本仓库readme中的For multipleviews scenes部分，或者手动运行multipleviewprogress.sh中的colmap命令（我是这样做的，因为默认参数匹配很容易失败，好像是SiftExtraction.estimate_affine_shape导致的）