search

zju3dv / EasyVolcap

[SIGGRAPH Asia 2023 (Technical Communications)] EasyVolcap: Accelerating Neural Volumetric Video Research
Other
577 stars 41 forks source link

How to run my own video? #9

Open zhanghongyong123456 opened 5 months ago

zhanghongyong123456 commented 5 months ago

How can I use it on my own data

dendenxu commented 5 months ago

Hi @zhanghongyong123456 . There are several documentations to look into:

  1. We listed some examples to run some baseline methods on a multi-view dataset in the readme here.
  2. In the documentation on static scene, we showed how to run EasyVolcap on a video file. This doc includes the full process of extracting camera poses with COLMAP, convert camera parameters and then train models.
  3. In this documentation, we showed the full process of how to prepare a custom multi-view dataset from multiple videos, including mask extraction, bounding box & visual hull preparation and background preparation.
aleatorydialogue commented 5 months ago

Is possible to render from a custom dataset generated from monocular video? I understand quality may suffer depending on dataset. I am very excited about the potential here, thank you for all your work.

dendenxu commented 5 months ago

Hi @aleatorydialogue , are you referring to monocular static or dynamic dataset?

For static dataset, we have a complete guide on data preparation, model training and inference here.

For monocular dynamic dataset, it indeed sounds like an interesting direction. Thus we're currently working on open-sourcing official support for deformable algorithms (like DyNeRF, NeRFies) as you can see by the deformation field entries in the default volumetric_video_network.py file. The full data preparation and model training pipeline will be open-sourced along the deformable model. We've already finalized the implementation and are currently working out kinks, so expect an update soon!

aleatorydialogue commented 5 months ago

awesome, appreciate the reply. Yes I am hoping to be able to do monocular dynamic datasets. For now, this will obviously limit quality and probably limit the novel views that are possible to infer, but I imagine over time we will have the ability to infer with less and less input. I look forward to the update, in the meantime i will work on your examples to get up to speed. Very promising project, thank you for open sourcing

aleatorydialogue commented 2 months ago

Just wanted to check back in here. Still very focused on dynamic monocular video, still haven't had too much success. Especially with regard to quality