search

yanqinJiang / Consistent4D

[ICLR 2024] Official Implementation of Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video
https://consistent4d.github.io/
Apache License 2.0
230 stars 6 forks source link

How to export mesh sequence? #5

Open sam598 opened 4 months ago

sam598 commented 4 months ago

It appears to render video of a mesh, but I am struggling to find where the video is rendered from, and how to export a mesh and color/texture.

yanqinJiang commented 4 months ago

In this work, we adopt dynamic NeRF as the 4D representation. Unfortunately, as far as I know, currently, there is no method to export dynamic NeRF to mesh with animations, because it's challenging to transform the dynamic motion to mesh animation.

But your can export per-frame mesh, referring to the command in Threestudio. Note: Commands of exporting mesh in Threestudio is designed for static NeRF, and you might need to modify the codes slightly to fit our dynamic NeRF.

BTW, Although dynamic NeRF is not easy to export to mesh with animations, dynamic Gaussians, a recent popular 4D representation, could be exported to animated mesh more easily. And DreamGaussian4D, which follows similar pipeline as ours except for replacing dynamic NeRF with dynamic Gaussians, has some special designs for exporting animated mesh.

There might be other better works about Generative Dynamic Gaussians in the near future, please stay tuned : ).

yanqinJiang commented 4 months ago

If you meet any problems when exporting mesh, feel free to ask me for help.

sam598 commented 4 months ago

Thank you for the fast reply, I appreciate it!

Yes, my goal is to export a per frame mesh.

Gaussian splat and temporal information are interesting eventually, but a mesh sequence is fine for now.

In the rendered videos there are three images, the color and depth images on the left and right appear to be NERF renders, but the middle “normals” image looks like a polygonal mesh.

So it seems that these meshes are already being generated somewhere in this project. But I am struggling to find them and figure out how to save them.

yanqinJiang commented 4 months ago

We didn't generate mesh in our code, the normal map was obtained from 3d_normal computed by dynamic NeRF. But you can follow the command in Threestudio to export mesh : ), although you may need to do some modifications to our code. (I guess only slight modifications is needed.)