search

zju3dv / 4K4D

[CVPR 2024] 4K4D: Real-Time 4D View Synthesis at 4K Resolution
https://zju3dv.github.io/4k4d/
Other
1.51k stars 62 forks source link

Can it render poses that have never appeared before? #5

Open CanYueXiBing opened 9 months ago

CanYueXiBing commented 9 months ago

Great research. I have a question for you: 4k4D seems to perform well in free-viewpoint rendering of videos, and the majority of the demo is focused on human bodies. Could this capability potentially extend to another similar task, such as creating animatable human bodies from a video segment, which can be generalized to any pose and undergo free-viewpoint rendering, akin to Animatable NeRF? I'm curious if there are any upcoming demos in this direction. Thank you.

dendenxu commented 9 months ago

Hi @CanYueXiBing, that's an interesting question! On one hand, 4K4D targets real-time general volumetric video & dynamic view synthesis where any type of motion could be present (like objects held by the human (ENeRF-Outdoor), extremely complex clothes (DNA-Rendering), or a puppy (Neural3DV)). Thus we do not explicit model the human body since it would limit 4K4D's setting and scope. On the other hand, I do think what you proposed could be an interesting research direction. I'm assuming animating 4K4D would mean binding the geometry and appearance of the points onto a SMPL model (maybe look into SLRF?) or other similar human rigs. In this setting, the general motion of the human in the data should be easily controlled, but the generalization of complex clothes or other hand-held objects could prove quite challenging so I guess it would take some time (maybe not long) for people to achieve this.

CanYueXiBing commented 9 months ago

I've got it! Thank you very much for your answer!