search

jinseokbae / neural_marionette

Official pytorch implementation of "Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video" (AAAI 2022, Oral)
18 stars 3 forks source link

Is it possible to reconstruct photo-realistic human model from skeleton by using this work? #3

Open felixshing opened 1 year ago

felixshing commented 1 year ago

Hello, your work is very awesome and impressive! Congratulations!

I am currently building a live interactive volumetric video system, which means we want to capture, create, deliver, and render the volumetric content on both sides in real time. In order to reduce the data transmission size, we have an idea that, on the sender side, the sender extracts the skeleton information and transmits it to the receiver side. On the receiver side, it uses the updated skeleton information to reconstruct the body of the sender. Moreover, the sender can transmit the whole 3D model to the receiver from time to time.

In order to make our reconstruction model more photo-realistic, we may want to extract more key points on human's hand and face, for example, by using google's MediaPipe.

I would like to ask do you think we can leverage your work on our project? If so, how to do that? Any suggestion is truly appreciated!

jinseokbae commented 1 year ago

Hello,

I think the answer depends on the scene. If your scene is solely consisting of the articulated target, e.g. human or animal, yes, our system can support compression of volumetric data. However, if the target scenes are varying (i.e. number/categories of scene components are not consistent), keypoints and the corresponding skeletal graph may not be reliable.

Each node of the extracted skeletal graph should be observable during training. So if you can construct a proper dataset, I think our system may allow variation to some extent.

Thank you for considering our work.