How to build a new person?

[Ottomachine1]

For now , it's only have 4 pernson id to use I want to know how can I build a person design it by my self ,thanks if anyone give me a response

[alexanderrichard]

The answer is probably that you can't - unless you have access to a 3D full body capture stage.

In order to build a new drivable identity, you need a 3D capture of that person, track it, run 3D reconstruction, fit the rig and compute joint angles as well as unwrapped textures. You would then use these assets to learn a new body decoder as described in https://arxiv.org/abs/2105.10441.

[ForeverAurorak]

Is it possible to use openpose for human key capture and NerF-based human reconstruction to build a new person.

[alexanderrichard]

You'd have to collect a set of training data to learn the motion specifics of that new representation. If you want to use the pretrained models, you'd have to match the representation of our avatars exactly.

[ForeverAurorak]

The answer is probably that you can't - unless you have access to a 3D full body capture stage.

In order to build a new drivable identity, you need a 3D capture of that person, track it, run 3D reconstruction, fit the rig and compute joint angles as well as unwrapped textures. You would then use these assets to learn a new body decoder as described in https://arxiv.org/abs/2105.10441.

Thankes for your reply. I mean is whether the 3D full body capture stage can be replaced by openpose and 3D human body reconstruction, and if so, whether the details of the .npy file in the dataset can be supplemented.

[alexanderrichard]

Ah, I see. No, this will not be possible - or, it will require a new research effort way beyond what this approach is able to do right now. If I understand you correctly, you describe another body decoder that is driven by openpose keypoints. We don't have that pipeline built in this work but of course you can build it yourself. If you don't have a 3D capture stage, you can just render our avatars from different viewpoints and treat the result as ground truth to build your openpose + NeRF method on top.

[ForeverAurorak]

Ok, I'll try. Thank you for this wonderful project.

[voilentKiller0]

Anyone can please give me a detailed explanation of body_pose.npy, face_expression.npy and missing_face_frames.npy in dataset folder.

[alexanderrichard]

The data format is described here: https://github.com/facebookresearch/audio2photoreal?tab=readme-ov-file#dataset

Could you clarify what part exactly is unclear to you? Thanks!