search

facebookresearch / audio2photoreal

Code and dataset for photorealistic Codec Avatars driven from audio
Other
2.66k stars 250 forks source link

Training the model with different data format #49

Closed ShahZ181 closed 7 months ago

ShahZ181 commented 7 months ago

Hi im relatively new to ml and want to try training an audio to co gesture model that is more suitable for game engines like unreal engine 5

my question is if I input a different type of data to the body VQ VAE model where instead of joint angles i input join positions in 3d space and then input the vector quantized codes from that to the body transformer to essentially train a similar mode that outputs joint position instead of joint angles

I am wondering how practical is this and would it work if so what considerations would i have to make

any help would be greatly appreciated , thanks!

alexanderrichard commented 7 months ago

Hi! You won't be able to use our avatar decoder with that strategy. You would have to curate your own dataset if you want to switch representations.