search

leventt / surat

implementation based on "Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion"
159 stars 26 forks source link

training data. #1

Closed JunZhang0629 closed 2 years ago

JunZhang0629 commented 5 years ago

Hi, surat. Thanks for recreating the code available for the community. I am working on reproducing your results as part of research work. You did not provide trainable dataset and eval code. How would i create/acquire precalculatedAutoCorrelations.npy files, if i want to do train/inference on any arbitrary data.

leventt commented 5 years ago

Hi! Unfortunately the data I used were mostly from demos and example files I found part of evaluating some other commercial solutions. Those data wouldn't be mine to provide. I am working on generating my own data which I would provide when ready. I am not working on this actively until another month or two. precalculated auto correlations are auto generated when you specify your audio source to train with on the first run. Autocorrelation implementation I used wasn't fast enough so first time the audio is loaded, it will precalculate and save. You would need matching animation to your audio if you want to create your data. The way code is setup now, it would expect 51 morph targets. You can change that count in code if you have another rig. I am planning to write up some documentation along with the data when I am ready to provide that. Cheers!

leventt commented 4 years ago

I am hopefully going to put this online soon: https://vimeo.com/396595682

It won't be morph targets unfortunately it looks like but it may be more faithful to the original publication this way for other people to validate.

I might try to salvage morph targets with FACS later on.

leventt commented 4 years ago

Hi I would be willing to share my rough data so you can experiment as I do. Let me know what you are planning to use it for and I can ask the voice actor for his permission to share the data with you.

Also check out this post for the overall project called yakamoz.io https://medium.com/@leventtasel/understanding-the-face-8d2989fe93db

bigpo commented 4 years ago

Hi I would be willing to share my rough data so you can experiment as I do. Let me know what you are planning to use it for and I can ask the voice actor for his permission to share the data with you.

Also check out this post for the overall project called yakamoz.io https://medium.com/@leventtasel/understanding-the-face-8d2989fe93db

hi, @leventt , thanks for your nice work! i really appreciate if you can share your dataset so that i can reproduce this work , or let me know how to create the dataset , thanks!

lvZic commented 2 years ago

Hi I would be willing to share my rough data so you can experiment as I do. Let me know what you are planning to use it for and I can ask the voice actor for his permission to share the data with you. Also check out this post for the overall project called yakamoz.io https://medium.com/@leventtasel/understanding-the-face-8d2989fe93db

hi, @leventt , thanks for your nice work! i really appreciate if you can share your dataset so that i can reproduce this work , or let me know how to create the dataset , thanks!

你好 想问下 你后来有找到可以用来训练的数据吗

bigpo commented 2 years ago

Hi I would be willing to share my rough data so you can experiment as I do. Let me know what you are planning to use it for and I can ask the voice actor for his permission to share the data with you. Also check out this post for the overall project called yakamoz.io https://medium.com/@leventtasel/understanding-the-face-8d2989fe93db

hi, @leventt , thanks for your nice work! i really appreciate if you can share your dataset so that i can reproduce this work , or let me know how to create the dataset , thanks!

你好 想问下 你后来有找到可以用来训练的数据吗

sorry, i didn't find useful data

leventt commented 2 years ago

nvidia released their work as a tool where I believe they should have kept the work from original publication. you can use that and generate data with it too: https://www.nvidia.com/en-us/omniverse/apps/audio2face/

NedaZand commented 2 years ago

Hi @leventt I have worked with Nvidia Audio2Face but I dont know how to prepare the dataset Would you please share your training dataset then I can realize how to prepare my own dataset Thanks in advance

leventt commented 2 years ago

I have a custom dataset prepared after recording an actor and I am not going to share it. I am not going to try audio2face to make an example dataset either.