dtedesco1
commented
1 year ago AudioLDM
- Get GPT to create a text-to-audio prompt for background sounds/music appropriate for the response & text-to-image prompt
- AudioLDM will generate some background sounds
- Add the generated background sounds during the appropriate response
wav2lip
- Input is audio and generated images
- Match spoken words to lip movements with wav2lip
Video Driven Portrait Animation
- Get GPT to give mood/style/emotion tags for the response
- Options
- Find matching videos from a large video corpus (we can do the tagging ourselves using something like this)
- Find a large corpus that's already tagged
- (6 months from now) use a text-to-video model to generate the driving videos
hi Dan, I have thoughts on creating a video in autovidos, hope this can help you
AudioLDM: which can create background audio for your video Generate speech, sound effects, music and beyond, with text. e.g: background music for sci-fi videos source code: https://github.com/haoheliu/AudioLDM huggingface: https://huggingface.co/spaces/haoheliu/audioldm-text-to-audio-generation
wav2lip: make single static portrait image became to talking video source: https://github.com/Rudrabha/Wav2Lip webiste: https://bhaasha.iiit.ac.in/lipsync/
video driven portrait animation: Can generate talking heads or move body from single static image Source: https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model huggingface: https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model