Create a standalone script or notebook to test the AudioLayers and AudioFeatureModel.
Initialize the AudioLayers and AudioFeatureModel with the desired configuration.
Prepare a batch of audio features and latent codes as input.
Pass the latent codes and audio features through the AudioLayers and verify the output shape and values.
Pass the output of AudioLayers through the AudioFeatureModel and verify the output shape and values.
Compare the output with the expected audio embeddings.
With controlnet + SD we can create a lot of images.
If the wave signal can produce anything that resembles the images above -
it should be trivial to plug into animation pipeline.
there's is a way we could do this
sycle through training data
create a corresponding frameset with head position for every frame.
load a wav - train it on these - sound a frame 1 = head pose at frame 1.
https://github.com/johndpope/Emote-hack/tree/main/junk/AudioAttention
Create a standalone script or notebook to test the AudioLayers and AudioFeatureModel. Initialize the AudioLayers and AudioFeatureModel with the desired configuration. Prepare a batch of audio features and latent codes as input. Pass the latent codes and audio features through the AudioLayers and verify the output shape and values. Pass the output of AudioLayers through the AudioFeatureModel and verify the output shape and values. Compare the output with the expected audio embeddings.
With controlnet + SD we can create a lot of images. If the wave signal can produce anything that resembles the images above - it should be trivial to plug into animation pipeline.
there's is a way we could do this sycle through training data create a corresponding frameset with head position for every frame. load a wav - train it on these - sound a frame 1 = head pose at frame 1.
https://github.com/HeliosZhao/Make-A-Protagonist/blob/main/makeaprotagonist/pipelines/pipeline_stable_unclip_controlavideo.py