Example code for reconstruction

[christophschuhmann]

Hi, awesome work!

Can you share some example code for reconstructing audio files from the codes?

Can you also provide some numbers how many Tokens one second of speech or music would take?

Have you tried Training a Codec that is okay for music sound effects and speech?

[christophschuhmann]

I tried to reconstruct an audio file, but it fails: https://colab.research.google.com/drive/1AfRmV1sG0aLdGWM5fWNCnc4vLa6aAB_W?usp=sharing

[hubertsiuzdak]

Hey, your audio shape is [1, T], but the model expects it to be [1, 1, T]. Just add an extra dimension to the input tensor like this, and it should work perfectly:

with torch.inference_mode():
    audio_hat, _, _, _, _ = model(audio.unsqueeze(1))

Can you also provide some numbers how many Tokens one second of speech or music would take?

snac_24khz encodes audio into discrete codes at three temporal levels with rates of 12, 23, and 47 Hz, resulting in roughly 82 tokens per second. You can also find the corresponding numbers for music models: snac_32khz and snac_44khz.

[hubertsiuzdak]

Have you tried Training a Codec that is okay for music sound effects and speech?

Higher bitrate codecs could potentially be universal, since speech can be considered a subset of the wider range of audio signals. However, at the moment, I would not recommend using snac_32khz or snac_44khz for speech, as they were primarily trained on music. I plan to improve them by including more speech samples to balance the training set. I hope this won't affect the music quality, while it should improve speech clarity.

Note that this approach isn't unique. Conventional codecs, such as Opus, under the hood, uses one format optimized for lower-bitrate speech coding and another for general audio.

[christophschuhmann]

Thank you, here is my fixed Colab: https://colab.research.google.com/drive/1AfRmV1sG0aLdGWM5fWNCnc4vLa6aAB_W?usp=sharing

If you would like to train a universal codec, we could provide you with compute and data :)

I am from https://laion.ai

[christophschuhmann]

This is what we are working on: https://docs.google.com/document/d/1nfngl0lv3XAWLQg5z-euYcE6NGGaoH7BdyoujVf7tEc/edit?usp=sharing

:)

[hubertsiuzdak]

Great! I will follow up by email to discuss further