Open arthur19312 opened 4 months ago
I am having similar doubts.
When extracting text and audio embeddings, I can easily perform cosine similarity to find closely related pairs, and retrieve audio from text inputs and vice-versa.
However, I would like to know if there is a way to decode the embeddings into text. Decoding them into Audio seems manageable using AudioLDM.
@cvillela is there a way to decode CLAP embeddings to Audio using AudioLDM?
I'm new here, I run the method
get_audio_embedding_from_filelist
with modelmusic_audioset_epoch_15_esc_90.14.pt
and get the audio embeddings just likeI approximately know it represent the feature of the input audio somehow, while I don't know how to use it.
At first I try some method to decode it into text so that I can understand it roughly.
SimpleTokenizer
at first withbpe_simple_vocab_16e6
, while it doesn't work.RobertaTokenizerFast
in transformers to decode it, while the error says the decoder just support integers input tokens.So I start to be confused. Could someone tell me what is the audio embedding that I get in format of float? And whether this audio embedding is common to other models? And how should I use it?
(PS: I'm really interested in this work while it seems like I lack some necessary background knowledge, so it would be better if someone could recommend me some relevant materials to get me into the field. Thank you so much ❤)