Open sbelharbi opened 1 month ago
Hi, the functions here were designed for English speech in ABAW3 video data using the Vosk. I am not sure if it works with other videos.
Vosk, when fed with audio or video, would produce the transcription with the time stamps for each spoken word. My code then repair the transcript (adding punctuation, tokenize, add/delete wrongs, feed to bert producing the token-level embedding, get the mapping between the tokens and words, and "average" the embedding of those grouped tokens, and get the exact "word" embedding, then populate it according to the time stamp.
It seems that you are only interested in extracting word embedding from given text? Without considering the time stamp, there are many ways to obtain the sentence-level or word-level embedding can be obtained. One may be explained at here. More should be found by Google or ChatGPT.
hi, the provided code was designed to extract word embedding after extraction of transcripts which is formatted in a specific way. i changed the code to extract the word embedding from a given text transcript.
extract_word_embedding_fn
:there is an issue with
average_merge_embeddings
. after simplifying avereging, another issues shows incompress_single_quote
. this last one seems also to add a new header '["start", "end", "word", "confidence", *np.arange(768)]'.here is a standalone code to do all the above for a simple example.
i currently changed to the storing format of the average embedding to:
np.save('bert_features_example.npy', average_merged_bert_features)
instead of csv as you did sincecompress_single_quote
is not working.any idea why your averaging fails here? thanks