search

m-bain / whisperX

WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)
BSD 4-Clause "Original" or "Old" License
10k stars 1.04k forks source link

emotion detection #384

Open MyraBaba opened 11 months ago

MyraBaba commented 11 months ago

Hi

Is there any way to detect emotion, stress of the speaker within the whisper domain ?

Best

code-switched commented 11 months ago

Hi

Is there any way to detect emotion, stress of the speaker within the whisper domain ?

Best

The github repo named Whisper.cpp has a feature called "Confidence color-coding" --print-colors

It can be used to highlight words with high or low confidence and that's the closest I've seen to emotion detection.

sorgfresser commented 11 months ago

Another way would be sentiment analysis on the transcript in a postprocessing step. Maybe have a look at BERT or newer versions for this.

chenrq2005 commented 10 months ago

https://arxiv.org/pdf/2306.12991.pdf

https://huggingface.co/speechbrain/emotion-diarization-wavlm-large