Open Arche151 opened 1 month ago
This worked for me:
from diarizers import SegmentationModel
diarize_model = whisperx.DiarizationPipeline(use_auth_token="Your_hftoken",
device = device)
segmentation_model = SegmentationModel().from_pretrained('diarizers-community/speaker-segmentation-fine-tuned-callhome-deu')
fine_tuned_model = segmentation_model.to_pyannote_model()
diarize_model.model._segmentation.model = fine_tuned_model.to(device)
diarize_segments= diarize_model(audio, num_speakers=2)
@Dream-gamer
I will try that out, thanks! :) Heads up, you accidentally shared your HF_Token.
Do you maybe also happen to know, how I can use speaker embeddings, that I extracted via reference speakers, in the diarization pipeline? I asked about that in the Pyannote Github, but didn't get a response unfortunately. https://github.com/pyannote/pyannote-audio/issues/1750
Hey thanks for the heads up lol. Do let me know if you face any error in the above code cause I just tried this today and it worked. As for your follow-up question, I haven't really used speaker embeddings, but yeah I am currently working on improving my diarized transcriptions, so will play around will that as well and get back to you if I found anything,
Thanks @Dream-gamer , works just fine using the tuned model loading!
@Arche151 having the same problem, but using the fine-tuned model doesnt increase the accuracy of diarization that much.. how is it going on your side? any further steps taken to improve it? =)
Glad to know. Yeah, it didn't noticably increase the accuracy. I have been using LLMs like gemini to parse the generated transcript and get corrected transcript. You can use the prompt like "Here is the speaker separated transcript. Some of the words in the transcript are in the wrong speaker labels. Correct them and give corrected transcript:" This has given me much better results for hindi-english transcript. Also, if this didn't improve performace, you can use gemini for diarization itself in google's ai studio: https://aistudio.google.com/app/prompts/new_chat?pli=1 Just upload the audio file, give some context in the prompt, for.ex number of speakers, language in the audio and ask it to generate speaker separated transcript.
thanks! :) yeah I thought also about doing similar stuff for the transcription itself. I am doing stuff offline, so gemini is not a thing ;D the transcription itself is not that big issue for me, its more the diarization itself. if there are questions and some1 else answers quickly with just 1 or 2 words.. it most of the time not really seperated and both the question and answer belong to the same speaker. weirdly sometimes even the question will count to the answer-er :D
I have a WhisperX Python script for transcribing meetings, but the speaker diarization for German is really bad, unfortunately.
After some research I came across the fine-tuned German segmentation model diarizers-community/speaker-segmentation-fine-tuned-callhome-deu but I haven't figured out how to get WhisperX to use it.
Here's my Python script:
I'd greatly appreciate any help!