search

akashmjn / tinydiarize

Minimal extension of OpenAI's Whisper adding speaker diarization with special tokens
MIT License
417 stars 14 forks source link

HF Transformers Weights #15

Open sanchit-gandhi opened 1 year ago

sanchit-gandhi commented 1 year ago

Hey @akashmjn!

Firstly, thank you for creating such an awesome project for the Whisper community! Very excited to try out these checkpoints.

I understand from your README that you used HF Transformers + Datasets to fine-tune the model? It would be super cool to push the resulting HF checkpoints to the Hub, such that all HF users can call your fine-tuned checkpoint using .from_pretrained.

This can be done directly from the model repository (which is a standard Git repo):

git add . && git commit -m "Add weights and training results" && git push

Or, you can do it Pythonically using the huggingface_hub package (installed with Transformers): https://huggingface.co/docs/huggingface_hub/guides/upload

Note that we're already getting feature requests for these checkpoints in Transformers (see issue), so there's demand from the community!

akashmjn commented 1 year ago

Hey @sanchit-gandhi - thanks for reaching out! Ack - will look into this later in the week.

Meanwhile, worth calling out that you'll also need small edits to inference code so that the extra token is not suppressed.

For an example, see https://github.com/akashmjn/tinydiarize/pull/4 https://github.com/akashmjn/tinydiarize/pull/11 (edits on this python repo) or https://github.com/ggerganov/whisper.cpp/pull/1058 (for what was done in cpp)

sanchit-gandhi commented 1 year ago

For the first PR #4 - I think it'll be possible to do something similar in HF transformers without any code changes by simply updating the EOS token to be the speaker turn token (model.generation_config.eos_token=...). For the second PR #11, this might require some more involvement to make sure it works with the transformers batching algorithm, however I think first we can try simply updating the EOS token and running generation to see what the quality is like.

akashmjn commented 10 months ago

Hi @sanchit-gandhi!

Sorry about the silence here. I just posted a project update on https://github.com/akashmjn/tinydiarize/issues/14 that I want to make sure I brought to your attention.

While releasing Transformers weights is conceptually equal to what's already released, given the wider public use TBH I'm playing it safe here for now 😅 . Feel free to DM me (@ akashmjn) or email if you'd like to discuss further.

Great job with Distil-Whisper! (given that, pretty sure you should be able to replicate this too 😉 )

sanchit-gandhi commented 7 months ago

Hey @akashmjn - sorry to hear about the complications with open-sourcing more artefacts. We'd love to facilitate this project and more open-sourcing of checkpoints + code when possible (as @Vaibhavs10 suggested we can provide some compute if required). It's awesome to see what you've built so far here and I hope you can continue contributing!