search

EtienneAb3d / WhisperTimeSync

Synchronize Whisper's timestamps over an existing accurate transcription
131 stars 22 forks source link

Synchronization offset #14

Closed skittlesvampir closed 11 months ago

skittlesvampir commented 11 months ago

Problem description: https://github.com/openai/whisper/discussions/1770#discussioncomment-7526482

I've uploaded the data at: https://ben.ist-toll.xyz/k/whisper-test-files/

EtienneAb3d commented 11 months ago

@skittlesvampir Bug fixed: when the accurate text is also in SRT format, both timestamps were in output 🙃

skittlesvampir commented 11 months ago

Oh my god, now it works!! Thank you so much.

Just two small details:

  1. The Whisper subtitles often quite long compared to the original subtitle that is broken into smaller pieces. (Screenshot 1) Is there a way to get shorter segments?
  2. Sometimes, lot's of consecutive subtitles are combined into a single subtitle (Screenshot 2), because Whisper didn't detect anything for a while. Would it be possible to approximate the subtitles in-between using framerates or something like that?

mpv-shot0002 Screenshot 1

mpv-shot0001 Screenshot 2

EtienneAb3d commented 11 months ago

I think it would be very hard to do a good job when guessing timestamps interpolations. In-between texts could be partially fast or slow and may include some sub-parts without spoken text.

For point 2: the real solution is to improve the Whisper recognition. This can be obtained with WhisperHallu. https://github.com/EtienneAb3d/WhisperHallu

For both points 1 and 2: I'm currently working on a solution using word-level timestamps and some complementary pre-/post-processing around WhisperHallu. I don't plan to release it fully open-source. We can discuss about it if you have a budget.

skittlesvampir commented 11 months ago

I will check WhisperHallu out, it seems cool.

Unfortunately, I don't have a budget, I'm just synchronizing my own shows so I can understand them better.

Anyways, I think the errors are acceptable, so thank you for your work! I wish your business much success in the future!