today's update seems to not transcribe right

[ClaireCJS]

So I've been working on several different LRC solutions.

The default whisper gets the lyrics to this song:

Here is the code i used:

But when I run your version, I only get a few words:

It seems like whisperx.exe is not getting the words out of the audio file:

But whisper.exe (the official openai one) is:

So, something is broken with whisperx. Any idea what it might be?

[zh-plus]

It looks like whisperx fail to extract all of the transcriptions because of the extra VAD model: https://github.com/m-bain/whisperX/blob/befe2b242eb59dcd7a8a122d127614d5c63d36e9/whisperx/asr.py#L259C19-L259C19

Could you please share the audio file that hasn't been transcribed accurately? Google Drive or OneDrive would be great.

[ClaireCJS]

Thank you for your resopnse :)

I don't understand what you mean by "Extra VAD model" - does that mean there is a file i can delete to fix it? I'd gladly delete and re-download all my models, since it's automatic (?) anyway.

Anyway, I use Mega to share because they give 50G unlike google or onedrive! and I like their spirit! I uploaded the current folder here: https://mega.nz/folder/MdVWkIxQ#jx7d9JtwRnXQEmCw-VTW2w The song is "1_Fences.flac".

I sure hope this helps!

SOME SIDE STORY:

My dream has been AI-generated LRC files. I've spent 20 yrs making 450 or so LRC files but there are about 30,000 that I'd like to generate! Once I get the proof of concept working to the standard I want to, I plan to unleash it on 30,000 songs even if it takes months to do it all! :)

[zh-plus]

The extra model is an integral part of whisperx and cannot fix it by deleting any files. Thanks for your shared song example, I'm trying to figure out the reason.

I am also currently working on adding support for automatically splitting long sentences in word-boundary languages, such as English. This feature will help resolve the problem of excessively long lines mentioned in issue #7.

I will be on vacation this weekend, so both of them may not be fixed promptly.

You can also try https://github.com/johnwmillr/LyricsGenius to download lyrics directly from https://genius.com/, which may provide non-timestamped lyrics with higher text quality than transcribed one. Hope your dream finds you well!

[ClaireCJS]

The extra model is an integral part of whisperx and cannot fix it by deleting any files. Thanks for your shared song example, I'm trying to figure out the reason.

No problem! I'm also working with the whisper-faster.exe team and experienced similar lyrical failures and for their situation, adding the option --vad_filter False really seemed to help a lot!

I am also currently working on adding support for automatically splitting long sentences in word-boundary languages, such as English. This feature will help resolve the problem of excessively long lines mentioned in issue #7.

that sounds amazing!!!!!! whisper-faster.exe seems to do this song fairly well with that option, but it didn't without that option:

I will be on vacation this weekend, so both of them may not be fixed promptly.

I noticed you were super fast which is why I kept talikng to you. Thnks for the warning lol! and for the hard work :)

You can also try https://github.com/johnwmillr/LyricsGenius to download lyrics directly from https://genius.com/, which may provide non-timestamped lyrics with higher text quality than transcribed one. Hope your dream finds you well!

Interesting! I use EvilLyrics ( https://www.evillabs.sk/ ) for over 20 yrs now, and MiniLyrics for 2 yrs now, and both tend to automatically download from lots of sources (and they play together, evillyrics finds stuff better and minilyrics will end up displaying it by sharing the same repository).

[ClaireCJS]

As an update, I'm working on the same problem with whisper-faster, and it, too, was missing most lyrics, until I disabled all VAD functionality. I'm not sure if there's a way to do that with openlrc? But maybe try that, once you're back from vacation :)

[zh-plus]

I'm trying to alleviate the impact of VAD by varying the vad_onset and vad_offset used in whisperX, but it seems there is an existing issue on it: https://github.com/m-bain/whisperX/issues/340 (the VAD parameter are fixed to be offset = onset = 0.5). And the corresponding PR is not merged https://github.com/m-bain/whisperX/pull/357.

Considering that whisperX is no longer actively maintained, I do not anticipate prompt review and merging of this pull request. As an alternative solution, I may consider switching back to faster-whisper which sacrifices some accuracy in word-level time-stamps but benefits from active maintenance.

[zh-plus]

Update: I have released a new version that replaces whisperx with faster-whisper. To install this update, use

pip install openlrc -U

You can adjust the VAD parameters by using the code snippet below:

vad_options = {"threshold": 0.382}
lrcer = LRCer(vad_options=vad_options)
lrcer.run(file, prompter='base_trans', skip_trans=True)

For default parameter values, refer to: https://github.com/zh-plus/Open-Lyrics/blob/5ec44d192f037e7ed12192d627566c1b7966c4b3/openlrc/defaults.py#L25

To understand the meaning of each parameter, visit: https://github.com/guillaumekln/faster-whisper/blob/3b4a6aa1c22d293ddde9f08bdd31fc842086a6ea/faster_whisper/vad.py#L14

Feel free to reduce the threshold value in order to preserve more unfiltered voices.