search

abdeladim-s / pywhispercpp

Python bindings for whisper.cpp
https://abdeladim-s.github.io/pywhispercpp/
MIT License
180 stars 26 forks source link

Malformed multi-byte UTF8 characters #39

Closed raivisdejus closed 2 months ago

raivisdejus commented 4 months ago

As noted in https://github.com/ggerganov/whisper.cpp/issues/1798 sometimes a multi byte utf-8 character will be split in multiple tokens, some part in first token, some part is second.

Sample audio where this happens is here https://github.com/chidiwilliams/buzz/blob/main/testdata/whisper-latvian.wav

If we can get to the bytes of the segment "text" we can work around this by gluing two tokens if they have some issue. Current version will fail with UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 0: unexpected end of data on whisper_full_get_segment_text call.

A solution could be to add some function like whisper_full_get_segment_bytes that would return raw bytes of the segment text for manual processing.

UsernamesLame commented 2 months ago

This hasn't been touched in a while, so let me take a peek at the issue. No promises.

UsernamesLame commented 2 months ago

I tested the provided audio file and got this transcription: Money, Ustrouts like a Pestakli, Tapets Uzior, is D.S. by Brausch.

Is this correct? If so it seems like the problem was fixed upstream in whisper.cpp itself, and we can close this.

abdeladim-s commented 2 months ago

@UsernamesLame, I tested it on my end as well without any issue. So I will close this for now!

UsernamesLame commented 2 months ago

@raivisdejus before we close this forever, could you confirm this is what it's intended to spit out?

raivisdejus commented 2 months ago

Tiny model with Latvian (lv) as language should produce something similar to Mani uzstrauts, laikabstākļi, tapēc uz jūru, es diezvajī braukša.

Problematic part is in laikabstākļi where ļ gets returned from whisper.cpp in two segments, first segment has first byte b'\xc4' and the second has second byte b'\xbc'.

abdeladim-s commented 2 months ago

Still couldn't replicate the issue, I used the tiny model with lv language as described

model = Model('tiny')
res = model.transcribe(media="./whisper-latvian.wav", language="lv")
print(res)

Here are the results:

[t0=0, t1=700, text=Mani uzstrauts, laikabstākļi, tapēc uz jūru, es diezvajī braukša.]
UsernamesLame commented 2 months ago

Tiny model with Latvian (lv) as language should produce something similar to Mani uzstrauts, laikabstākļi, tapēc uz jūru, es diezvajī braukša.

Problematic part is in laikabstākļi where ļ gets returned from whisper.cpp in two segments, first segment has first byte b'\xc4' and the second has second byte b'\xbc'.

please re-test with the latest version of pywhispercpp. We're unable to reproduce, so I assume it's fixed in whisper / whisper.cpp?