Closed pajowu closed 1 year ago
Whisper tokens can be invalid utf-8 (grapheme-cluster is split over 2 tokens). This functions allows the user of this library to re-construct the valid string themself.
What does this PR address?
Whisper tokens can be invalid utf-8 (grapheme-cluster is split over 2 tokens). This functions allows the user of this library to re-construct the valid string themself.
Before submitting: