Closed jkrukowski closed 6 months ago
Based on a discussion with @ZachNagengast in https://github.com/argmaxinc/WhisperKit/pull/45 this PR introduces:
SpecialTokens
WhisperTokenizer
Tokenizer
specialTokens
WhisperTokenizerWrapper
TextDecoding
isModelMultilingual
SuppressBlankFilter
TimestampRulesFilter
TestUtils
Based on a discussion with @ZachNagengast in https://github.com/argmaxinc/WhisperKit/pull/45 this PR introduces:
SpecialTokens
structWhisperTokenizer
which inherits fromTokenizer
protocol and hasspecialTokens
propertyWhisperTokenizerWrapper
which implementsWhisperTokenizer
protocol and forwards tokenizer related methodsTextDecoding
has newisModelMultilingual
propertySuppressBlankFilter
andTimestampRulesFilter
depend onSpecialTokens
TestUtils