search

k2-fsa / sherpa-onnx

Speech-to-text, text-to-speech, and speaker recognition using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, Raspberry Pi, RISC-V, x86_64 servers, websocket server/client, C/C++, Python, Kotlin, C#, Go, NodeJS, Java, Swift, Dart, JavaScript, Flutter
https://k2-fsa.github.io/sherpa/onnx/index.html
Apache License 2.0
2.44k stars 280 forks source link

Hotwords encoding for phonemes #981

Open w11wo opened 1 month ago

w11wo commented 1 month ago

Hi. I have a phoneme-based Zipformer model.

Before this PR, I was able to apply hotwords encoding for phoneme sequences, e.g. ɪ z/dʒ ʌ s t/b ɛ s t, following the older implementation of e.g. Chinese character hotwords encoding. But now, I noticed that the Chinese character hotwords encoding have changed from 深 度 学 习 (whitespace between chars) to 深度学习 (no whitespace). And I assume the string parser will simply iterate through the non-whitespace characters in the string sequence.

This, however, breaks my use case, since phoneme sequence with digraphs, e.g. dʒ ʌ s t will be incorrectly split to d ʒ ʌ s t. The issue is that my model's vocab supports digraph and requires the old implementation.

Is it possible to add another modeling unit, other than the currently supported ones (cjk, BPE, cjk+BPE)? Maybe instead of iterating for every non-whitespace character, split by whitespace first? This new modeling unit can hopefully support other use cases similar to mine.

Massive thanks for all the work and help thus far!

csukuangfj commented 1 month ago

@pkufool Could you have a look?

pkufool commented 1 month ago

Emm, May be I we can add a option like do-not-tokenize, I think it should fix your issue.

pkufool commented 1 month ago

For now, I think you can use the older version, v1.9.24 .

w11wo commented 1 month ago

@pkufool Yes, the do-not-tokenize option sounds good.

I can stick to older versions for now, but I wanted to try the customizable per-word hotwords scores, which comes in the latest releases only, hence the need for this new feature.

pkufool commented 1 month ago

@w11wo OK, will make a PR.

w11wo commented 2 weeks ago

Hi @pkufool, is there an update on the PR?

pkufool commented 2 weeks ago

Hi @pkufool, is there an update on the PR?

There is an on-going PR https://github.com/k2-fsa/sherpa-onnx/pull/1039

w11wo commented 2 weeks ago

Thank you so much @pkufool. Looking forward to it getting merged.