search

k2-fsa / sherpa-onnx

Speech-to-text, text-to-speech, speaker diarization, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, Raspberry Pi, RISC-V, x86_64 servers, websocket server/client, C/C++, Python, Kotlin, C#, Go, NodeJS, Java, Swift, Dart, JavaScript, Flutter, Object Pascal, Lazarus, Rust
https://k2-fsa.github.io/sherpa/onnx/index.html
Apache License 2.0
3.61k stars 421 forks source link

Hotwords encoding for phonemes #981

Open w11wo opened 5 months ago

w11wo commented 5 months ago

Hi. I have a phoneme-based Zipformer model.

Before this PR, I was able to apply hotwords encoding for phoneme sequences, e.g. ɪ z/dʒ ʌ s t/b ɛ s t, following the older implementation of e.g. Chinese character hotwords encoding. But now, I noticed that the Chinese character hotwords encoding have changed from 深 度 学 习 (whitespace between chars) to 深度学习 (no whitespace). And I assume the string parser will simply iterate through the non-whitespace characters in the string sequence.

This, however, breaks my use case, since phoneme sequence with digraphs, e.g. dʒ ʌ s t will be incorrectly split to d ʒ ʌ s t. The issue is that my model's vocab supports digraph and requires the old implementation.

Is it possible to add another modeling unit, other than the currently supported ones (cjk, BPE, cjk+BPE)? Maybe instead of iterating for every non-whitespace character, split by whitespace first? This new modeling unit can hopefully support other use cases similar to mine.

Massive thanks for all the work and help thus far!

csukuangfj commented 5 months ago

@pkufool Could you have a look?

pkufool commented 5 months ago

Emm, May be I we can add a option like do-not-tokenize, I think it should fix your issue.

pkufool commented 5 months ago

For now, I think you can use the older version, v1.9.24 .

w11wo commented 5 months ago

@pkufool Yes, the do-not-tokenize option sounds good.

I can stick to older versions for now, but I wanted to try the customizable per-word hotwords scores, which comes in the latest releases only, hence the need for this new feature.

pkufool commented 5 months ago

@w11wo OK, will make a PR.

w11wo commented 5 months ago

Hi @pkufool, is there an update on the PR?

pkufool commented 4 months ago

Hi @pkufool, is there an update on the PR?

There is an on-going PR https://github.com/k2-fsa/sherpa-onnx/pull/1039

w11wo commented 4 months ago

Thank you so much @pkufool. Looking forward to it getting merged.