英文热词(English Hotword)

treya-lin commented 6 months ago

Notice: In order to resolve issues more efficiently, please raise issue following the template. （注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）

❓ Questions and Help

Before asking:

search the issues.
search the docs.

What is your question?

Thanks for your great work. I am using this model (https://www.modelscope.cn/models/iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) to process Chinese audio which contains a lot of English words, and I noticed that while the Chinese hot words worked pretty good (e.g. "英伟达"), the English hotwords never took effect (e.g. "ChatGPT", "prompt"). I am curious if the team has noticed it and had any thought about it? Is it possible to improve it by finetuning? Any advice is greatly appreciated!

Code

from funasr import AutoModel
model = AutoModel(model="iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch", model_revision=None,
                  vad_model=""iic/speech_fsmn_vad_zh-cn-16k-common-pytorch", vad_model_revision=None,
                  punc_model="iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch", punc_model_revision=None,
                  # spk_model="cam++", spk_model_revision="v2.0.2",
                  )
res = model.generate(input=wavpath, batch_size_s=300, hotword = "ChatGPT OpenAI")

What have you tried?

I tried many different hotwords

What's your environment?

OS (e.g., Linux): Linux ubuntu 22.04
FunASR Version (e.g., 1.0.0): 1.0.23
ModelScope Version (e.g., 1.11.0): 1.13.3
PyTorch Version (e.g., 2.0.0): 2.1.0+cu118
How you installed funasr (pip, source): pip
Python version: 3.10
GPU (e.g., V100M32): 4090
CUDA/cuDNN version (e.g., cuda11.7): 11.8
Docker version (e.g., funasr-runtime-sdk-cpu-0.4.1): not using docker
Any other relevant information:

R1ckShi commented 6 months ago

Thanks for using seaco-paraformer model, English hotword has been a noticed issue for a long time, the main problem is that the correction of English hotword always come together with token number changes (because of BPE modeling units), so it is difficult to support English hotword under paraformer backbone(CIF predicts token number at the very beginning), we have make some progress in this issue and the advanced models may be open-sourced one day in the future.

treya-lin commented 6 months ago

Thanks for using seaco-paraformer model, English hotword has been a noticed issue for a long time, the main problem is that the correction of English hotword always come together with token number changes (because of BPE modeling units), so it is difficult to support English hotword under paraformer backbone(CIF predicts token number at the very beginning), we have make some progress in this issue and the advanced models may be open-sourced one day in the future.

Hi I see. Thank you very much for the clarification! I will be looking forward to the advanced models then. :D

modelscope / FunASR