Support for new zipformer model in sherpa-online-websocket-server

[uni-sagar-raikar]

Hi @csukuangfj ,

We are trying to use the librispeech zipformer2 model on sherpa, but the models from https://huggingface.co/Zengwei/icefall-asr-librispeech-zipformer-2023-05-15/tree/main are not being loaded properly with latest websocket server. Do we need to update something in sherpa? or is it not tested?

Additionally, we can't see zipformer2 as an option here although we can see zipformer2 model definition here.

Thanks in advance Sagar

[csukuangfj]

It is already supported.

Please show the complete command you are using and post any error logs you have.

[uni-sagar-raikar]

Sure,

Command is here:

/workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server --port=6006 --use-gpu=true --decoding-method=greedy_search --nn-model=/mnt/efs/manju/if/repos/streaming_asr/icefall-asr-librispeech-zipformer-2023-05-15/exp/jit_script.pt --tokens=/mnt/efs/manju/if/repos/streaming_asr/icefall-asr-librispeech-zipformer-2023-05-15/data/lang_bpe_500/tokens.txt --doc-root=/workspace/sherpa/sherpa/bin/web

Here is the log:

[F] /workspace/sherpa/sherpa/cpp_api/online-recognizer.cc:276:sherpa::OnlineRecognizer::OnlineRecognizerImpl::OnlineRecognizerImpl(const sherpa::OnlineRecognizerConfig&) 2023-08-09 11:35:38.280 Support only the following models from icefall: conv_emformer_transducer_stateless2 lstm_transducer_stateless2 pruned_stateless_emformer_rnnt2 pruned_transducer_stateless{2,3,4,5} pruned_transducer_stateless7_streaming Given: EncoderModel

[ Stack-Trace: ] /usr/local/lib/python3.8/dist-packages/sherpa/lib/libsherpa_core.so(sherpa::GetStackTrace()+0x4b) [0x7fde3f17dbdb] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(sherpa::Logger::~Logger()+0x62) [0x556355a234b2] /usr/local/lib/python3.8/dist-packages/sherpa/lib/libsherpa_cpp_api.so(sherpa::OnlineRecognizer::OnlineRecognizerImpl::OnlineRecognizerImpl(sherpa::OnlineRecognizerConfig const&)+0x869) [0x7fde3f2040c9] /usr/local/lib/python3.8/dist-packages/sherpa/lib/libsherpa_cpp_api.so(sherpa::OnlineRecognizer::OnlineRecognizer(sherpa::OnlineRecognizerConfig const&)+0x26) [0x7fde3f1ff746] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x1f9f4) [0x556355a2d9f4] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x1fe19) [0x556355a2de19] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x11ddb) [0x556355a1fddb] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fdde72bb083] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x12f8e) [0x556355a20f8e]

terminate called after throwing an instance of 'std::runtime_error' what():
Some bad things happened. Please read the above error messages and stack trace. If you are using Python, the following command may be helpful:

  gdb --args python /path/to/your/code.py

(You can use `gdb` to debug the code. Please consider compiling
a debug version of sherpa.).

If you are unable to fix it, please open an issue at:

  https://github.com/k2-fsa/sherpa/issues/new

Aborted (core dumped)

[csukuangfj]

Please post your complete command.

Also, https://huggingface.co/Zengwei/icefall-asr-librispeech-zipformer-2023-05-15/tree/main the above model is an offline model, i.e., a non-streaming model.

That means, you cannot use it with an online recognizer.

[csukuangfj]

If you don't know whether a given model is a streaming model or not, a very easy way is to check the name of the model. If the name contains streaming, then it is a streaming model, otherwise, it is a non-streaming model.

In your case, the model name is

icefall-asr-librispeech-zipformer-2023-05-15

there is no streaming in the name, so it is a non-streaming model. Super easy.

[uni-manjunath-ke]

HI @csukuangfj , we tried with streaming models available at https://huggingface.co/Zengwei/icefall-asr-librispeech-streaming-zipformer-2023-05-17/tree/main. But, we are getting the same error as below: Please suggest. Thanks

/workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server --port=6006 --use-gpu=true --decoding-method=greedy_search --nn-model=/mnt/efs/manju/if/repos/streaming_asr/icefall-asr-librispeech-streaming-zipformer-2023-05-17/exp/jit_script_chunk_16_left_128.pt --tokens=/mnt/efs/manju/if/repos/streaming_asr/icefall-asr-librispeech-streaming-zipformer-2023-05-17/data/lang_bpe_500/tokens.txt --doc-root=/workspace/sherpa/sherpa/bin/web

[I] /workspace/sherpa/sherpa/csrc/parse-options.cc:495:int sherpa::ParseOptions::Read(int, const char const) 2023-08-09 12:20:45.805 /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server --port=6006 --use-gpu=true --decoding-method=greedy_search --nn-model=/mnt/efs/manju/if/repos/streaming_asr/icefall-asr-librispeech-streaming-zipformer-2023-05-17/exp/jit_script_chunk_16_left_128.pt --tokens=/mnt/efs/manju/if/repos/streaming_asr/icefall-asr-librispeech-streaming-zipformer-2023-05-17/data/lang_bpe_500/tokens.txt --doc-root=/workspace/sherpa/sherpa/bin/web

[F] /workspace/sherpa/sherpa/cpp_api/online-recognizer.cc:276:sherpa::OnlineRecognizer::OnlineRecognizerImpl::OnlineRecognizerImpl(const sherpa::OnlineRecognizerConfig&) 2023-08-09 12:20:47.320 Support only the following models from icefall: conv_emformer_transducer_stateless2 lstm_transducer_stateless2 pruned_stateless_emformer_rnnt2 pruned_transducer_stateless{2,3,4,5} pruned_transducer_stateless7_streaming Given: StreamingEncoderModel

[ Stack-Trace: ] /usr/local/lib/python3.8/dist-packages/sherpa/lib/libsherpa_core.so(sherpa::GetStackTrace()+0x4b) [0x7f789a909bdb] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(sherpa::Logger::~Logger()+0x62) [0x562fce6154b2] /usr/local/lib/python3.8/dist-packages/sherpa/lib/libsherpa_cpp_api.so(sherpa::OnlineRecognizer::OnlineRecognizerImpl::OnlineRecognizerImpl(sherpa::OnlineRecognizerConfig const&)+0x869) [0x7f789a9900c9] /usr/local/lib/python3.8/dist-packages/sherpa/lib/libsherpa_cpp_api.so(sherpa::OnlineRecognizer::OnlineRecognizer(sherpa::OnlineRecognizerConfig const&)+0x26) [0x7f789a98b746] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x1f9f4) [0x562fce61f9f4] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x1fe19) [0x562fce61fe19] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x11ddb) [0x562fce611ddb] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7842a47083] /workspace/sherpa/build/sherpa/bin/sherpa-online-websocket-server(+0x12f8e) [0x562fce612f8e]

terminate called after throwing an instance of 'std::runtime_error' what():
Some bad things happened. Please read the above error messages and stack trace. If you are using Python, the following command may be helpful:

  gdb --args python /path/to/your/code.py

(You can use `gdb` to debug the code. Please consider compiling
a debug version of sherpa.).

If you are unable to fix it, please open an issue at:

  https://github.com/k2-fsa/sherpa/issues/new

Aborted (core dumped)

[csukuangfj]

It works perfectly on my mac.

---

(py38) fangjuns-MacBook-Pro:exp fangjun$ ls -lh jit_*
-rwxr-xr-x  1 fangjun  staff   305B May 22 17:42 jit_pretrained_streaming.sh
-rw-r--r--  1 fangjun  staff   255M May 22 17:44 jit_script_chunk_16_left_128.pt

Please check that your model is the latest one.

[csukuangfj]

From your error log

[F] /workspace/sherpa/sherpa/cpp_api/online-recognizer.cc:276:sherpa::OnlineRecognizer::OnlineRecognizerImpl::OnlineRecognizerImpl(const sherpa::OnlineRecognizerConfig&) 2023-08-09 12:20:47.320 Support only the following models from icefall:
conv_emformer_transducer_stateless2
lstm_transducer_stateless2
pruned_stateless_emformer_rnnt2
pruned_transducer_stateless{2,3,4,5}
pruned_transducer_stateless7_streaming

Please always use the latest version of sherpa when you have an issue.

(The error logs shows that you are using an old version of sherpa, which does not support the latest zipformer.)

[uni-manjunath-ke]

Sure, Thanks. Will try with latest

[csukuangfj]

Sure, Thanks. Will try with latest

By the way, we have updated the installation method for sherpa. Please see https://k2-fsa.github.io/sherpa/sherpa/install/from_wheels.html

It should be much easier than before.

[uni-manjunath-ke]

I have installed using this procedure. I followed https://k2-fsa.github.io/sherpa/sherpa/install/check_your_installation.html#check-sherpa-installation for verifying the installation. The python cmd produces expected output. But, what other cmd's like "sherpa-online --help", does not produce any output.

Should I add some path to $PATH? Thanks

[csukuangfj]

Could you please post some screenshots about your output?

[uni-manjunath-ke]

Thanks. But, I managed to build docker, so no issues. FY reference: $ python3 -c "import sherpa; print(sherpa.file); print(sherpa.version)" /home/manjunath/.local/lib/python3.10/site-packages/sherpa/init.py 1.3.dev20230725+cpu.torch2.0.1 $ sherpa-online --help sherpa-online: command not found

[csukuangfj]

Are you able to find sherpa-online inside

/home/manjunath/.local/bin

?

[uni-manjunath-ke]

Yes. Thanks. It is in /home/manjunath/.local/bin

[csukuangfj]

Could you please run

export PATH=/home/manjunath/.local/bin:$PATH

and run

sherpa-online --help

again?

Normally, you don't need to set the environment variable PATH to include the bin directory of your python virtual environment.