Closed vinodhian closed 1 year ago
Please run the server under gdb like this:
gdb --args python3 ./asr_server_gpu.py
and collect stacktrace.
Please run the server under gdb like this:
gdb --args python3 ./asr_server_gpu.py
and collect stacktrace.
Please find the backtrace of the segmentation fault below,
INFO:root:Listening on 0.0.0.0:8010
INFO:websockets.server:server listening on 0.0.0.0:8010
INFO:websockets.server:connection open
INFO:root:Connection from ('127.0.0.1', 55712)
INFO:root:Config {'sample_rate': 16000}
INFO:websockets.server:connection closed
Thread 2058 "python37" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffc18994700 (LWP 6796)]
0x00007fffeb03b0fc in kaldi::WordAlignLattice(fst::VectorFst<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, fst::VectorState<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, std::allocator<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> > > > > const&, kaldi::TransitionInformation const&, kaldi::WordBoundaryInfo const&, int, fst::VectorFst<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, fst::VectorState<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, std::allocator<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> > > > >*) ()
from /usr/local/lib/python3.7/site-packages/vosk-0.3.32-py3.7.egg/vosk/libvosk.so
Missing separate debuginfos, use: debuginfo-install nvidia-driver-latest-dkms-cuda-libs-510.47.03-1.el7.x86_64
(gdb) backtrace
#0 0x00007fffeb03b0fc in kaldi::WordAlignLattice(fst::VectorFst<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, fst::VectorState<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, std::allocator<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> > > > > const&, kaldi::TransitionInformation const&, kaldi::WordBoundaryInfo const&, int, fst::VectorFst<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, fst::VectorState<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, std::allocator<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> > > > >*) ()
from /usr/local/lib/python3.7/site-packages/vosk-0.3.32-py3.7.egg/vosk/libvosk.so
#1 0x00007fffeaefc60a in BatchRecognizer::PushLattice(fst::VectorFst<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, fst::VectorState<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> >, std::allocator<fst::ArcTpl<fst::CompactLatticeWeightTpl<fst::LatticeWeightTpl<float>, int> > > > >&, float) ()
from /usr/local/lib/python3.7/site-packages/vosk-0.3.32-py3.7.egg/vosk/libvosk.so
#2 0x00007fffeaefd400 in ?? () from /usr/local/lib/python3.7/site-packages/vosk-0.3.32-py3.7.egg/vosk/libvosk.so
#3 0x00007fffeaf09ba3 in kaldi::cuda_decoder::BatchedThreadedNnet3CudaOnlinePipeline::FinalizeDecoding(int) ()
from /usr/local/lib/python3.7/site-packages/vosk-0.3.32-py3.7.egg/vosk/libvosk.so
#4 0x00007fffeaeff34d in kaldi::cuda_decoder::ThreadPoolLightWorker::Work() ()
from /usr/local/lib/python3.7/site-packages/vosk-0.3.32-py3.7.egg/vosk/libvosk.so
#5 0x00007fffec2fc8e0 in ?? () from /usr/local/lib/python3.7/site-packages/vosk-0.3.32-py3.7.egg/vosk/libvosk.so
#6 0x00007ffff798fea5 in start_thread (arg=0x7ffc18994700) at pthread_create.c:307
#7 0x00007ffff6fafb0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
We have just updated the image. Please try again and let us know if still broken.
Feel free to reopen if needed. It should be fixed now.
System details:
CUDA: 11.6 GCC: gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2) GPU: 16GB CPU: 64GB CPU Cores: 16 Python: 3.7.4 OS: redhat:enterprise_linux:7.9:GA:server Audio sample rate: 16000 Audio size: 10MB Language: English Model Used: vosk‑model‑en‑us‑0.22 websockets==10.2 I compiled and installed kaldi & vosk-api for GPU manually by following all the steps available in https://github.com/alphacep/vosk-api/blob/master/travis/Dockerfile.manylinux file.
Note:
Please find the error details below,
Wav file properties of test16k.wav, which works properly:
Wav file properties of HSC_20210915_123429_3.wav, for which segmentation fault occurs:
Kindly throw some light on this issue.