alumae / kaldi-gstreamer-server

Real-time full-duplex speech recognition server, based on the Kaldi toolkit and the GStreamer framwork.
BSD 2-Clause "Simplified" License
1.07k stars 341 forks source link

master stops receiving messgaes from the worker (which stuck at post processing) #156

Open gilamsalem opened 5 years ago

gilamsalem commented 5 years ago

Hi,

I have an issue while running online decoding using websockets. My setup runs Aspire nnet3 model (http://kaldi-asr.org/models/m4). The issue is that master stop getting partial results from the worker after x responses (arbitrary number).

I tried to dig in the worker log and I saw that the worker itself processing all the partial and also the final result.

From the worker log it seems that a second post processing call is launched before the previous one ends (this is the exact point where the master stop to get messages).

2018-10-17 10:33:47 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:47 -   DEBUG:       root: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing returned: mhm.
2018-10-17 10:33:47 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing done.
2018-10-17 10:33:48 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:48 -   DEBUG:       root: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing returned: wow.
2018-10-17 10:33:48 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing done.
2018-10-17 10:33:49 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:49 -   DEBUG:       root: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing returned: london.
2018-10-17 10:33:49 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing done.
2018-10-17 10:33:49 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:50 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:51 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:51 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:52 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:52 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:53 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:53 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:54 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:54 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..
2018-10-17 10:33:55 -    INFO:   __main__: 9b7c7ad8-fc3a-4974-bc4a-5d5cf6c3c015: Postprocessing (final=False) result..

What can be the problem?

gilamsalem commented 5 years ago

My worker yaml file:

timeout-decoder : 10
use-nnet2: True
decoder:
    # All the properties nested here correspond to the kaldinnet2onlinedecoder GStreamer plugin properties.
    # Use gst-inspect-1.0 ./libgstkaldionline2.so kaldinnet2onlinedecoder to discover the available properties
    use-threaded-decoder:  true
    model : $ROOT/kaldi/egs/aspire/s5/exp/tdnn_7b_chain_online/final.mdl
    word-syms : $ROOT/kaldi/egs/aspire/s5/exp/tdnn_7b_chain_online/graph_pp/words.txt
    fst : $ROOT/kaldi/egs/aspire/s5/exp/tdnn_7b_chain_online/graph_pp/HCLG.fst
    mfcc-config : $ROOT/kaldi/egs/aspire/s5/exp/tdnn_7b_chain_online/conf/mfcc.conf
    ivector-extraction-config : $ROOT/kaldi/egs/aspire/s5/exp/tdnn_7b_chain_online/conf/ivector_extractor.conf
    min-active: 200
    max-active: 7000
    beam: 15.0
    lattice-beam: 6.0
    acoustic-scale: 1.0
    do-endpointing : false
    endpoint-silence-phones : "1:2:3:4:5:6:7:8:9:10"
    traceback-period-in-secs: 0.25
    chunk-length-in-secs: 0.25
    num-nbest: 10
    nnet-mode: 3
#    --frames-per-chunk=20
#    --extra-left-context-initial=0
#    --online=true
#    --frame-subsampling-factor=3
#    --config=$ROOT/kaldi/egs/aspire/s5/exp/tdnn_7b_chain_online/conf/online.conf
out-dir: tmp
use-vad: False
silence-timeout: 10

# Just a sample post-processor that appends "." to the hypothesis
post-processor: perl -npe 'BEGIN {use IO::Handle; STDOUT->autoflush(1);} s/(.*)/\1./;'

logging:
    version : 1
    disable_existing_loggers: False
    formatters:
        simpleFormater:
            format: '%(asctime)s - %(levelname)7s: %(name)10s: %(message)s'
            datefmt: '%Y-%m-%d %H:%M:%S'
    handlers:
        console:
            class: logging.StreamHandler
            formatter: simpleFormater
            level: DEBUG
    root:
        level: DEBUG
        handlers: [console]
alumae commented 5 years ago

Hmm, worker has code (see line 312) that shouldn't allow to start postprocessing before previous postprocessing request has finished. I need to reproduce this myself (thank you for the YAML file).

gilamsalem commented 5 years ago

Currently I overcomed this problem by removing the post-processor attribute.

Kidist-Abraham commented 5 years ago

I am seeing this problem too. Is there a particular version of perl that this used to work for?