search

alumae / kaldi-gstreamer-server

Real-time full-duplex speech recognition server, based on the Kaldi toolkit and the GStreamer framwork.
BSD 2-Clause "Simplified" License
1.07k stars 342 forks source link

Client stops delivering any output #176

Closed Jendker closed 5 years ago

Jendker commented 5 years ago

I am starting with Kaldi-Gstreamer-Server and trying out some MP3 Ted talks with different models and I am getting to the point (if the recording is longer then ~1 min), that the client stops delivering any output.

Here is the output from the worker, which I am getting on the beginning 

2019-03-06 15:10:23 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Postprocessing done.
2019-03-06 15:10:23 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:10:23 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:10:23 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:10:23 -    INFO:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Got partial result: IT'S SO TO SPEAK OF DINNER PARTIES I WANTED TAKE YOU BACK TO TWO THOUSAND FIFTEEN AND A DINNER PARTY THAT I WENT TO IT PLACES IN TIME THIS WAS WHEN KATE WHEN JENNER WAS FIRST COMING OUT SHEDDING HER IDENTITY ZACHAR DASH IN AND MOVING INTO HER LIFE AS A TRENCH UNDER

but then it gets into:

2019-03-06 15:11:04 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Postprocessing (final=False) result..
2019-03-06 15:11:04 -   DEBUG:       root: 99e23dd3-973f-4333-b116-49547707b3a8: Skipping postprocessing since post-processor already in use
2019-03-06 15:11:04 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:04 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:04 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:04 -    INFO:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Got partial result: UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND
2019-03-06 15:11:04 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:04 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:04 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:04 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:04 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:04 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:04 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:04 -    INFO:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Got final result: UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND
2019-03-06 15:11:04 -    INFO:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Got full final result: {"status": 0, "result": {"final": true, "hypotheses": [{"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 456.134}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR HAD IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 456.105}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 456.027}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR HAND IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 456.023}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER HAD IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 455.997}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER HAND IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 455.915}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR HAND IT IS SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 455.689}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAVEMEN JENNER ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 455.689}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAVEMEN JENNER HAD IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 455.659}, {"transcript": "UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER HAND IT IS SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND", "likelihood": 455.582}]}, "segment-start": 82.63, "segment-length": 20.0, "total-length": 102.63}
2019-03-06 15:11:04 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Before postprocessing: {u'status': 0, u'segment-start': 82.63, u'segment-length': 20.0, u'total-length': 102.63, u'result': {u'hypotheses': [{u'likelihood': 456.134, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 456.105, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR HAD IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 456.027, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 456.023, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR HAND IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 455.997, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER HAD IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 455.915, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER HAND IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 455.689, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME AGENOR HAND IT IS SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 455.689, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAVEMEN JENNER ADDED SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 455.659, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAVEMEN JENNER HAD IT SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}, {u'likelihood': 455.582, u'transcript': u'UNDER FULL FIND WHEN MY HOST GOES ON A RANT ABOUT CAME ENGENDER HAND IT IS SHE IS SAYING THAT IT IS DISRESPECTFUL FOR CONGENER TO FORCE HER TO USE A NEW NAME AND TO USE THESE NEW PRONOUNS HUT SNATCHED UP ONE OF THEM AND LISTENING AND'}], u'final': True}, 'segment': 6, 'id': u'99e23dd3-973f-4333-b116-49547707b3a8'}
2019-03-06 15:11:05 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:05 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:05 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:05 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:05 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:05 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:06 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:06 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:06 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:06 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:06 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:06 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:07 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:07 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:07 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:07 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:07 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:07 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:08 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:08 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:08 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:08 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:08 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:08 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:09 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:09 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:09 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:09 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:09 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:09 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:10 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:10 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:10 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:10 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:10 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:10 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:11 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:11 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:11 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:11 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:11 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:11 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:12 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:12 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:12 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:12 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:12 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:12 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:13 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:13 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:13 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:13 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:13 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:13 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:14 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:14 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:14 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Checking that decoder hasn't been silent for more than 10 seconds
2019-03-06 15:11:14 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:14 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:14 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:15 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:15 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:15 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:15 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:15 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer of size 6144 to pipeline
2019-03-06 15:11:15 -   DEBUG:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Pushing buffer done
2019-03-06 15:11:15 - WARNING:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: More than 10 seconds from last decoder hypothesis update, cancelling
2019-03-06 15:11:15 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Master disconnected before decoder reached EOS?
2019-03-06 15:11:15 -    INFO:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Sending EOS to pipeline in order to cancel processing
2019-03-06 15:11:15 -    INFO:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Cancelled pipeline
2019-03-06 15:11:15 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Waiting for EOS from decoder
2019-03-06 15:11:15 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:15 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:15 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:15 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:16 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:16 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:16 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:16 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:16 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Waiting for EOS from decoder
2019-03-06 15:11:16 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:16 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:16 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:16 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:17 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:17 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:17 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:17 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:17 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Waiting for EOS from decoder
2019-03-06 15:11:17 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:17 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:17 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:17 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:18 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:18 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:18 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:18 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8

and finaly the only output is 

2019-03-06 15:11:44 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:44 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Waiting for EOS from decoder
2019-03-06 15:11:44 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:44 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:45 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:45 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8

or later

2019-03-06 15:11:46 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:46 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 100
2019-03-06 15:11:46 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:46 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 100
2019-03-06 15:11:46 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:46 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 100
2019-03-06 15:11:46 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:46 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 100

With the transition between both:

2019-03-06 15:11:45 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:45 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:45 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:45 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:45 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 8
2019-03-06 15:11:45 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Giving up waiting after 31 tries
2019-03-06 15:11:45 -    INFO:   decoder2: 99e23dd3-973f-4333-b116-49547707b3a8: Resetting decoder state
2019-03-06 15:11:45 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:45 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 100
2019-03-06 15:11:46 -   DEBUG:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2019-03-06 15:11:46 -    INFO:   __main__: 99e23dd3-973f-4333-b116-49547707b3a8: Ignoring data, worker already in state 100

and the master output at the end in this endless loop is:

    INFO 2019-03-06 15:10:50,170 99e23dd3-973f-4333-b116-49547707b3a8: Forwarding client message (<type 'str'>) of length 6144 to worker 
    INFO 2019-03-06 15:10:50,422 99e23dd3-973f-4333-b116-49547707b3a8: Forwarding client message (<type 'str'>) of length 6144 to worker 
    INFO 2019-03-06 15:10:50,675 99e23dd3-973f-4333-b116-49547707b3a8: Forwarding client message (<type 'str'>) of length 6144 to worker 
    INFO 2019-03-06 15:10:50,928 99e23dd3-973f-4333-b116-49547707b3a8: Forwarding client message (<type 'str'>) of length 6144 to worker 
    INFO 2019-03-06 15:10:51,181 99e23dd3-973f-4333-b116-49547707b3a8: Forwarding client message (<type 'str'>) of length 6144 to worker 
    INFO 2019-03-06 15:10:51,434 99e23dd3-973f-4333-b116-49547707b3a8: Forwarding client message (<type 'str'>) of length 6144 to worker

And it goes like this forever and it happens for every configuration file which I tested.

The model configuration file, which I am using in this case is the default librispeech_english_nnet2.yaml.

For the debugging purpose I am starting the elements in the following way:

python kaldigstserver/worker.py -u ws://localhost:8888/worker/ws/speech -c librispeech_english_nnet2.yaml
python kaldigstserver/master_server.py --port=8888
python kaldigstserver/client.py -r <bitrate> <data_file>

Every command executed in separate terminal of course.

I tried also the Docker image to avoid this problem, but there I wasn't able to start the worker for some reason, so I am trying to concentrate on this installation.

I will be thankful for any help! If I should provide anything more to investigate this, I will be happy to do so.

gilamsalem commented 5 years ago

I suggest to start with disabling the post processing (comment out the lines at the yaml file), and report if you see any change.

Jendker commented 5 years ago

That's how easy it was, thank you! This line is every yaml configuration file.

Would be nice if someone could mention this solution in the README file :)

alumae commented 5 years ago

I struggled with this myself. It seems that the postprocessor mechanism doesn't work with Tornado 5. It works with Tornado 4.5.3.

Jendker commented 5 years ago

Is it then worth downgrading to Tornado 4.5.3 and leave postprocessor running?

alumae commented 5 years ago

It depends whether you need the post-processor. Post-processor is required only when you need to rewrite recognition results somehow (it's application specific).

Jendker commented 5 years ago

Thank you for your input!