search

jcsilva / docker-kaldi-gstreamer-server

Dockerfile for kaldi-gstreamer-server.
BSD 2-Clause "Simplified" License
289 stars 139 forks source link

recognition occurs, no result returned to client, then worker/master hangs #72

Closed rpdrewes closed 4 years ago

rpdrewes commented 4 years ago

Thanks for your work, jcsilva, and your excellent instructions in the README.md. I have the recognizer working, sort of: the recognition is performed correctly but a result is not returned to the client, and if I kill the client and retry, there seems to be something wrong with the server and subsequent attempts fail until the server is restarted.

My test client is the client.py referenced in the README.md. I have the suggested ws4py version (0.3.2). I have also tried Konele client with similar results.

I am running both worker and master in the same docker, as shown in the Practical Example portion of the README.md. I just pulled the docker image yesterday. I am running on Ubuntu 19.10. I get similar behavior when I run worker and master in separate docker instances. I get similar behavior when my client is on the same physical machine as the worker/master and also when the client is on another machine on the local network.

In detail, when I invoke client.py with an audio test file, the end of the worker.log looks like this:

2020-01-27 18:53:37 -    INFO:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Got final result: here is a more complicated audio sample to try to recognize
2020-01-27 18:53:37 -    INFO:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Got full final result: {"status": 0, "result": {"final": true, "hypotheses": [{"transcript": "here is a more complicated audio sample to try to recognize", "likelihood": 312.754}]}, "segment-start": 0.0, "segment-length": 9.36, "total-length": 9.36}
2020-01-27 18:53:37 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Before postprocessing: {u'status': 0, u'segment-start': 0.0, u'segment-length': 9.36, u'total-length': 9.36, u'result': {u'hypotheses': [{u'likelihood': 312.754, u'transcript': u'here is a more complicated audio sample to try to recognize'}], u'final': True}, 'segment': 0, 'id': u'd442722e-8f9f-43b1-aae1-234ff7ee5e1a'}
2020-01-27 18:53:37 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:37 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2020-01-27 18:53:37 -   DEBUG:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Pushing buffer of size 7096 to pipeline
2020-01-27 18:53:37 -   DEBUG:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Pushing buffer done
2020-01-27 18:53:37 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Got message from server of type <class 'ws4py.messaging.TextMessage'>
2020-01-27 18:53:37 -    INFO:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Pushing EOS to pipeline
2020-01-27 18:53:38 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:39 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:40 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:41 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:42 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:43 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:44 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:45 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:46 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Checking that decoder hasn't been silent for more than 10 seconds
2020-01-27 18:53:47 - WARNING:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: More than 10 seconds from last decoder hypothesis update, cancelling
2020-01-27 18:53:47 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Master disconnected before decoder reached EOS?
2020-01-27 18:53:47 -    INFO:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Sending EOS to pipeline in order to cancel processing
2020-01-27 18:53:47 -    INFO:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Cancelled pipeline
2020-01-27 18:53:47 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:48 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:49 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:50 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:51 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:52 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:53 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:54 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:55 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:56 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:57 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:58 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:53:59 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:00 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:01 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:02 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:03 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:04 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:05 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:06 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:07 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:08 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:09 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:10 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:11 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:12 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:13 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:14 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:15 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:16 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Waiting for EOS from decoder
2020-01-27 18:54:17 -    INFO:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Giving up waiting after 31 tries
2020-01-27 18:54:17 -    INFO:   decoder2: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Resetting decoder state

The audio is correctly recognized ("here is a more complicated audio sample to try to recognize"). But something seems to go wrong at the end of things. No result seems to make it back to the client. The client hangs like this:

drewes@ladastra:~/kaldi-test$ python client.py -u ws://localhost:8080/client/ws/speech -r 32000 morecomplicated.wav
Audio sent, now sending EOS

During this period the master log looks like this:

    INFO 2020-01-27 18:53:17,612 101 GET /client/ws/speech?content-type= (172.17.0.1) 0.43ms 
    INFO 2020-01-27 18:53:17,612 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: OPEN 
    INFO 2020-01-27 18:53:17,612 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Request arguments: content-type="" 
    INFO 2020-01-27 18:53:17,612 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Using worker <__main__.DecoderSocketHandler object at 0x7f81dc286e10> 
    INFO 2020-01-27 18:53:17,840 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:18,091 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:18,342 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:18,593 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:18,845 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:19,097 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:19,347 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:19,599 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:19,850 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:20,102 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:20,353 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:20,604 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:20,855 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:21,107 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:21,358 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:21,609 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:21,860 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:22,111 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:22,363 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:22,615 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:22,866 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:23,117 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:23,368 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:23,620 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:23,871 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:24,122 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:24,373 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:24,625 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:24,877 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:25,127 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:25,379 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:25,630 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:25,881 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:26,133 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:26,384 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:26,635 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:26,887 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:27,138 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:27,389 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:27,640 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:27,891 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:28,143 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:28,395 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:28,645 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:28,896 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:29,148 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:29,400 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:29,651 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:29,902 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:30,153 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:30,405 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:30,656 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:30,907 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:31,158 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:31,409 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:31,661 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:31,913 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:32,163 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:32,414 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:32,666 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:32,917 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:33,168 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:33,419 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:33,671 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:33,922 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:34,174 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:34,425 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:34,676 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:34,927 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:35,179 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:35,431 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:35,681 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:35,933 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:36,184 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:36,436 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:36,687 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:36,938 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:37,189 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:37,441 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 18:53:37,692 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'str'>) of length 7096 to worker 
    INFO 2020-01-27 18:53:37,694 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Forwarding client message (<type 'unicode'>) of length 3 to worker

If I kill the client (control-C does nothing; I have to control-Z and then kill the process manually), then the worker sees the connection drop:

2020-01-27 19:04:46 -   DEBUG:      ws4py: Closing message received (1000) ''
2020-01-27 19:04:46 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Websocket closed() called
2020-01-27 19:04:46 -   DEBUG:   __main__: d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Websocket closed() finished
2020-01-27 19:04:47 -    INFO:   __main__: Opening websocket connection to master server
2020-01-27 19:04:47 -    INFO:   __main__: Opened websocket connection to server

But the next time I try the client, the worker only does this:

2020-01-27 19:08:56 -   DEBUG:   __main__: <undefined>: Got message from server of type <class 'ws4py.messaging.TextMessage'>
2020-01-27 19:08:56 -    INFO:   decoder2: 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Initializing request

And the client hangs as above. The master log looks like this during this period:

    INFO 2020-01-27 19:04:46,173 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Handling on_connection_close() 
    INFO 2020-01-27 19:04:46,173 d442722e-8f9f-43b1-aae1-234ff7ee5e1a: Closing worker connection 
    INFO 2020-01-27 19:04:46,174 Worker <__main__.WorkerSocketHandler object at 0x7f81dc286710> leaving 
    INFO 2020-01-27 19:04:47,178 101 GET /worker/ws/speech (127.0.0.1) 0.62ms 
    INFO 2020-01-27 19:04:47,178 New worker available <__main__.WorkerSocketHandler object at 0x7f81dc22c090> 
    INFO 2020-01-27 19:08:56,907 101 GET /client/ws/speech?content-type= (172.17.0.1) 0.38ms 
    INFO 2020-01-27 19:08:56,907 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: OPEN 
    INFO 2020-01-27 19:08:56,907 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Request arguments: content-type="" 
    INFO 2020-01-27 19:08:56,907 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Using worker <__main__.DecoderSocketHandler object at 0x7f81dc22c4d0> 
    INFO 2020-01-27 19:08:57,136 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:57,388 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:57,639 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:57,890 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:58,142 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:58,393 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:58,645 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:58,895 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:59,147 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:59,398 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:59,650 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:08:59,901 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:00,152 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:00,404 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:00,655 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:00,907 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:01,157 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:01,409 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:01,660 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:01,911 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:02,163 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:02,414 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:02,665 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:02,917 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:03,168 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:03,419 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:03,670 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:03,922 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:04,173 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:04,425 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:04,676 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:04,927 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:05,178 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:05,430 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:05,681 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:05,932 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:06,183 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:06,435 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:06,686 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:06,937 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:07,189 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:07,440 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:07,691 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:07,942 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:08,193 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:08,444 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:08,696 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:08,948 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:09,198 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:09,450 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:09,701 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:09,953 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:10,203 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:10,455 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:10,706 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:10,957 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:11,209 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:11,460 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:11,711 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:11,962 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:12,214 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:12,465 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:12,716 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:12,967 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:13,219 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:13,471 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:13,721 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:13,973 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:14,224 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:14,476 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:14,727 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:14,978 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:15,229 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:15,481 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:15,732 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:15,983 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:16,234 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:16,486 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:16,737 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 8000 to worker 
    INFO 2020-01-27 19:09:16,989 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'str'>) of length 7096 to worker 
    INFO 2020-01-27 19:09:16,990 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Forwarding client message (<type 'unicode'>) of length 3 to worker 
    INFO 2020-01-27 19:10:31,472 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Handling on_connection_close() 
    INFO 2020-01-27 19:10:31,472 1ba93fc0-1558-476e-acdf-5cfaffeeeb7c: Closing worker connection 
    INFO 2020-01-27 19:10:36,476 Worker <__main__.WorkerSocketHandler object at 0x7f81dc22c090> leaving

The same thing happens when I use the test audio files (bill_gates-TED.mp3, 1272-128104-0000.wav) at the end of the README.md: the recognition is performed once, but no result is returned to the client (the client simply hangs) and subsequent submissions from client fail until the server is stopped and restarted.

Any suggestion for how to most quickly find the problem?

rpdrewes commented 4 years ago

Thinking this might be a duplicate of issue #59. Testing.

rpdrewes commented 4 years ago

Yes indeed forcing tornado 4.3 in the docker image fixed it. Thanks @craklyn and @phiresky, and of course @jcsilva.

I do wonder what the root of the problem is ... why is forcing tornado 4.3 in the docker image necessary for things to work properly? I had a vague feeling it was something about EOS message being interpreted as unicode but that is just a wild guess ...