Worker - Couldn't connect to server, waiting for 5 seconds

[mdundek]

Hello,

First of all, huge thank you for this docker image. I really appreciate you sharing this with the community, you deserve lot's of credits for this.

I do have an issue when testing my setup. When I use the test command from my host:

python /tmp/client.py -u ws://localhost:8080/client/ws/speech -r 8192 /tmp/bill_gates-TED.mp3

Nothing happens, the process just stays stuck and I have to kill it manually. If I do wait for long enough, I get the following error:

Traceback (most recent call last):
  File "/tmp/client.py", line 127, in <module>
    main()
  File "/tmp/client.py", line 122, in main
    ws.connect()
  File "/home/nasadmin/.local/lib/python2.7/site-packages/ws4py/client/__init__.py", line 216, in connect
    bytes = self.sock.recv(128)
socket.error: [Errno 104] Connection reset by peer

Here is my worker.log file after the above attempt:

   DEBUG 2016-11-24 12:02:17,217 Starting up worker
2016-11-24 12:02:17 -    INFO:   decoder2: Creating decoder using conf: {'post-processor': "perl -npe 'BEGIN {use IO::Handle; STDOUT->autoflush(1);} s/(.*)/\\1./;'", 'logging': {'version': 1, 'root': {'level': 'DEBUG', 'handlers': ['console']}, 'formatters': {'simpleFormater': {'datefmt': '%Y-%m-%d %H:%M:%S', 'format': '%(asctime)s - %(levelname)7s: %(name)10s: %(message)s'}}, 'disable_existing_loggers': False, 'handlers': {'console': {'formatter': 'simpleFormater', 'class': 'logging.StreamHandler', 'level': 'DEBUG'}}}, 'use-vad': False, 'decoder': {'ivector-extraction-config': '/opt/models/english/tedlium_nnet_ms_sp_online/conf/ivector_extractor.conf', 'num-nbest': 10, 'lattice-beam': 6.0, 'acoustic-scale': 0.083, 'do-endpointing': True, 'beam': 10.0, 'max-active': 10000, 'fst': '/opt/models/english/tedlium_nnet_ms_sp_online/HCLG.fst', 'mfcc-config': '/opt/models/english/tedlium_nnet_ms_sp_online/conf/mfcc.conf', 'use-threaded-decoder': True, 'traceback-period-in-secs': 0.25, 'model': '/opt/models/english/tedlium_nnet_ms_sp_online/final.mdl', 'word-syms': '/opt/models/english/tedlium_nnet_ms_sp_online/words.txt', 'endpoint-silence-phones': '1:2:3:4:5:6:7:8:9:10', 'chunk-length-in-secs': 0.25}, 'silence-timeout': 10, 'out-dir': 'tmp', 'use-nnet2': True}
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'delta'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'max-mem'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'phone-determinize'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'word-determinize'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'minimize'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'beam'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'max-active'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'min-active'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'lattice-beam'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'prune-interval'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'determinize-lattice'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'beam-delta'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'hash-ratio'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
/opt/kaldi-gstreamer-server/kaldigstserver/decoder2.py:48: Warning: When installing property: type 'Gstkaldinnet2onlinedecoder' already has a property named 'acoustic-scale'
  self.asr = Gst.ElementFactory.make("kaldinnet2onlinedecoder", "asr")
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: ivector-extraction-config = /opt/models/english/tedlium_nnet_ms_sp_online/conf/ivector_extractor.conf
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: num-nbest = 10
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: lattice-beam = 6.0
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: acoustic-scale = 0.083
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: do-endpointing = True
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: beam = 10.0
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: max-active = 10000
2016-11-24 12:02:17 -    INFO:   decoder2: Setting decoder property: fst = /opt/models/english/tedlium_nnet_ms_sp_online/HCLG.fst
2016-11-24 12:02:22 -    INFO:   decoder2: Setting decoder property: mfcc-config = /opt/models/english/tedlium_nnet_ms_sp_online/conf/mfcc.conf
2016-11-24 12:02:22 -    INFO:   decoder2: Setting decoder property: traceback-period-in-secs = 0.25
2016-11-24 12:02:22 -    INFO:   decoder2: Setting decoder property: model = /opt/models/english/tedlium_nnet_ms_sp_online/final.mdl
2016-11-24 12:02:22 -    INFO:   decoder2: Setting decoder property: word-syms = /opt/models/english/tedlium_nnet_ms_sp_online/words.txt
2016-11-24 12:02:23 -    INFO:   decoder2: Setting decoder property: endpoint-silence-phones = 1:2:3:4:5:6:7:8:9:10
2016-11-24 12:02:23 -    INFO:   decoder2: Setting decoder property: chunk-length-in-secs = 0.25
2016-11-24 12:02:23 -    INFO:   decoder2: Created GStreamer elements
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstAppSrc object at 0x7f6f5bdb5410 (GstAppSrc at 0x10771b0)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstDecodeBin object at 0x7f6f5bdb5320 (GstDecodeBin at 0x106a090)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstAudioConvert object at 0x7f6f5bdb52d0 (GstAudioConvert at 0x108ba60)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstAudioResample object at 0x7f6f5bdb53c0 (GstAudioResample at 0xf26770)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstTee object at 0x7f6f5bdb5280 (GstTee at 0x109c000)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstQueue object at 0x7f6f5bdb5370 (GstQueue at 0x10a0210)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstFileSink object at 0x7f6f5bdb5460 (GstFileSink at 0x10a4800)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstQueue object at 0x7f6f5bdb54b0 (GstQueue at 0x10a0500)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.Gstkaldinnet2onlinedecoder object at 0x7f6f5bd64320 (Gstkaldinnet2onlinedecoder at 0x1071c70)> to the pipeline
2016-11-24 12:02:23 -   DEBUG:   decoder2: Adding <__main__.GstFakeSink object at 0x7f6f5bd64370 (GstFakeSink at 0xfae200)> to the pipeline
2016-11-24 12:02:23 -    INFO:   decoder2: Linking GStreamer elements
LOG (ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ComputeDerivedVars():ivector-extractor.cc:204) Done.
2016-11-24 12:02:23 -    INFO:   decoder2: Setting pipeline to READY
2016-11-24 12:02:23 -    INFO:   decoder2: Set pipeline to READY
2016-11-24 12:02:23 -    INFO:   __main__: Opening websocket connection to master server
2016-11-24 12:02:23 -    INFO:   __main__: Opened websocket connection to server

I followed your instructions to the letter, did I miss something?

Just in case, here is my master.log file as well

DEBUG 2016-11-24 12:02:17,218 Starting up server
    INFO 2016-11-24 12:02:23,587 New worker available <__main__.WorkerSocketHandler object at 0x7f8b60739b90>

I started the Docker image as such:

docker run -it -p 8080:80 -v /media/kaldi_models:/opt/models jcsilva/docker-kaldi-gstreamer-server:latest /bin/bash

It looks like I am almost there, any suggestions would be very much appreciated.

Thanks,

Michael

[jcsilva]

Hi,

it seems everything is all right... could you confirm if you are running the docker in the same machine you are running the client.py?

Also, if possible, try it: https://github.com/jcsilva/docker-kaldi-gstreamer-server/tree/master/examples/practical-example

[mdundek]

I did the test from the host machine that runs the container. The script works when I run it from within the running docker container on port 80. Looks like there is a problem with the port mapping 8080:80. Not sure what I am doing wrong here...

Quick question beside this pb. The script prints out the text at the pace of the recorded audio file. Is there a way to accelerate this? Using the HTTP REST endpoint maybe (http://localhost:8888/client/dynamic/recognize)?

Thanks in advance,

Michael

[jcsilva]

Interesting your problem with the port mapping... I've never faced it... would it be possible you configured some rule in your iptable and it is causing the problem? It's a guess... and I have no idea. But I would like to learn about your solution :)

About your question, I imagine you just want to receive the recognition result and you don't want to get the partial results in your application, right? If it is the point, you just have to use the responses with the field "final" equals to "true", ignoring all responses in which final==false

About accelerating the recognition time, it is possible. But I discourage it. You should have to change some decoding parameters, and it will damage your recognition accuracy. Usually you would "fail faster" and it would produce useless results.

[wang850228803]

I have the same problem. A few month ago, I can use it normally. But today I try to use it again, this problem appear. I don't know what leads to the problem.

[warp1337]

bump same here in 2018. I have everything running and got stuck because the client py wont connect. Any suggestions?

[warp1337]

Okay everything works as expected. I followed the tutorial and tried everything directly. I just needed to reboot, I guess that actually set the app-amor rules for docker correctly.

[warp1337]

Additionally, for people running into the same error: please check if you have a VPN running, my VPN blocked the connection to localhost for instance. Maybe this can go into the docs somehow.

[jcsilva]

Hi @warp1337, thanks for your comments. I've never tried to use this in a VPN and I could not reproduce the error here. I think it might be useful to be somewhere in the doc. Would you like to write something explaining the details and make a PR?

[warp1337]

Can do. But this will take a while. I add this on my todo list.

[salisaim]

Hi, I am using the tutorial and everything is working fine as far as running both master and worker in the same container using the following. docker run -it -p 8080:80 -v /media/kaldi_models:/opt/models jcsilva/docker-kaldi-gstreamer-server:latest /bin/bash and then inside the container launched worker /opt/start.sh -y /opt/model/nnet.yaml

But if I run 2 docker instances on the same machine, first one using the command docker run -it -p 8080:80 -v /media/kaldi_models:/opt/models jcsilva/docker-kaldi-gstreamer-server:latest /bin/bash

and the second one using docker run -it -v /media/kaldi_models:/opt/models jcsilva/docker-kaldi-gstreamer-server:latest /bin/bash

and running worker inside the second container /opt/start.sh -y /opt/model/nnet.yaml -m first-container -p 8080

the client is unable to connect. Remember both containers are on the same machine. What am I doing wrong?

[warp1337]

I guess you forgot to forward the port from the second container, thus you have no endpoint to connect to.

[salisaim]

forward as in with IPTables or some docker conf?

[jcsilva]

@salisaim, are you able to ping "first-container" from your second container?

[salisaim]

yeap! first container is able to ping the IPaddress of the second one. If I get you correctly you are referring to internal network 172.x.x.x

[jcsilva]

Just to be sure we are aligned ... when you enter in your second container (where there is only a recognition worker) and send a ping to your first container (where your master server is located) everything goes fine, right?

But, if you enter in your second container and execute /opt/start.sh -y /opt/model/nnet.yaml -m IP_ADDRESS_OF_MASTER_CONTAINER -p 8080 nothing happens, right?

Could you paste your master log file from the first container and worker log file from your second container?

[salisaim]

Hey I have it running - Totally bad on my part - appologies - I messed up running only a worker on master. Once I looked into the start.sh script. I realized master isn't up . jcsilva your mention of looking closely got me wondering and i found out that I am not starting the master in standalone mode properly.

Thanks guys - so conclusion is below. We have a clean install (Centos instance on AWS, running 2 docker containers, identical images). Every things works fine. (no routing changes, no IPTABLE entries). I simple started my master in container manually, not using the start script. On the other container started the worker with start-script and (-m MASTER_IP) (-p MASTER_PORT) flag. and it worked.

Thanks Guys