docker container exits within ~15 seconds

[bigrig2212]

Hi there. After running the docker container using: docker run -d -p 2700:2700 alphacep/kaldi-en:latest with python8.0 it exits after about 15 seconds. i don't see any errors in docker console (below) i do see an error from command line, but i think b/c the docker container shuts down any tips?

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:192) Decoding params beam=13 max-active=7000 lattice-beam=6

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:195) Silence phones 1:2:3:4:5:6:7:8:9:10:11:12:13:14:15

LOG (VoskAPI:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 1 orphan nodes.

LOG (VoskAPI:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 2 orphan components.

LOG (VoskAPI:Collapse():nnet-utils.cc:1472) Added 1 components, removed 2

LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.0211859 seconds in looped compilation.

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:219) Loading i-vector extractor from /opt/vosk-model-en/model/ivector/final.ie

LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor

LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:204) Done.

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:237) Loading HCLG from /opt/vosk-model-en/model/graph/HCLG.fst

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:256) Loading words from /opt/vosk-model-en/model/graph/words.txt

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:264) Loading winfo /opt/vosk-model-en/model/graph/phones/word_boundary.int

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:272) Loading CARPA model from /opt/vosk-model-en/model/rescore/G.carpa

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:192) Decoding params beam=13 max-active=7000 lattice-beam=6

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:195) Silence phones 1:2:3:4:5:6:7:8:9:10:11:12:13:14:15

LOG (VoskAPI:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 1 orphan nodes.

LOG (VoskAPI:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 2 orphan components.

LOG (VoskAPI:Collapse():nnet-utils.cc:1472) Added 1 components, removed 2

LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.013211 seconds in looped compilation.

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:219) Loading i-vector extractor from /opt/vosk-model-en/model/ivector/final.ie

LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor

LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:204) Done.

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:237) Loading HCLG from /opt/vosk-model-en/model/graph/HCLG.fst

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:256) Loading words from /opt/vosk-model-en/model/graph/words.txt

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:264) Loading winfo /opt/vosk-model-en/model/graph/phones/word_boundary.int

LOG (VoskAPI:ReadDataFiles():vosk/model.cc:272) Loading CARPA model from /opt/vosk-model-en/model/rescore/G.carpa

brig-iMac:websocket brig$ ./test.py test.wav
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/http.py", line 139, in read_response
    status_line = await read_line(stream)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/http.py", line 219, in read_line
    raise EOFError("line without CRLF")
EOFError: line without CRLF

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/client.py", line 101, in read_http_response
    status_code, reason, headers = await read_response(self.reader)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/http.py", line 141, in read_response
    raise EOFError("connection closed while reading HTTP status line") from exc
EOFError: connection closed while reading HTTP status line

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./test.py", line 22, in <module>
    asyncio.get_event_loop().run_until_complete(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "./test.py", line 8, in hello
    async with websockets.connect(uri) as websocket:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/client.py", line 517, in __aenter__
    return await self
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/client.py", line 542, in __await_impl__
    await protocol.handshake(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/client.py", line 290, in handshake
    status_code, response_headers = await self.read_http_response()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/client.py", line 103, in read_http_response
    raise InvalidMessage("did not receive a valid HTTP response") from exc
websockets.exceptions.InvalidMessage: did not receive a valid HTTP response
brig-iMac:websocket brig$ docker run -d -p 2700:2700 docker.io/alphacep/kaldi-en:latest
6219b699d57e515c1be75c9f6fa83448d18acba2c2aa50580a3b6f545ac38ab5
brig-iMac:websocket brig$ ping localhost:2700
ping: cannot resolve localhost:2700: Unknown host
brig-iMac:websocket brig$  

[nshmyrev]

Please try to run the server without -d and let us know what happens on console.

[nshmyrev]

Also please provide the version of websockets library you have:

python3 -c 'import websockets; print (websockets, websockets.__version__)'

[bigrig2212]

Thanks for the quick response. Here's the output from both of those commands. Same deal from the docker command, where it returns to console (exits), after that last G.carpa log line after about 15 seconds. Btw, I got the same result on another computer also...

Last login: Tue Nov  3 20:35:33 on ttys000
brig-iMac:vosk brig$ docker run -p 2700:2700 alphacep/kaldi-en:latest
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:192) Decoding params beam=13 max-active=7000 lattice-beam=6
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:195) Silence phones 1:2:3:4:5:6:7:8:9:10:11:12:13:14:15
LOG (VoskAPI:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 1 orphan nodes.
LOG (VoskAPI:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 2 orphan components.
LOG (VoskAPI:Collapse():nnet-utils.cc:1472) Added 1 components, removed 2
LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.0225 seconds in looped compilation.
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:219) Loading i-vector extractor from /opt/vosk-model-en/model/ivector/final.ie
LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:237) Loading HCLG from /opt/vosk-model-en/model/graph/HCLG.fst
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:256) Loading words from /opt/vosk-model-en/model/graph/words.txt
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:264) Loading winfo /opt/vosk-model-en/model/graph/phones/word_boundary.int
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:272) Loading CARPA model from /opt/vosk-model-en/model/rescore/G.carpa

brig-iMac:vosk brig$ python3 -c 'import websockets; print (websockets, websockets.__version__)'
<module 'websockets' from '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/websockets/__init__.py'> 8.1
brig-iMac:vosk brig$ 

[nshmyrev]

after that last G.carpa log line after about 15 seconds.

Looks like your server is very slow, what is your hardware? CPU? Memory? On modern Linux server it takes 2 seconds to start.

Do you use docker on iMac? Maybe you just have to wait for a minute or two until it initializes.

[nshmyrev]

Docker on mac is known to be very slow:

https://stackoverflow.com/questions/55951014/docker-in-macos-is-very-slow

Maybe you can just try the server directly, you need to install vosk, clone the server, download the model and run websocket/asr_server.py, thats it. It will work much faster.

[bigrig2212]

Got some help! We spun up the following on google cloud: e2-medium (2 vCPUs, 4 GB memory) AMD Rome ubuntu-2004-focal-v20201014

And it worked fine there.

[bigrig2212]

after that last G.carpa log line after about 15 seconds.

Looks like your server is very slow, what is your hardware? CPU? Memory? On modern Linux server it takes 2 seconds to start.

Do you use docker on iMac? Maybe you just have to wait for a minute or two until it initializes.

It's a pretty beefy imac.

Also same results from laptop macAir. And my buddy has same results from his mac. Yes, using Docker. It was initialized for a good bit. tried it about 20 times... same result every time. Just exits after 15-20secs.

[nshmyrev]

Ok, maybe you can try on mac without docker.

[jamesoliver1981]

Got some help! We spun up the following on google cloud: e2-medium (2 vCPUs, 4 GB memory) AMD Rome ubuntu-2004-focal-v20201014

And it worked fine there.

Hey, I want to use run this on google cloud too but don't know where to start. I am new to docker and cloud computing. Could you write up a step by step of how you got this to work on Google?

I have VOSK (non docker) working on my local machine in a python script, but I need to start creating my pipeline for my app so I need to get this working on the cloud.

I know this is very generic but whilst I understand the theory of docker, I have no idea about getting it up and running.

Thank you in advance, J

[ab36245]

I'm coming late to this but I had the exact same problem on my Mac Notebook running Docker Desktop.

I think it's a memory problem. I noticed that Docker Desktop has a max memory setting of 2G. I've got 16G in my Notebook so I cranked it up to 7G (randomly) and it now gets as far as listening on the port and responding (i.e. runs perfectly :-) )

My guess is that the (virtual) host OS (hyperkit) is killing the process during startup as it grows.

Great software btw. I really appreciate the effort that's gone into this.

[ab36245]

Just to finish on this, I built a new image using the full en-us model (vosk-model-en-us-0.22). With Docker Desktop memory still set to 7G I ran this and the behaviour happened again. The output of docker run (foreground) is:

LOG (VoskAPI:ReadDataFiles():model.cc:211) Decoding params beam=13 max-active=7000 lattice-beam=6
LOG (VoskAPI:ReadDataFiles():model.cc:214) Silence phones 1:2:3:4:5:11:12:13:14:15
LOG (VoskAPI:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 0 orphan nodes.
LOG (VoskAPI:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 0 orphan components.
LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.064292 seconds in looped compilation.
LOG (VoskAPI:ReadDataFiles():model.cc:238) Loading i-vector extractor from /opt/vosk-model-en/model/ivector/final.ie
LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (VoskAPI:ReadDataFiles():model.cc:268) Loading HCLG from /opt/vosk-model-en/model/graph/HCLG.fst
LOG (VoskAPI:ReadDataFiles():model.cc:283) Loading words from /opt/vosk-model-en/model/graph/words.txt
LOG (VoskAPI:ReadDataFiles():model.cc:292) Loading winfo /opt/vosk-model-en/model/graph/phones/word_boundary.int
LOG (VoskAPI:ReadDataFiles():model.cc:299) Loading subtract G.fst model from /opt/vosk-model-en/model/rescore/G.fst
LOG (VoskAPI:ReadDataFiles():model.cc:301) Loading CARPA model from /opt/vosk-model-en/model/rescore/G.carpa
LOG (VoskAPI:ReadDataFiles():model.cc:307) Loading RNNLM model from /opt/vosk-model-en/model/rnnlm/final.raw

and that's where it ends - exactly as before.

Increasing the Docker Desktop memory setting to 8G lets it run:

...
LOG (VoskAPI:ReadDataFiles():model.cc:299) Loading subtract G.fst model from /opt/vosk-model-en/model/rescore/G.fst
LOG (VoskAPI:ReadDataFiles():model.cc:301) Loading CARPA model from /opt/vosk-model-en/model/rescore/G.carpa
LOG (VoskAPI:ReadDataFiles():model.cc:307) Loading RNNLM model from /opt/vosk-model-en/model/rnnlm/final.raw
INFO:root:Listening on 0.0.0.0:2700

The key is that it has finished its initialisation and is now listening.

Needing 8G for the full en-us model exactly agrees with the Troubleshooting section here.

The moral of the story is that these models are memory hungry and you need to allocate sufficient resources. On top of that, running the server in a container makes it slightly more tricky to track this down and on top that running on a mac (and I'm guessing on windows as well) where the docker host is itself virtualised makes this a little more difficult still. In these cases the (virtual) OS is terminating the process (with extreme prejudice :-)) during its initialisation stage but there is no indication that this is what has happened. I'd say the key thing to notice is that the server never gets beyond initialisation to the listening stage.

[damascene]

I think it will help people understand the issue if the error message was more verbose and clear. Now it's just a single word:

LOG (VoskAPI:ReadDataFiles():model.cc:323) Loading RNNLM model from /path/vosk-model/vosk-model-en-us-0.42-gigaspeech//rnnlm/final.raw
Terminated

Also, even though vosk-transcriber --help shows --log-level LOG_LEVEL it doesn't show anything if I set it to DEBUG. Is there somewhere where I can read more about CLI options, did some search but couldn't find much.

[nshmyrev]

@damascene it is a message from OS that kills the process, not from the application.

[damascene]

@nshmyrev I see, would it possible for the app to do a check before it starts loading a module that require more memory than the current available one or maybe handle that in the app in someway?

[nshmyrev]

Sure, maybe we shall add such check in the future

[aninditasadhukhan]

Hi @nshmyrev , I am facing the same problem, after running "docker run -d -p 2700:2700 alphacep/kaldi-en:latest" the vosk docker stopped after few second, below is the log when run without -d.

LOG (VoskAPI:ReadDataFiles():model.cc:213) Decoding params beam=13 max-active=7000 lattice-beam=6 LOG (VoskAPI:ReadDataFiles():model.cc:216) Silence phones 1:2:3:4:5:11:12:13:14:15 LOG (VoskAPI:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 0 orphan nodes. LOG (VoskAPI:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 0 orphan components. LOG (VoskAPI:ReadDataFiles():model.cc:249) Loading i-vector extractor from /opt/vosk-model-en/model/ivector/final.ie LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:204) Done. LOG (VoskAPI:ReadDataFiles():model.cc:282) Loading HCLG from /opt/vosk-model-en/model/graph/HCLG.fst LOG (VoskAPI:ReadDataFiles():model.cc:300) Loading words from /opt/vosk-model-en/model/graph/words.txt LOG (VoskAPI:ReadDataFiles():model.cc:311) Loading winfo /opt/vosk-model-en/model/graph/phones/word_boundary.int LOG (VoskAPI:ReadDataFiles():model.cc:318) Loading subtract G.fst model from /opt/vosk-model-en/model/rescore/G.fst LOG (VoskAPI:ReadDataFiles():model.cc:320) Loading CARPA model from /opt/vosk-model-en/model/rescore/G.carpa LOG (VoskAPI:ReadDataFiles():model.cc:326) Loading RNNLM model from /opt/vosk-model-en/model/rnnlm/final.raw

Please help me on this regards. Thanks,

[ab36245]

Have you checked that you have enough memory?

See my earlier comments in this issue about why this happens: https://github.com/alphacep/vosk-server/issues/77#issuecomment-953702889

[aninditasadhukhan]

Have you checked that you have enough memory?

See my earlier comments in this issue about why this happens: #77 (comment)

Yes I have checked that, I have 8 GB RAM in my machine, is it sufficient to run vosk server in docker?

[ab36245]

It all depends on the model you are using. Some models are very big and require a lot of memory! In my example above I was using vosk-model-en-us-0.22 and I needed to allocate 8G to the docker container. That probably won't work on a host with 8G.

Can you test with a smaller model?

[aninditasadhukhan]

Can please suggest the smaller model, I mean the version, I don't have the idea which one will run on this configuration. Thanks for quick response.

[ab36245]

I'm no expert. Have a look at the listed models: https://alphacephei.com/vosk/models.

I found both vosk-model-en-us-0.22 and vosk-model-en-us-0.42-gigaspeech to need a lot of memory.

vosk-model-small-en-us-0.15 works well for me for testing, and will definitely run in an 8G machine. It's less accurate but in my (limited) experience, still pretty good.

I don't know anything about vosk-model-en-us-0.22-lgraph or models for other languages.

[aninditasadhukhan]

Thanks for the details. I have checked in docker hub, but did not find any images regarding docker images for vosk-model-small-en-us-0.15 https://hub.docker.com/u/alphacep. Can you please help how can I run that particular image in docker container?

Thank you so much for your response.

[ab36245]

I tend to build my own images. Have a look at https://github.com/alphacep/vosk-server/tree/master/docker.

If you look at the English Dockerfile you can see that it builds vosk-model-en-us-0.22 by default but it is trivial to change it, either in the file directly or through the MODEL_VERSION environment variable.

[aninditasadhukhan]

Thank you so much @ab36245 , I am able to run vosk-model-small-en-us-0.15 in my machine and it is running successfully.