search

codeforequity-at / botium-speech-processing

Botium Speech Processing
https://www.botium.at
MIT License
945 stars 58 forks source link

503: Service Unavailable #2

Closed pjgmobile closed 4 years ago

pjgmobile commented 4 years ago

I ran:

curl -X POST "http://127.0.0.1/api/stt/en" -H "Content-Type: audio/wav" -T test1.wav

response:

{"status":"error","message":"Calling url http://stt_en:80/client/dynamic/recognize failed with code 503: Service Unavailable"}

I have tried a test through the Swagger API page and get the same response.

Am I required to do further configuration?

codeforequity-at commented 4 years ago

view log output from the kaldi server:

docker logs botiumspeechprocessing_stt_en
pjgmobile commented 4 years ago

I get the following:

Error: No such container: botiumspeechprocessing_stt_en

codeforequity-at commented 4 years ago

sorry should be

docker logs botiumspeechprocessing_stt_en_1
pjgmobile commented 4 years ago

Same error:

Error: No such container: botiumspeechprocessing_stt_en_1

I tried docker logs botium-speech-processing-master_stt_en_1

which had a similar name, I get the following log in case it's of any use:

2020-01-27 11:49:57,404 CRIT Supervisor running as root (no user in config file) 2020-01-27 11:49:57,406 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing 2020-01-27 11:49:57,752 INFO RPC interface 'supervisor' initialized 2020-01-27 11:49:57,752 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2020-01-27 11:49:57,754 INFO supervisord started with pid 1 2020-01-27 11:49:58,758 INFO spawned: 'worker1' with pid 8 2020-01-27 11:49:58,761 INFO spawned: 'worker2' with pid 9 2020-01-27 11:49:58,763 INFO spawned: 'master' with pid 10 2020-01-27 11:50:00,032 INFO success: worker1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:50:00,032 INFO success: worker2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:50:00,032 INFO success: master entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:51:38,509 INFO exited: worker1 (terminated by SIGKILL; not expected) 2020-01-27 11:51:39,674 INFO spawned: 'worker1' with pid 13 2020-01-27 11:51:40,606 INFO success: worker1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:51:44,082 INFO exited: worker2 (terminated by SIGKILL; not expected) 2020-01-27 11:51:44,112 INFO spawned: 'worker2' with pid 14 2020-01-27 11:51:46,008 INFO success: worker2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:52:00,203 INFO exited: worker1 (terminated by SIGKILL; not expected) 2020-01-27 11:52:01,326 INFO spawned: 'worker1' with pid 15 2020-01-27 11:52:02,737 INFO success: worker1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:52:42,752 INFO exited: worker2 (terminated by SIGKILL; not expected) 2020-01-27 11:52:42,834 INFO spawned: 'worker2' with pid 16 2020-01-27 11:52:43,929 INFO success: worker2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:53:38,467 INFO exited: worker1 (terminated by SIGKILL; not expected) 2020-01-27 11:53:39,507 INFO spawned: 'worker1' with pid 17 2020-01-27 11:53:40,775 INFO success: worker1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-01-27 11:53:43,591 INFO exited: worker2 (terminated by SIGKILL; not expected) 2020-01-27 11:53:44,610 INFO spawned: 'worker2' with pid 18 2020-01-27 11:53:46,015 INFO success: worker2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) ...

codeforequity-at commented 4 years ago

ok, so you downloaded the repository instead of cloning - container names are generated with folder names prepended automatically.

The log output shows that the worker nodes for the speech recogniction fail to start. maybe too less memory, is this possible ?

pjgmobile commented 4 years ago

Thank you. I'll try cloning and rebuilding just in case that helps.

It is possible there is a memory issue, I have 4GB allocated in Docker. Do you have any guidelines or a guesstimate on the minimum amount of memory that should be available?

codeforequity-at commented 4 years ago

From my experience, one worker node for Kaldi needs 1GB of RAM. There are four included in the default configuration (2 german 2 english). you could just remove the german kaldi from the docker-compose file.

pjgmobile commented 4 years ago

Thanks for the help, just reporting some information on STT in case it helps anyone else.

Looking at docker stats, the botium-speech-processing_stt_en_1 increased to the 4GB limit and kept restarting. I increased the limit to 8GB and it is now working. On my system it requires ~4.6GB to start.

I removed the German Kaldi as you suggested to help memory usage. That also required the nginx.conf to be modded too.

My 16 bit .wav test file didn't work, 24 bit and 32 bit are working fine.

Great project, thank you.

codeforequity-at commented 4 years ago

thanks for the report. I will add this to the readme.

there is also a audio conversion service included to make the audio files suitable for ASR.

pnadj commented 2 years ago

Having the same issue as pj except increasing the memory did not help

I did notice the botiumspeechprocessing_stt_en CPU usage is at 203.88% when in idle, docker settings is already at full 8 cores - strange