search

SEPIA-Framework / sepia-docs

Documentation and Wiki for SEPIA. Please post your questions and bug-reports here in the issues section! Thank you :-)
https://sepia-framework.github.io/
236 stars 16 forks source link

install new stt server error (on Celeron CPU or certain VMs) #218

Open royrogermcfreely opened 1 year ago

royrogermcfreely commented 1 year ago

hey, i run sepia on a debian vm with 4 cores and 16gb ram (for stt testing)

i tried to download/install the new stt server via docker commands from the wiki. but when i run sudo docker run --rm --name=sepia-stt -p 20741:20741 -it sepia/stt-server:latest

i get this error:

SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch

the same happens when i try to install the stt server with this command 

wget https://raw.githubusercontent.com/SEPIA-Framework/sepia-stt-server/master/scripts/install.sh
bash install.sh -y
bash run.sh

then i get this error:

Running SEPIA STT-Server
run.sh: Zeile 23: 27702 Ungültiger Maschinenbefehl   nohup python3 -u -m launch &>> ../log.out
curl: (7) Failed to connect to localhost port 20741: Verbindungsaufbau abgelehnt

2022_12_10_11:58:56 - Start
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...

i removed all old docker stt images and cleaned the folder where i downloaded the files from install.sh script.

the old stt server was working.

could you give me a hint how i get the new stt server working?

/roy

fquirin commented 1 year ago

Hey roy,

this is the x86 container right? Just tried it with the same command and a fresh load from DockerHUB and its working for me 🤔 .

Failed to connect to localhost port 20741: Verbindungsaufbau abgelehnt

I wonder if there is something blocking the port maybe?

vanseforge commented 1 year ago

Hi,

I have the same problem like roy: I tried to install the stt server on a Synology DS920+ (CPU Intel Celeron J4125 -> x86 container) via docker an get also the following message:

sepia-stt     | SEPIA STT Server - Settings file used: './server.conf'
sepia-stt     | SEPIA STT Server - Settings file tag: 'Dynamic server settings'
sepia-stt     | SEPIA STT Server - Starting...
sepia-stt     | on-docker.sh: line 2:     7 Illegal instruction     (core dumped) python3 -m launch
sepia-stt exited with code 132
PlanetSmasher commented 1 year ago

Hi Florian,

I have the same issues on my standard "docker" host but aswell as on my testing machine where this image was never pulled before. It always tells me the following.

sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     7 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1 exited with code 132

I am running Debian GNU/Linux 11 (bullseye) in a Proxmox VM on an Intel NUC so it is amd64

As I do not use any custom models yet I just did not set up a custom settings.conf

fquirin commented 1 year ago

Ok, so clearly something is going on here :-/. Unfortunately I cannot reproduce the issue and wonder what could make the same Docker image fail on certain machines and work on others 🤔, after all this is exactly what should NOT happen with Docker 🙈 .

@royrogermcfreely , @PlanetSmasher , @vanseforge could you all run the following commands for me once plz:

Are you using the Docker containers in the default network mode (bridge I think) or in host mode without network isolation? (or any other mode?)

I'll try to spin up more test systems in the meantime to see if any of them show the same error.

PlanetSmasher commented 1 year ago

I am running Debian 11 

user@systemname:~$ uname -a
Linux systemname 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
user@systemname:~$ docker --version
Docker version 20.10.21, build baeda1f

For easier use I wrote a quick docker-compose.yml :

version: '3'

services:
    sepia:
#       image: sepia/stt-server:vosk_amd64 #2018 model
#       image: sepia/stt-server:latest #2021 rewrite
       image: sepia/stt-server:dynamic_v1.0.0_amd64
       ports:
          - 20741:20741
       volumes:
           - './sepia-stt-share:/apps/share'
           - './server.conf:/home/admin/sepia-stt/share/my.conf'
       restart: unless-stopped
       environment:
           - SEPIA_STT_SETTINGS=/home/admin/sepia-stt/share/my.conf
fquirin commented 1 year ago

That looks perfectly fine, as far as I can tell. sepia/stt-server:latest and sepia/stt-server:dynamic_v1.0.0_amd64 should point to the same image btw at the moment. Can you run the same command without ports, volumes and environment just to see what happens plz. Or simply: sudo docker run --rm --name=sepia-stt -it sepia/stt-server:dynamic_v1.0.0_amd64

PlanetSmasher commented 1 year ago

here it comes

root@someserver:/opt/docker/sepia-stt-server# sudo docker run --rm --name=sepia-stt -it sepia/stt-server:dynamic_v1.0.0_amd64
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
root@someserver:/opt/docker/sepia-stt-server#

what I've found out so far: The on-docker.sh runs the python3 -m launch command which tries to start the vosk stuff

I started the docker with the default docker run and added /bin/bash so I can get to the console. now i changed to the ~/sepia-stt/server directory and executed the launch command again

-> 

> admin@a6b00abb49f1:~/sepia-stt/server$ python3 -m launch
> SEPIA STT Server - Settings file used: './server.conf'
> SEPIA STT Server - Settings file tag: 'Dynamic server settings'
> SEPIA STT Server - Starting...
> Illegal instruction (core dumped)
> admin@a6b00abb49f1:~/sepia-stt/server$
fquirin commented 1 year ago

The on-docker.sh runs the python3 -m launch command which tries to start the vosk stuff

Correct, the weird thing is that the only situation I've ever seen this error is when calling a x86 command from arm64 or the other way around.

I think we have to go step by step through the Python script and see where it breaks. Could you do the following:

from settings import SERVER_NAME, SERVER_VERSION from launch_setup import settings from http_api import HttpApiEndpoint, SettingsRequest from socket_api import WebsocketApiEndpoint

app = FastAPI() app.mount("/www", StaticFiles(directory="www"))

app.add_middleware(CORSMiddleware, allow_origins=settings.cors_origins, allow_credentials=True, allow_headers=[""], allow_methods=[""])

http_endpoint = HttpApiEndpoint() socket_endpoint = WebsocketApiEndpoint()

fquirin commented 1 year ago

I found something interesting that might be related to this problem on Stackoverflow. The post claims:

In summary, VBox Guest VM is not able to run tensorflow inside the docker-engine while HyperV Guest VM is able to run.

and:

deploying the containers in Synology NAS 920+, I am experiencing the same problem with tensorflow. In this case maybe is normal as this model uses Intel Celeron J4125 which does not support AVX and AVX2 instructions

The SEPIA STT-Server currently does not use Tensorflow, but maybe Numpy or something related has the same issue. Weird thing is that these libraries are not loaded at start immediately but only when the model is used first time 🤔

PlanetSmasher commented 1 year ago

It looks like Websocket is the bad guy here

admin@a6b00abb49f1:~/sepia-stt/server$ python3 -m server
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$
admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from socket_api import WebsocketApiEndpoint
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$
fquirin commented 1 year ago

ok interesting, I'll dig a bit deeper, brb ^^.

fquirin commented 1 year ago

Can you please skip from socket_api import WebsocketApiEndpoint and instead continue with these lines:

from fastapi import WebSocket, WebSocketDisconnect
from starlette.websockets import WebSocketState
from pydantic import ValidationError

from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)

import time
import asyncio

from chunk_processor import ChunkProcessor

and if that breaks at chunk_processor try:

from engine_vosk import VoskProcessor
from engine_coqui import CoquiProcessor
PlanetSmasher commented 1 year ago

here it comes

admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from socket_api import WebsocketApiEndpoint
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from fastapi import WebSocket, WebSocketDisconnect
>>> from starlette.websockets import WebSocketState
>>> from pydantic import ValidationError
>>> from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)
>>> import time
>>> import asyncio
>>> from chunk_processor import ChunkProcessor
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$

2nd Machine

on my other testing machine I got it working by setting the hypervisor CPU type to host instead of default so it can use probably all features

image

on this second machine the output from the command is: 

user@server2:/opt/docker/sepia-stt-server$ sudo docker run --rm --name=sepia-stt2 -it sepia/stt-server:latest
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
SEPIA STT Server - Server running at: 0.0.0.0:20741
SEPIA STT Server - Speech recognition engine: dynamic
SEPIA STT Server - Models defined for engine: 3
fquirin commented 1 year ago

Can you do one more try with simply:

from engine_interface import EngineInterface, EngineNotFound
from engine_vosk import VoskProcessor
from engine_coqui import CoquiProcessor

I bet the last line is the deal breaker :-|

on my other testing machine I got it working by setting the hypervisor CPU type to host instead of default so it can use probably all features

That is very good to know! I'm assuming that is the Proxmox VM configuration?

fquirin commented 1 year ago

Can you do one more try with simply:

and even more precise:

from text_processor import TextToNumberProcessor, DateAndTimeOptimizer
import numpy as np
from vosk import Model, SpkModel, KaldiRecognizer, SetLogLevel
from stt import Model
PlanetSmasher commented 1 year ago 
from engine_interface import EngineInterface, EngineNotFound
from engine_vosk import VoskProcessor
from engine_coqui import CoquiProcessor

I bet the last line is the deal breaker :-|

yep

admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from fastapi import WebSocket, WebSocketDisconnect
>>> from starlette.websockets import WebSocketState
>>> from pydantic import ValidationError
>>>
>>> from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)
>>> import time
>>> import asyncio
>>> from engine_interface import EngineInterface, EngineNotFound
>>> from engine_vosk import VoskProcessor
>>> from engine_coqui import CoquiProcessor
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$

and even more precise:

from text_processor import TextToNumberProcessor, DateAndTimeOptimizer
import numpy as np
from vosk import Model, SpkModel, KaldiRecognizer, SetLogLevel
from stt import Model
admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from fastapi import WebSocket, WebSocketDisconnect
>>> from starlette.websockets import WebSocketState
>>> from pydantic import ValidationError
>>> from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)
>>> import time
>>> import asyncio
>>> from text_processor import TextToNumberProcessor, DateAndTimeOptimizer
>>> import numpy as np
>>> from vosk import Model, SpkModel, KaldiRecognizer, SetLogLevel
>>> from stt import Model
Illegal instruction (core dumped)

and Yes it is a Proxmox VM

fquirin commented 1 year ago

Ok, thanks a lot for running through all of this! 😎

The silver lining is that we could theoretically kick Coqui engine and it should keep running. In your server config (server.conf) can you change asr_engine=dynamic to asr_engine=vosk and try python3 -m launch again :-)

PlanetSmasher commented 1 year ago

that does the trick aswell

admin@a6b00abb49f1:~/sepia-stt/server$ python3 -m launch
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
SEPIA STT Server - Server running at: 0.0.0.0:20741
SEPIA STT Server - Speech recognition engine: vosk
SEPIA STT Server - Models defined for engine: 2
INFO:     Started server process [81]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:20741 (Press CTRL+C to quit)
royrogermcfreely commented 1 year ago

sorry for the late replay.

thanks to @PlanetSmasher - after i changed my proxmox vm cpu from "default" to "host", i was able to start the stt server with the default run.sh script without changing anything.

so i didnt execute the script line by line as mentiont above.

/roy

vanseforge commented 1 year ago

Ok, thanks a lot for running through all of this! 😎

The silver lining is that we could theoretically kick Coqui engine and it should keep running. In your server config (server.conf) can you change asr_engine=dynamic to asr_engine=vosk and try python3 -m launch again :-)

That worked for me also. Thank you @fquirin and @PlanetSmasher .

@PlanetSmasher in your docker-compose example you wrote a ":" instead a "=" in the environment section. Maybe you can edit this if somebody copies your example. ``environment:

I copied it and wondered why the wrong file was loaded 🫣