Connection Drops when Sending Audio Data to Sherpa Server via Flask

[Md-Aashiqq]

Description:

I'm experiencing an issue where the connection is unexpectedly terminated when I forward audio data from my Flask server to the Sherpa server using WebSockets. When sending string data, the connection remains stable; however, with audio data (binary), the connection gets closed.

Expected Behavior:

• Audio data should be transmitted from the Flask server to the Sherpa server without interruption.
• The WebSocket connection should remain active during and after the transmission.

Current Behavior:

• Connection drops immediately after attempting to send audio data.
• No issues were observed when transmitting string data.

Steps to Reproduce:

1. Set up a Flask server with integrated WebSocket support.
2. Initiate a connection to a frontend application and receive audio data.
3. Attempt to forward this audio data to the Sherpa server.
4. Observe the connection getting terminated.

Code Snippet (if applicable):

@socketio.on("message")
def receive_from_frontend(data):
    print(f'Sending data to Sherpa Server {datetime.now()}')
    try:
        ws.send(data)  # Connection drops when data is audio bytes
    except Exception as e:
        print(f"Error sending data: {e}")

Environment:

• Python version: [e.g., 3.8]
• Flask version: [e.g., 2.0.1]
• WebSocket library version: [e.g., 0.57.0]

Additional Context:

The primary goal is to forward audio data in chunks from the frontend to the Sherpa server via the Flask intermediary server. This workflow is crucial for real-time audio processing and analysis.

---

[csukuangfj]

Could you please post the error logs?

Also, how large is the data? That is, for the following line:

        ws.send(data)  # Connection drops when data is audio bytes

What is the output of

print(len(data))

---

Which websocket server are you using, i.e., which file are you running?

[Md-Aashiqq]

Flask Application Code

from flask import Flask, jsonify
from flask_socketio import join_room, send, SocketIO
import websocket
import random
from string import ascii_uppercase
from flask_cors import CORS
import uuid
import threading
import time
from datetime import datetime
app = Flask(__name__)
CORS(app)

app.config["SECRET_KEY"] = "somesecretkey"
socketio = SocketIO(app, cors_allowed_origins=["http://localhost:5500"])

SERVER_B_WS_URL = 'ws://localhost:6006'

ws = None  # Declare ws here to ensure it's globally accessible

def on_close(ws, *args):
    print("WebSocket connection closed. Reconnecting...")
    time.sleep(5)  # Wait for 5 seconds before retrying
    ws.connect(SERVER_B_WS_URL)  # Reconnect logic

def on_open(ws):
    print("WebSocket connection opened.")

def on_message(ws, message):
    print(f'Received response from Server B:  ', message)

def ws_run():
    global ws
    ws = websocket.WebSocketApp(SERVER_B_WS_URL,
                                on_message=on_message,
                                on_close=on_close,
                                on_open=on_open)
    ws.run_forever()

if ws is None:
    ws_thread = threading.Thread(target=ws_run)
    ws_thread.start()

@socketio.on("message")
def recieve_from_frontend(data):
    print(f'Sending data to Server B {datetime.now()}')
    print("Len of the Data",len(data))
    if not isinstance(data, bytes):

        print("Expected bytes, received another datatype.")
        return jsonify({"error": "Invalid data format received from frontend"}), 400

    ws.send(data) 

@socketio.on("connect")
def connect():
    user_uuid = str(uuid.uuid4())
    print(f"New connection. Assigning UUID: {user_uuid}")

    # Join the user to the room with the UUID
    join_room(user_uuid)
    send({"status": "Connected to the Server", "uuid": user_uuid})

if __name__ == "__main__":
    socketio.run(app,debug=True, use_reloader=False)

Size of the data and Error Message.

WebSocket connection closed. Reconnecting...
Sending data to Server B 2023-08-11 14:45:48.410658
Len of the Data 2972
<class 'bytes'>

[csukuangfj]

What are your error logs?

Which file from sherpa-onnx are you using?

[Md-Aashiqq]

I'm encountering an issue. When I attempt to send data to sherpa-onnx, the connection automatically closes. there was no Error Message.

And Im using Streaming_Server.py in Python-api-emaple folder

[csukuangfj]

Could you use the latest streaming_server.py?

There should be error logs from the server as well as from the client side.

Any output is helpful for debugging your issue.

[Md-Aashiqq]

Yes, I'm using the latest streaming_Server.py.

This is the log file from the sherpa-onnx- server.

2023-08-11 07:13:45,856 INFO [streaming_server.py:631] {'encoder_model': './sherpa-onnx-streaming-zipformer-en-2023-06-21/encoder-epoch-99-avg-1.int8.onnx', 'decoder_model': './sherpa-onnx-streaming-zipformer-en-2023-06-21/decoder-epoch-99-avg-1.int8.onnx', 'joiner_model': './sherpa-onnx-streaming-zipformer-en-2023-06-21/joiner-epoch-99-avg-1.int8.onnx', 'tokens': './sherpa-onnx-streaming-zipformer-en-2023-06-21/tokens.txt', 'sample_rate': 16000, 'feat_dim': 80, 'decoding_method': 'greedy_search', 'num_active_paths': 4, 'use_endpoint': 1, 'rule1_min_trailing_silence': 2.4, 'rule2_min_trailing_silence': 1.2, 'rule3_min_utterance_length': 20, 'port': 6006, 'nn_pool_size': 1, 'max_batch_size': 50, 'max_wait_ms': 10, 'max_message_size': 1048576, 'max_queue_size': 32, 'max_active_connections': 500, 'num_threads': 1, 'certificate': None, 'doc_root': './web'}
2023-08-11 07:13:53,899 INFO [streaming_server.py:476] No certificate provided
2023-08-11 07:13:53,909 INFO [server.py:711] server listening on [::]:6006
2023-08-11 07:13:53,912 INFO [server.py:711] server listening on 0.0.0.0:6006
2023-08-11 07:13:53,918 INFO [streaming_server.py:493] Please visit one of the following addresses:

  http://0.0.0.0:6006
  http://localhost:6006
  http://127.0.0.1:6006
  http://172.17.0.2:6006

2023-08-11 07:13:59,361 INFO [server.py:646] connection open
2023-08-11 07:13:59,363 INFO [streaming_server.py:534] Connected: ('172.17.0.1', 56670). Number of connections: 1/500
2023-08-11 07:15:14,339 INFO [streaming_server.py:513] ('172.17.0.1', 56670) disconnected
2023-08-11 07:15:14,343 INFO [streaming_server.py:518] Disconnected: ('172.17.0.1', 56670). Number of connections: 0/500
2023-08-11 07:15:14,345 INFO [server.py:268] connection closed
2023-08-11 07:15:23,890 INFO [server.py:646] connection open
2023-08-11 07:15:23,891 INFO [streaming_server.py:534] Connected: ('172.17.0.1', 57322). Number of connections: 1/500
2023-08-11 07:15:45,342 INFO [streaming_server.py:513] ('172.17.0.1', 57322) disconnected
2023-08-11 07:15:45,346 INFO [streaming_server.py:518] Disconnected: ('172.17.0.1', 57322). Number of connections: 0/500
2023-08-11 07:15:45,347 INFO [server.py:268] connection closed
2023-08-11 09:15:30,493 INFO [server.py:646] connection open
2023-08-11 09:15:30,494 INFO [streaming_server.py:534] Connected: ('172.17.0.1', 59944). Number of connections: 1/500
2023-08-11 09:15:53,406 INFO [streaming_server.py:513] ('172.17.0.1', 59944) disconnected
2023-08-11 09:15:53,409 INFO [streaming_server.py:518] Disconnected: ('172.17.0.1', 59944). Number of connections: 0/500
2023-08-11 09:15:53,411 INFO [server.py:268] connection closed

After sending the first chuck audio data, the connection was closed.

WebSocket connection opened.
127.0.0.1 - - [11/Aug/2023 14:45:46] "GET /socket.io/?EIO=4&transport=polling&t=OdZuMIH HTTP/1.1" 200 -
127.0.0.1 - - [11/Aug/2023 14:45:46] "GET /socket.io/?EIO=4&transport=polling&t=OdZuMIQ&sid=7W9D6BtYrcewzJHAAAAA HTTP/1.1" 200 -
New connection. Assigning UUID: d0d71114-3f81-4868-bab7-0cc13df133d4
127.0.0.1 - - [11/Aug/2023 14:45:46] "POST /socket.io/?EIO=4&transport=polling&t=OdZuMIP&sid=7W9D6BtYrcewzJHAAAAA HTTP/1.1" 200 -
Sending data to Server B 2023-08-11 14:45:48.372287
Len of the Data 2972
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
WebSocket connection closed. Reconnecting...
Sending data to Server B 2023-08-11 14:45:48.410658
Len of the Data 2972
<class 'bytes'>
Error sending chunk: Connection is already closed.

[csukuangfj]

Yes, I'm using the latest streaming_Server.py.

No, yours is not the latest one.

---

def recieve_from_frontend(data):

What is the format of data?

[csukuangfj]

Does https://github.com/k2-fsa/sherpa-onnx/blob/master/python-api-examples/online-websocket-client-decode-file.py work for you?

If yes, I suggest that you check the message format you are using.

[Md-Aashiqq]

Its a -> <class 'bytes'>

[csukuangfj]

I know that it is a byte array. What I am meaning is that what is the content in the byte array.

Does https://github.com/k2-fsa/sherpa-onnx/blob/master/python-api-examples/online-websocket-client-decode-file.py work for you?

[Md-Aashiqq]

I'm utilizing the web code from the Sherpa example, and I've transmitted the audio data.

Could you please explain how can I use this - https://github.com/k2-fsa/sherpa-onnx/blob/master/python-api-examples/online-websocket-client-decode-file.py

[csukuangfj]

Please see the documentation https://k2-fsa.github.io/sherpa/onnx/websocket/online-websocket.html

By the way, k2-fsa/sherpa and k2-fsa/sherpa-onnx do NOT share the same message format.

[csukuangfj]

Sorry, the correct link should be

https://k2-fsa.github.io/sherpa/onnx/python/streaming-websocket-server.html

[csukuangfj]

@Md-Aashiqq Have you fixed it?