Cannot handle multiple streams concurrently

[MohammedShokr]

Issue

I implemented a WebSocket-based version of the whisper_online_server to handle audio streams from clients over WebSocket connections. The implementation works as expected when a single client is streaming; however, when two clients stream simultaneously, significant issues arise:

• High Latency: Latency increases drastically, sometimes reaching up to a minute for both clients.
• Non-concurrent Handling: Instead of processing both streams concurrently, it appears as if the server handles them in turns, causing delays and bottlenecks.

  Troubleshooting Attempts

  I’ve tried both of the following approaches, but neither resolved the issue:

• Shared ASR Model: Using a single ASR model instance shared across both streams with threading lock.
• Separate ASR Model Instances: Creating a separate ASR model instance for each client stream. Both approaches resulted in the same high-latency, turn-taking behavior.

  Code

  
  import sys
  import asyncio
  import io
  import os
  import threading  
  import librosa
  import soundfile as sf
  import numpy as np
  import websockets
  import uuid
  from whisper_online import FasterWhisperASR, VACOnlineASRProcessor

Initialize the model once and share it among all clients

model = FasterWhisperASR(lan="ar", modelsize="large-v2", compute_type="float16", device="cuda") model.use_vad()

Lock to ensure thread-safe access to the model

model_lock = threading.Lock()

def process_audio_sync(online_asr_processor, audio): """Synchronous function to process audio using the ASR model.""" online_asr_processor.insert_audio_chunk(audio)

Lock the model during the critical section

with model_lock:
    output = online_asr_processor.process_iter()
return output

async def process_audio(websocket: websockets.WebSocketServerProtocol, path):

Create a per-client processor using the shared model

online_asr_processor = VACOnlineASRProcessor(
    online_chunk_size=1,
    asr=model,
    tokenizer=None,
    buffer_trimming=("segment", 15),
    logfile=sys.stderr
)

loop = asyncio.get_running_loop()

async for message in websocket:
    if not isinstance(message, bytes):
        print(message)
        continue

    # Process the audio data
    sound_file = sf.SoundFile(
        io.BytesIO(message),
        channels=1,
        endian="LITTLE",
        samplerate=16000,
        subtype="PCM_16",
        format="RAW"
    )
    audio, _ = librosa.load(sound_file, sr=16000, dtype=np.float32, mono=True)

    # Offload the blocking ASR processing to a thread pool executor
    output = await loop.run_in_executor(None, process_audio_sync, online_asr_processor, audio)

    if output[0] is None:
        continue
    else:
        output_formatted = online_asr_processor.to_flush([output])
        await websocket.send(output_formatted[2].encode())

async def main():

Disable the server's keepalive pings to prevent timeouts

async with websockets.serve(process_audio, "localhost", 8765, ping_interval=None, ping_timeout=None):
    await asyncio.Future()

if name == "main": asyncio.run(main())

[Gldkslfmsd]

hi, it's known limitation. GPUs are the fastest with one process only. Sequential processing is faster than concurrent. Current Whisper-Streaming is intended for one client at a time. Batching -- #42 , could help, but still, there will be slow down. Refer to #42.

[MohammedShokr]

Hi @Gldkslfmsd, thanks for you reply. so the current Whisper-Streaming cannot be used in production applications with many users? is it just a POC for the streaming?

[Gldkslfmsd]

yes, it's demo, POC. Not for many users concurrently.