Audio is inconsistently reaching Deepgram (1/50 connections)

I am getting really inconsistent results trying to adapt this repo for my own use-case (chrome extension). Ever so randomly it does seem to connect successfully and produce transcriptions but I cannot consistently replicate it.

Audio data is being sent from the client (verified through console logs) to the express server which is opening up a connection to Deepgram and then sending to Deepgram servers. I have checked my Deepgram logs and I always get a status 101. But the sane logs also show that the audio duration is always zero so it seems as if the server is not handing off the messages streamed from the browser to Deepgram)

As no audio seems to be received by Deepgram's servers, Deepgram is disconnecting the websocket after 10 seconds or so resulting in a 1006 error.

The audio is being streamed from the mic and seemingly the audio packets are being sent to Deepgram's servers when I check the logs)

The only glimmer of a clue that I have seems to be if the Deepgram instance is made immediately, before messages from the websocket are sent to Deepgram's servers then the transcription works. But this rarely happens and I cannot reproduce it.

However, more often than not, the websocket sends a few messages before the Deepgram instance is created resulting in getReadyState == 2. After the Deepgram connection is made, the audio packets seem to be sent to Deepgram but they never seem to be received.

Has anyone encountered this issue? I have barely modified the express server.

const express = require("express");
const http = require("http");
const WebSocket = require("ws");
const cors = require("cors");
const { createClient, LiveTranscriptionEvents } = require("@deepgram/sdk");

const app = express();
const server = http.createServer(app);
const wss = new WebSocket.Server({ server });

const deepgramClient = createClient(process.env.DG_KEY);
let keepAlive;

const setupDeepgram = (ws) => {
  const dgConnection = deepgramClient.listen.live({
    model: "nova-2",
    language: "id",
    smart_format: true,
  });

  if (keepAlive) clearInterval(keepAlive);
  keepAlive = setInterval(() => {
    console.log("Deepgram: KeepAlive sent");
    if (dgConnection && dgConnection.keepAlive) {
      dgConnection.keepAlive();
    }
  }, 3000);

  dgConnection.on(LiveTranscriptionEvents.Open, () => {
    console.log("Deepgram: Connection opened");

    dgConnection.on(LiveTranscriptionEvents.Transcript, (data) => {
      console.log("Deepgram: Transcript received");
      console.log(data.channel.alternatives[0].transcript);
      ws.send(JSON.stringify(data));
    });

    dgConnection.on(LiveTranscriptionEvents.Close, () => {
      console.log("Deepgram: Connection closed");
      clearInterval(keepAlive);
    });

    dgConnection.on(LiveTranscriptionEvents.Error, (err) => {
      console.error("Deepgram: Error received", err);
    });

    dgConnection.on(LiveTranscriptionEvents.Warning, (warning) => {
      console.warn("Deepgram: Warning received", warning);
    });

    dgConnection.on(LiveTranscriptionEvents.Metadata, (data) => {
      console.log("Deepgram: Metadata received", data);
    });
  });

  return dgConnection;
};

wss.on("connection", (ws) => {
  console.log("ws: client connected");
  let deepgram = setupDeepgram(ws);

  ws.on("message", (message) => {
    if (deepgram.getReadyState() === 1 /* OPEN */) {
      console.log("ws: data sent to deepgram");
      deepgram.send(message);
    } else if (deepgram.getReadyState() >= 2 /* 2 = CLOSING, 3 = CLOSED */
    ) {
      console.log("ws: data couldn't be sent to deepgram");
      console.log("ws: retrying connection to deepgram");
      /* Attempt to reopen the Deepgram connection */
      deepgram.finish();
      deepgram.removeAllListeners();
      deepgram = setupDeepgram(ws);
    } else {
      console.log("ws: data couldn't be sent to deepgram");
    }
  });

  ws.on("close", () => {
    console.log("ws: client disconnected");
    if (deepgram) {
      deepgram.finish();
      deepgram.removeAllListeners();
    }
    clearInterval(keepAlive);
  });

  ws.on("error", (error) => {
    console.error("ws: error occurred", error);
  });
});

app.use(cors());

app.get("/api/status", (req, res) => {
  res.json({ status: "ok" });
});

const PORT = process.env.PORT || 3000;

server.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

deepgram-starters / live-node-starter

Audio is inconsistently reaching Deepgram (1/50 connections) #9