Event handlers not triggered and client disconnected after 1 message (Deno)

[elimydlarz]

Here's my code:

import { RealtimeClient } from "../node_modules/@openai/realtime-api-beta/index.js";

import { OPENAI_API_KEY, USER_NAME } from "./config/config.ts";
import { getInstructions } from "./instructions/get-instructions.ts";

Deno.serve({ port: 80 }, async (userReq) => {
  const { socket: userSocket, response } = Deno.upgradeWebSocket(userReq);

  const modelClient = new RealtimeClient({ apiKey: OPENAI_API_KEY });
  modelClient.updateSession({ instructions: getInstructions(USER_NAME) });
  modelClient.on("conversation.updated", (event: unknown) => {
    console.debug("on", "conversation.updated", "event", event);
    userSocket.send("OK");
  });
  await modelClient.connect();

  userSocket.addEventListener("message", async (event) => {
    const message = event.data;
    console.debug("modelClient.isConnected()", modelClient.isConnected());
    modelClient.sendUserMessageContent([{ type: "input_text", text: message }]);
  });

  return response;
});

Here's my own client output:

ws-client wscat -c ws://localhost:80
Connected (press CTRL+C to quit)
> 1
> 2
Disconnected (code: 1006, reason: "")

Here's the output from my server:

Listening on http://0.0.0.0:80/
Warning: Not implemented: ClientRequest.options.createConnection
modelClient.isConnected() true
modelClient.isConnected() false
error: Uncaught (in promise) Error: RealtimeAPI is not connected
    at RealtimeAPI.send (file:///***/node_modules/@openai/realtime-api-beta/lib/api.js:191:13)
    at RealtimeClient.sendUserMessageContent (file:///***/node_modules/@openai/realtime-api-beta/lib/client.js:554:21)
    at WebSocket.<anonymous> (file:///***/src/wsrt.ts:20:17)
    at innerInvokeEventListeners (ext:deno_web/02_event.js:754:7)
    at invokeEventListeners (ext:deno_web/02_event.js:801:5)
    at dispatch (ext:deno_web/02_event.js:658:9)
    at WebSocket.[[[eventLoop]]] (ext:deno_websocket/01_websocket.js:454:11)
    at eventLoopTick (ext:core/01_core.js:175:7)
Watcher Process failed. Restarting on file change...

Note the first isConnected() result is after message "1" while the second is after message "2".

I'm confused about some things

• Warning: Not implemented: ClientRequest.options.createConnection might indicate some fundamental incompatibility with Deno, since it's WebSocket implementation is not the same as Node's as far as I can tell (for example, it doesn't support a WS client setting auth headers)
• It was pretty hard to use this at all in Deno, which can add file dependencies from GitHub (as URLs) but not modules. You can see my workaround: import { RealtimeClient } from "../node_modules/@openai/realtime-api-beta/index.js";
• Why does the client lose its connection after sending 1 message?
• Why does the conversation.updated handler never fire?

[gegenschall]

I have the same problem but on Node, except it stays connected just fine. Switching on debug mode I can definitely see WebSocket events coming in but event handlers are just never triggered.

[elimydlarz]

Oh that's a bit discouraging - I was hoping improved Node compatibility in Deno 2 would help me out of this mess 😅 I really like the approach supported by this client - keen to use it.