Multimodal Agent with complete API

This should essentially make both the core package and the OpenAI plugin feature-complete. Node agents is now capable of powering the realtime playground.

[x] Rename OmniAssistant to MultimodalAgent
[x] Move playout and MMA to multimodal namespace in main package
[x] Use inputAudioBuffer for mic input
[x] Restore function calling implementation
[x] Expose the session so users may inject conversation items
[x] Implement AgentState attribute
[x] Fix interruption handling by properly cancelling playout tasks (involved the introduction of a CancellablePromise and gracefullyCancel util to mirror asyncio)
[x] Change all private vars/functions to use #
[x] Fix wait for participant bugs (also in https://github.com/livekit/agents-js/pull/75)
[x] Forward initial … transcription on speech start
[x] replace a bunch of promises with futures (and add a done flag to Future)
[x] Create an AsyncIterableQueue and use it for textStream and audioStream instead of Queue
[x] Fix VAD none type
[x] Reduce repetition in type lists

Note on the "main package" move:

I spent a while trying to make circular dependencies work but it ended up being quite difficult and is also explicitly incompatible with turbo.

I ultimately decided that we should punt work on that for now and instead just dropped a couple of abstract top-level types and any for some of the underlying stuff and called it a day. The any type is exposed to customers only when they acquire the session, which requires a simple as openai.realtime.RealtimeSession.

The only other downside to this approach is I think it precludes the use of TypedEmitter on RealtimeSession, because then we'd have to define all the messages it produces and their signatures in the main abstract class, which felt like too much.

The other options would be to leave MMA in the OpenAI plugin or get rid of the OpenAI plugin and make it part of the main package. Both options felt worse to me.

Final note: The names of some new emitted events and the base abstract class are subject to change, pending @theomonnom 's final choices in Python.

livekit / agents-js

Multimodal Agent with complete API #74

🦋 Changeset detected