search

livekit / agents

Build real-time multimodal AI applications 🤖🎙️📹
https://docs.livekit.io/agents
Apache License 2.0
1.03k stars 198 forks source link

Stream Manipulation using Custom Plugins #385

Open parshvadaftari opened 2 months ago

parshvadaftari commented 2 months ago

Due to lack of Documentation in plugins section, I'm not sure how to build one let alone how to customise the already existing one. I want manipulate the input and output stream using the custom plugins as descirbed in the documentation but don't know how to do so. It just says:

The plugin framework is designed to be extensible, allowing anyone to build their own plugin. Your plugin can integrate with various providers or directly load models for local inference.

By adopting the standard STT or TTS interfaces, you can abstract away implementation specifics and simplify switching between different providers in your agent code. https://docs.livekit.io/agents/plugins/#Building-your-own

ChrisFeldmeier commented 1 month ago

I also don't know, but look at these plugins and try to adapt it / rewrite it? https://github.com/livekit/agents/tree/main/livekit-plugins

keepingitneil commented 1 month ago

Hey @parshvadaftari - custom behavior can be achieved by subclassing one of our base classes (STT, TTS, or LLM). For example, the openai LLM is implemented in this file: https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/llm.py

Which is a subclass of this base class: https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/llm/llm.py

It sounds like you aren't necessarily trying to create a brand new plugin, but rather a wrapper plugin that can "map" the output of an existing plugin. Is that correct?

parshvadaftari commented 1 month ago

Hey @keepingitneil , I tried making my own plugin, but failed and used a different approach of making changes in the https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/llm.py file and manipulating the output using my requirements.

Is there any simply way to do the same? Like you mentioned about the custom plugin, any code snippet or any example of it would be helpfull, example which is not on the website.

parshvadaftari commented 1 month ago

I closed the issue by mistake, apologies for it.