livekit / agents

Build real-time multimodal AI applications 🤖🎙️📹
https://docs.livekit.io/agents
Apache License 2.0
3.67k stars 360 forks source link

I can't get the TTS 11labs pluguin to work #921

Open llllCharliellll opened 1 week ago

llllCharliellll commented 1 week ago

When I use OpenAI TTS there is no problem, but when I use ElevenLabs TTS, it says the first message but then, I get this message in the terminal: {"message": "11labs reported an error: detected_unusual_activity", "level": "ERROR", "name": "livekit.plugins.elevenlabs"}.

import asyncio from dotenv import load_dotenv from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm from livekit.agents.voice_assistant import VoiceAssistant from livekit.plugins import openai, silero, elevenlabs from livekit import rtc from api import AssistantFnc import logging

Cargar las variables de entorno

load_dotenv()

logger = logging.getLogger("elevenlabs-tts-assistant") logger.setLevel(logging.INFO)

def _text_to_chunks(text: str) -> list[str]: """Dividir el texto en fragmentos más pequeños para el streaming.""" sizes = [2, 3, 4] chunks, i = [], 0

for size in sizes:
    while i + size <= len(text):
        chunks.append(text[i : i + size])
        i += size

chunks.append(text[i:])  # resto del texto
return chunks

async def _playout_task(playout_q: asyncio.Queue, audio_source: rtc.AudioSource) -> None: """Reproducir los frames de audio de la cola.""" while True: frame = await playout_q.get() if frame is None: break await audio_source.capture_frame(frame)

async def entrypoint(ctx: JobContext): initial_ctx = llm.ChatContext().append( role="system", text=( "Eres un asistente de voz. Tu interacción con los usuarios será a través de la voz. Debes utilizar respuestas cortas y concisas, y evitar el uso de puntuación que no se pueda pronunciar." ), ) await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY) fnc_ctx = AssistantFnc()

# Usar ElevenLabs TTS con el ID de la voz y el modelo
voice = elevenlabs.Voice(
    id="UOIqAnmS11Reiei1Ytkc", name="Carolina", category="premade"
)
tts_11labs = elevenlabs.TTS(model_id="eleven_multilingual_v2", voice=voice)

# Inicializar el asistente de voz con la voz de ElevenLabs
assistant = VoiceAssistant(
    vad=silero.VAD.load(),
    stt=openai.STT(),  # Usar OpenAI para Speech-to-Text
    llm=openai.LLM(),  # Usar OpenAI para el modelo de lenguaje
    tts=tts_11labs,  # Usar ElevenLabs TTS
    chat_ctx=initial_ctx,
    fnc_ctx=fnc_ctx,
)

assistant.start(ctx.room)
await asyncio.sleep(1)
await assistant.say("Hey, como puedo ayudarte hoy!", allow_interruptions=True)

if name == "main": cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

GabrielGalatti commented 1 week ago

Is your 11labs account free tier? I'm facing the same error with a free tier account.

llllCharliellll commented 1 week ago

Yes, but in the free plan, I thin you are also supposed to have full access to the API.

GabrielGalatti commented 6 days ago

Yes, but in the free plan, I thin you are also supposed to have full access to the API.

I have done it work, the solution is in this issue: #936