search

OpenInterpreter / 01

The #1 open-source voice interface for desktop, mobile, and ESP32 chips.
https://01.openinterpreter.com/
GNU Affero General Public License v3.0
4.98k stars 533 forks source link

After the first message the voice does not work #307

Open RootMePLS opened 2 months ago

RootMePLS commented 2 months ago

Describe the bug The response is not returned to the mobile application(neither text nor voice). Here is copy-paste from the terminal.

> And what can you do?

  ●
> {START}

  I can assist with●

  I can assist with a variety of tasks, including web searches, calendar management, email and SMS messaging, file editing, and more. What do you need help with today?●

2024-09-15 15:03:17,633 - ERROR livekit.plugins.elevenlabs - 11labs reported an error: detected_unusual_activity {"pid": 58892, "job_id": "AJ_gWXL8sFCXMze"}

  I can assist with a variety of tasks, including web searches, calendar management, email and SMS messaging, file editing, and more. What do you need help with today?

  ●
                                                                                                                                                                                                                                                                                                                               2024-09-15 15:03:18,434 - ERROR livekit.plugins.elevenlabs - Error in _run
Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 259, in _run
    await self._run_ws(word_stream)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 364, in _run_ws
    await utils.aio.gracefully_cancel(*tasks)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 362, in _run_ws
    await asyncio.gather(*tasks)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 341, in recv_task
    raise Exception(
Exception: 11labs connection closed unexpectedly, not all tokens have been consumed {"pid": 58892, "job_id": "AJ_gWXL8sFCXMze"}
2024-09-15 15:03:18,438 - ERROR livekit.plugins.elevenlabs - Error in _main_task
Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 268, in _main_task
    await utils.aio.gracefully_cancel(*tasks)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 266, in _main_task
    await asyncio.gather(*tasks)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 259, in _run
    await self._run_ws(word_stream)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 364, in _run_ws
    await utils.aio.gracefully_cancel(*tasks)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 362, in _run_ws
    await asyncio.gather(*tasks)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 341, in recv_task
    raise Exception(
Exception: 11labs connection closed unexpectedly, not all tokens have been consumed {"pid": 58892, "job_id": "AJ_gWXL8sFCXMze"}
2024-09-15 15:03:18,439 - ERROR livekit.agents.voice_assistant - Error in _read_generated_audio_task
Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 236, in _read_generated_audio_task
    handle._tr_fwd.mark_audio_segment_end()
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/transcription/tts_forwarder.py", line 188, in mark_audio_segment_end
    self._forming_segments.audio = self._forming_segments.q[0]

  I can assist with a variety of tasks, including web searches, calendar management, email and SMS messaging, file editing, and more. What do you need help with today?

  Let●
                                                                                                                                                                                                                                                                                                                               2024-09-15 15:03:18,758 - ERROR livekit.agents.voice_assistant - Error in _stream_synthesis_task
Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 251, in _stream_synthesis_task
    tts_stream.push_text(seg)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 107, in push_text
    self._check_not_closed()
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 130, in _check_not_closed
    raise RuntimeError(f"{cls.__module__}.{cls.__name__} is closed")
RuntimeError: livekit.plugins.elevenlabs.tts.SynthesizeStream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 262, in _stream_synthesis_task
    await utils.aio.gracefully_cancel(read_atask)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 236, in _read_generated_audio_task
    handle._tr_fwd.mark_audio_segment_end()
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/transcription/tts_forwarder.py", line 188, in mark_audio_segment_end
    self._forming_segments.audio = self._forming_segments.q[0]
                                   ~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: deque index out of range {"pid": 58892, "job_id": "AJ_gWXL8sFCXMze"}
2024-09-15 15:03:18,760 - ERROR livekit.agents.voice_assistant - Error in _synthesize_task
Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 251, in _stream_synthesis_task
    tts_stream.push_text(seg)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 107, in push_text
    self._check_not_closed()
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 130, in _check_not_closed
    raise RuntimeError(f"{cls.__module__}.{cls.__name__} is closed")
RuntimeError: livekit.plugins.elevenlabs.tts.SynthesizeStream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 170, in _synthesize_task
    await utils.aio.gracefully_cancel(synth)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 262, in _stream_synthesis_task
    await utils.aio.gracefully_cancel(read_atask)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 236, in _read_generated_audio_task
    handle._tr_fwd.mark_audio_segment_end()

  I can assist with a variety of tasks, including web searches, calendar management, email and SMS messaging, file editing, and more. What do you need help with today?

  Let me●
                                                                                                                                                                                                                                                                                                                               2024-09-15 15:03:18,762 - ERROR asyncio - Task exception was never retrieved
future: <Task finished name='Task-33' coro=<AgentOutput._synthesize_task() done, defined at /Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py:13> exception=IndexError('deque index out of range')>
Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 251, in _stream_synthesis_task
    tts_stream.push_text(seg)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 107, in push_text
    self._check_not_closed()
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 130, in _check_not_closed
    raise RuntimeError(f"{cls.__module__}.{cls.__name__} is closed")
RuntimeError: livekit.plugins.elevenlabs.tts.SynthesizeStream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 170, in _synthesize_task
    await utils.aio.gracefully_cancel(synth)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 262, in _stream_synthesis_task
    await utils.aio.gracefully_cancel(read_atask)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/aio/__init__.py", line 16, in gracefully_cancel
    await asyncio.gather(*futures)
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dm/Library/Caches/pypoetry/virtualenvs/01-jZug3wYD-py3.11/lib/python3.11/site-packages/livekit/agents/voice_assistant/agent_output.py", line 236, in _read_generated_audio_task
    handle._tr_fwd.mark_audio_segment_end()

  I can assist with a variety of tasks, including web searches, calendar management, email and SMS messaging, file editing, and more. What do you need help with today?

  Let me know what you'd like to do next.

To Reproduce

Expected behavior The mobile application responds with voice, there are no errors in the console.

Desktop (please complete the following information):

MikeBirdTech commented 2 months ago

@RootMePLS Do you have a free or paid plan for elevenlabs?

RootMePLS commented 1 month ago

@RootMePLS Do you have a free or paid plan for elevenlabs?

Free

markojak commented 1 month ago

Having the same problem. I see that Elevenlabs has given me 10,000 "Credits" to start off with. so not sure why this is happening.

markojak commented 1 month ago

I upgraded to paid and the same thing happens

Exception: 11labs connection closed unexpectedly, not all tokens have been consumed {"pid": 3816, "job_id": "AJ_fRgjhVSVrUBy"} 2024-09-25 22:53:30,581 - ERROR livekit.plugins.elevenlabs - Error in _main_task Traceback (most recent call last):