Listens, but doesn't answer

xenorio commented 11 months ago

So I just installed ovos-docker using the official docs. Got everything except for the GUI. When I use the wakeword ("Hey Mycroft"), it is playing a "listening" sound, but never answers queries. Just stays completely silent.

Looking at the CLI (docker exec --interactive --tty ovos_cli ovos-cli-client) the queries seem to properly arrive, as in it understands what I'm saying, but it never generates an answer.

I can confirm that speech output works properly by running docker exec --interactive --tty ovos_cli ovos-speak "hello world"

It also seems like all containers that should be running are in fact running:

Although the ovos-core container seems to restart exactly every 60 seconds (no idea if that's intended)

In case it somehow matters, i'm running Arch Linux

goldyfruit commented 11 months ago

Hi @Xenorio

Thanks for the detailed issue! Could you please run this command to provides the logs from the ovos_core containers?

docker logs --tail 500 -f ovos_core

xenorio commented 11 months ago

Ah, seems to be a good ol' permission issue ^^

Logs are simply repeating this over and over again:

/usr/local/bin/entrypoint.sh: line 14: cd: /home/ovos/.local/share/mycroft/skills: No such file or directory
mkdir: cannot create directory '/home/ovos/.local/share/mycroft/skills': Permission denied

No idea why that would happen though. I am using the compose files from this repo, and did not mess with any permissions (or anything else for that matter) whatsoever.

I'd guess that entrypoint script is somehow running as the wrong user. Or the ovos user does not have write permissions for its own home dir, which would be even weirder.

Tried to have a look at the permission situation in that directory, but the restart loop is preventing this

goldyfruit commented 11 months ago

According to the first screenshot, you ran docker compose as root which is why you got the "Permissions denied".

First you need to restore the permissions on ~/.config/pulse and /run/user/1000/pulse on your host.

Second you will have to follow this: https://openvoiceos.github.io/ovos-docker/getting-started/docker/prerequisites/engine/#dont-be-root-be-a-user

Then you should be good to go :+1:

xenorio commented 11 months ago

Oh, my bad, skipped right to the installation instructions in the docs...

I am now running the containers as myself, added myself to the audio group, and confirmed that I own the pulse config files:

However, now it's not listening anymore. I'm not getting the "listening" sound, and pavucontrol does not list any recording stream.

It seems that now, the ovos_listener container is restart-looping and spitting this error at me:

2023-09-22 15:43:37.801 - voice - ovos_dinkum_listener.service:__init__:121 - INFO - Starting Voice Service
2023-09-22 15:43:37.816 - voice - ovos_dinkum_listener.service:on_alive:56 - INFO - DinkumVoiceService is alive.
2023-09-22 15:43:37.817 - voice - ovos_dinkum_listener.service:_before_start:292 - INFO - Starting service...
2023-09-22 15:43:37.828 - voice - ovos_bus_client.session:reset_default_session:536 - INFO - New Default Session Start: 58545c91-8053-45c2-97e0-8698fdada21a
2023-09-22 15:43:38.151 - voice - ovos_dinkum_listener.service:_connect_to_bus:391 - INFO - Connected to Mycroft Core message bus
2023-09-22 15:43:38.499 - voice - ovos_plugin_manager.vad:create:129 - ERROR - VAD plugin ovos-vad-plugin-silero could not be loaded!
Traceback (most recent call last):
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_plugin_manager/vad.py", line 127, in create
    return clazz(plugin_config)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 62, in __init__
    self.vad = SileroVoiceActivityDetector(model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 14, in __init__
    self.session = onnxruntime.InferenceSession(onnx_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 432, in __init__
    raise e
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 451, in _create_inference_session
    raise ValueError(
ValueError: This ORT build has ['AzureExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['AzureExecutionProvider', 'CPUExecutionProvider'], ...)
Traceback (most recent call last):
  File "/home/ovos/.venv/bin/ovos-dinkum-listener", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_dinkum_listener/__main__.py", line 24, in main
    service = OVOSDinkumVoiceService(on_ready=ready_hook, on_error=error_hook,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_dinkum_listener/service.py", line 148, in __init__
    self.vad = OVOSVADFactory.create()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_plugin_manager/vad.py", line 127, in create
    return clazz(plugin_config)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 62, in __init__
    self.vad = SileroVoiceActivityDetector(model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 14, in __init__
    self.session = onnxruntime.InferenceSession(onnx_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 432, in __init__
    raise e
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 451, in _create_inference_session
    raise ValueError(
ValueError: This ORT build has ['AzureExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['AzureExecutionProvider', 'CPUExecutionProvider'], ...)

No idea what "ORT", "InferenceSession", or anything else here is supposed to mean

xenorio commented 11 months ago

Oh, also, I can confirm that it is generating answers now by manually typing queries in the CLI. It's just not listening anymore.

goldyfruit commented 11 months ago

2023-09-22 15:43:37.801 - voice - ovos_dinkum_listener.service:__init__:121 - INFO - Starting Voice Service
2023-09-22 15:43:37.816 - voice - ovos_dinkum_listener.service:on_alive:56 - INFO - DinkumVoiceService is alive.
2023-09-22 15:43:37.817 - voice - ovos_dinkum_listener.service:_before_start:292 - INFO - Starting service...
2023-09-22 15:43:37.828 - voice - ovos_bus_client.session:reset_default_session:536 - INFO - New Default Session Start: 58545c91-8053-45c2-97e0-8698fdada21a
2023-09-22 15:43:38.151 - voice - ovos_dinkum_listener.service:_connect_to_bus:391 - INFO - Connected to Mycroft Core message bus
2023-09-22 15:43:38.499 - voice - ovos_plugin_manager.vad:create:129 - ERROR - VAD plugin ovos-vad-plugin-silero could not be loaded!
Traceback (most recent call last):
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_plugin_manager/vad.py", line 127, in create
    return clazz(plugin_config)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 62, in __init__
    self.vad = SileroVoiceActivityDetector(model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 14, in __init__
    self.session = onnxruntime.InferenceSession(onnx_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 432, in __init__
    raise e
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 451, in _create_inference_session
    raise ValueError(
ValueError: This ORT build has ['AzureExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['AzureExecutionProvider', 'CPUExecutionProvider'], ...)
Traceback (most recent call last):
  File "/home/ovos/.venv/bin/ovos-dinkum-listener", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_dinkum_listener/__main__.py", line 24, in main
    service = OVOSDinkumVoiceService(on_ready=ready_hook, on_error=error_hook,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_dinkum_listener/service.py", line 148, in __init__
    self.vad = OVOSVADFactory.create()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_plugin_manager/vad.py", line 127, in create
    return clazz(plugin_config)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 62, in __init__
    self.vad = SileroVoiceActivityDetector(model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_vad_plugin_silero/__init__.py", line 14, in __init__
    self.session = onnxruntime.InferenceSession(onnx_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 432, in __init__
    raise e
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/ovos/.venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 451, in _create_inference_session
    raise ValueError(
ValueError: This ORT build has ['AzureExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['AzureExecutionProvider', 'CPUExecutionProvider'], ...)

Your are facing this issue related to onnxruntime https://github.com/microsoft/onnxruntime/issues/17631

For now you can change the VAD plugin in your configuration file from ovos-vad-plugin-silero to ovos-vad-plugin-webrtcvad. See https://openvoiceos.github.io/ovos-docker/getting-started/docker/installation/configuration/#initial-configuration

Let me see what we can do for the silero plugin.

goldyfruit commented 11 months ago

I'm rebuilding the ovos_audio and ovos_listener images with onnxruntime version 1.15, you will not have to change your configuration.

goldyfruit commented 11 months ago

Both images are now rebuild, just re-run the same docker compose command you used initially.

xenorio commented 11 months ago

It did pull down new images, but it seems like something went majorly wrong here

ovos_listener log spams this now: /usr/local/bin/entrypoint.sh: line 32: ovos-dinkum-listener: command not found

goldyfruit commented 11 months ago

It did pull down new images, but it seems like something went majorly wrong here

ovos_listener log spams this now: /usr/local/bin/entrypoint.sh: line 32: ovos-dinkum-listener: command not found

Stupid me... https://github.com/OpenVoiceOS/ovos-docker/commit/f54d5f9ba7a23d56e9e72793aa432a4df3af4e30#diff-89244cc4e9ad80ff4874e6c162d530547cf9412ee031ebde53e9155ee6a3fa77L2

I'm rebuilding listener...

xenorio commented 11 months ago

Oh, well, happens to the best of us I guess 😅

goldyfruit commented 11 months ago

Oh, well, happens to the best of us I guess 😅

Yeah... Image is build, should work this time.

xenorio commented 11 months ago

Yes! Amazing! I can finally talk to my new buddy now! Thank you so much ❤️

However, one last teeny tiny issue... It's not playing the "listening" sound after issuing the wakeword anymore... Poked around a bit, and found this error in ovos_audio which is probably the culprit:

2023-09-22 18:16:35.788 - audio - ovos_utils.file_utils:resolve_resource_file:153 - WARNING - Deprecation version=0.1.0. Caller=ovos_audio.service:395. Expected a dict config and got None.
exception calling callback for <Future at 0x7f422c6b9090 state=finished raised FileNotFoundError>
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/home/ovos/.venv/lib/python3.11/site-packages/pyee/_executor.py", line 57, in _callback
    self.emit("error", exc)
  File "/home/ovos/.venv/lib/python3.11/site-packages/pyee/_base.py", line 118, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/home/ovos/.venv/lib/python3.11/site-packages/pyee/_base.py", line 88, in _emit_handle_potential_error
    raise error
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/service.py", line 427, in handle_instant_play
    audio_file = self._resolve_sound_uri(audio_file)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/service.py", line 397, in _resolve_sound_uri
    raise FileNotFoundError(f"{audio_file} does not exist")
FileNotFoundError: None does not exist

goldyfruit commented 11 months ago

Yes! Amazing! I can finally talk to my new buddy now! Thank you so much ❤️

Awesome!

@JarbasAl seems related to the changes of sound been played in ovos_audio container instead of ovos_listener.

@Xenorio do you have this in your mycroft.conf?

  "sounds": {
    "start_listening": "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/res/snd/start_listening.wav",
    "end_listening": "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/res/snd/end_listening.wav",
    "acknowledge": "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/res/snd/acknowledge.mp3"
  },

JarbasAl commented 11 months ago

  "sounds": {
    "start_listening": "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/res/snd/start_listening.wav",
    "end_listening": "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/res/snd/end_listening.wav",
    "acknowledge": "/home/ovos/.venv/lib/python3.11/site-packages/ovos_audio/res/snd/acknowledge.mp3"
  },

this should no longer be needed, ovos-audio now resolves snd/XXX (the default config) automatically for built in sounds

unfortunately the logs don't show the passed string so not sure what went wrong there exactly, the file path didn't seem to exist in ovos-audio container

xenorio commented 11 months ago

Looks like the docs need an update ^^

Originally used the config from https://openvoiceos.github.io/ovos-docker/getting-started/docker/installation/configuration/

{
  "play_wav_cmdline": "aplay %1",
  "lang": "en-us",
  "listener": {
    "VAD": {
      "module": "ovos-vad-plugin-silero"
    }
  },
  "sounds": {
    "start_listening": "/home/ovos/.venv/lib/python3.11/site-packages/ovos_dinkum_listener/res/snd/start_listening.wav"
  }
}

After modifying the start_listening field to the new path (site-packages/ovos_audio instead of site-packages/ovos_dinkum_listener) and adding the additional paths, the sound is back again

goldyfruit commented 11 months ago

Fixed https://github.com/OpenVoiceOS/ovos-docker/commit/962b21be0d3e3f2b87c9130069e9c9b48f0b8896

@Xenorio Enjoy!

xenorio commented 11 months ago

Thanks for your time and patience ❤️

Since everything seems to work as intended now, guess I can close this 😅

OpenVoiceOS / ovos-docker

Listens, but doesn't answer #40