search

Romkabouter / ESP32-Rhasspy-Satellite

The repo has implementing an esp32 standalone MQTT audio streamer. Is is desinged to work as a satellite for Rhasspy (https://rhasspy.readthedocs.io/en/latest/). It supports multiple devices
GNU General Public License v3.0
358 stars 64 forks source link

More advanced LED patterns #51

Closed markusressel closed 3 years ago

markusressel commented 3 years ago

It would be really awesome if ESP32-Rhasspy-Satellite would support custom wake/sleep/error led patterns/animations, like Google or Alexa, possibly even with direction of arrival support lateron.

When playing around on a Matrix Voice I noticed that the LED color change is only in effect after the "wake" audio file has played. I tried to change this, since it seems unintuitive that, even though the wakeword has already been detected and the acknowledgment sound is already playing, the LEDs to not reflect this. Looking at the code this is due to the fact that the HotwordDetectedEvent is triggered after the sound file has been played. Is this part of the original hermes protocol? Or is this something that might be improved somehow?

The next step would be to allow LED animations, and then allow the acknowledgment sound and the LED animation to play in parallel, so they do not block each other.

Any hints whether this is achievable would be very much appreciated :nerd_face:

Romkabouter commented 3 years ago

Looking at the code this is due to the fact that the HotwordDetectedEvent is triggered after the sound file has been played. Is this part of the original hermes protocol? Or is this something that might be improved somehow?

The actual issue is that the message for playing audio arrives probably before the hotword/toggle message. It would not really be an issue, but the Matrix Voice cannot play audio and update leds at the same time. That is because it uses a "wishbonebus" which reads and writes. When doing this at the same time, it crashes. My solution was to exlude this by using a semaphore, but it might be possible reading and writing at the same time.

Animations are possible, but as I said there is an issue with the wishbone bus regarding parallel read/write which I was not yet able to solve, if it is even possible to solve.

markusressel commented 3 years ago

I have started working on making it possible to play an animation instead of setting a simple color, and this already works fine. However, I am not very experienced with programming for microcontrollers and I am having a difficult time finding a proper way to be able to process things async at all. I am not sure whether the ESP32 supports real threading at all, and even if it does, using something like coroutines / asyncio could be the more robust approach, given that the wishbonebus is synchronous anyway. Could you give me some links/hints on what you have used when you tried to do this?

Romkabouter commented 3 years ago

I encountered the problem because I was trying to read and write at the same time. My solution was to avoid this and implement a semaphore, so I don't have links of what I did sadly. Me nog being able to solve the parallel read/write might also be my limitation and not a whisbone limitation ;)

Romkabouter commented 3 years ago

WIP here:

https://github.com/Romkabouter/ESP32-Rhasspy-Satellite/issues/69