Closed flatsiedatsie closed 2 years ago
Romkabouter
commented
3 years ago What is the samplerate of voco? This software output 16000 16bit mono, each MQTT messsage containt 512 bytes wavedata and 44 but header, a total of 556 bytes per message.
And this software does not support snips anymore, which version of snips does voco use?
I can see if I can get it to work with voco, but I won't put that in master due to the fact that snips is not supported. I will create a new branch for it
flatsiedatsie
commented
3 years ago That would rock! I was looking into the header to see if I had to change something there.
As far as I know, Voco uses the latest version of Snips.
Does Rhasspy use different audio headers from that last version of Snips?
The mystery is: why does it sometimes actually work?
Romkabouter
commented
3 years ago Where can I find this "VoiceActivity" logs on such?
The audio produced is just a lot of small wave files. Rhasspy does not differ from snips with regards to that. But the chunksize from Voco is different than what is send by the streamer
This is the wave format: http://soundfile.sapp.org/doc/WaveFormat/ From Voco: RIFF4WAVEfmt From streamer: RIFF,WAVEfmt
The 4 and , are not a 4 and a , but represent the 4 bytes of the ChunkSize. So if the chunk size in not the same then depending on how Voco is implemented, it might deal with the audio unexpectedly
Romkabouter
commented
3 years ago I will use this branch https://github.com/Romkabouter/ESP32-Rhasspy-Satellite/tree/voco
flatsiedatsie
commented
3 years ago The 4 and , are not a 4 and a , but represent the 4 bytes of the ChunkSize.
Ah, interesting. I figured it might matter. I also wonder if the tim part was some kind of time index.
To debug, I use these commands:
Snips Watch: ~/.webthings/addons/voco/snips/snips-watch -vvvv
Looking at ALL MQTT traffic: mosquitto_sub -v -h 192.168.2.167 -t 'hermes/#'
(mosquitto sub isn't installed by default, so sudo apt-get install mosquitto-clients will fix that)
And if you need to quickly restart the gateway for some reason: sudo systemctl restart webthings-gateway.service
Romkabouter
commented
3 years ago Ah, interesting. I figured it might matter. I also wonder if the
timpart was some kind of time index.
Yes, when I see this: hermes/audioServer/azrxidia/replayResponse RIFF^WAVEfmt ?>}tim??wrpidazrxidia-1614003432719rprf that does not really look like a good waveheader to me, it should somewhere contain the plain letter data as well The link I posted was how a wave format header should look like, I do not know how the replayRespoonse is generated.
In the streamer I use a fixed header, because every wav audio send is the same length and format :) I will ignore the replayResponse for now and will dive into voco a bit to find out what voco expects.
flatsiedatsie
commented
3 years ago Yes I've only seen that replayRequest command once. It's not documented either, all I could find were two references to Snips code on Github. I'd just ignore it.
that does not really look like a good waveheader to me
0_0 :-)
Romkabouter
commented
3 years ago If have tried a bare snips with the demo assistant with a mic attached to the pi, I get timeouts:
I think Snips is broken and this is actually causing the timeouts, I will try this setup with voco as well.
flatsiedatsie
commented
3 years ago AI, that doesn't sound good... Is that with a different wav encoding? Are you using the 6.0 version?
Romkabouter
commented
3 years ago No, this is Snips installed on a Pi with an attached mic. Nothing to do with this code. I wanted to see which message are going back and forth to snips to see what is needed for a correct Dialogue session.
That is why I installed the demo assistant. I followed https://docs.snips.ai/articles/raspberrypi/manual-setup, which is old but still the latest version :)
As you can see, the hotword is detected ok, I also tested the mic outside snips. All works. But I still get a timeout, it should detect the weather intent
ok edit, no ASR is running ;)
flatsiedatsie
commented
3 years ago Keeping my fingers crossed over here :-)
Romkabouter
commented
3 years ago I followed this step and now snips is working :)
Romkabouter
commented
3 years ago progress :)
flatsiedatsie
commented
3 years ago Hey wow!! Great work!
What's still left to do? It looks to me like a 100% succes?
Romkabouter
commented
3 years ago This was a test with a Pi with a build-in mike. To verify that snips still works.
The next step for me is to check and adjust the code for the streamer to get Snips going (again). When that works, I can move on to Voco :) First with a build-in mike and if that works the streamer. So there is a road ahead :)
flatsiedatsie
commented
3 years ago Ah I see. Are you sure the effort is worth it? We could just wait for Voco to move to Rhasspy. That has to happen at some point anyway.
Romkabouter
commented
3 years ago Well, it would be nice to find what the issue is, but that is just me :D
I see in the messages RIFF4, as you also already found. The bytes send per message is also 572 instead of the 556 that the streamer sends. I believe this is the root cause of the bad performance, the wave audio does not match.
flatsiedatsie
commented
3 years ago That's what I figured as well.
Romkabouter
commented
3 years ago Romkabouter
commented
3 years ago Sorry about my lack of commitment, my sd card died so I had to start over which I did not have time for. This is still on my to-do list however :)
flatsiedatsie
commented
3 years ago No worries. Pretty busy overhere as well :-)
Romkabouter
commented
3 years ago I have the code working with snips, however there are some header changes which I can't get right atm. When hey snips is spoken, it works if you speak it a bit slower, so it is most likely a wave format thing. I have recorder an audio file which triggers the hotword 100% with the code. I will attach that to the branch :)
flatsiedatsie
commented
3 years ago That's great news! I'm going to check this out asap!
Romkabouter
commented
3 years ago This is the result of the audio file
After the hotword it picks up the audio as expected
The timeout occurs because I have no intent handler.
flatsiedatsie
commented
3 years ago Where can I find the code? I had a look at the voco branch, but that doesn't seem to be it?
Romkabouter
commented
3 years ago Just pushed it. I have also included a record.py which records a stream for a couple of seconds. I was experimenting with the header, but found the I did not have to make a change strange enough.
Just pushed branch works with met Atom Echo, but the recordlevel seems to be very low. This will also be the case with the master branch I assume
flatsiedatsie
commented
3 years ago Cool, I will try it now!
flatsiedatsie
commented
3 years ago It took a bit of work to be able to upload it via the Arduino IDE again. I had to strip out the LED parts.
Now that it uploads, I get this error. Nothing to worry about, I just have to look into it.
22:17:17.029 -> Creating I2Stask
22:17:17.029 -> Enter WifiDisconnected
22:17:17.029 -> Total heap: 293476
22:17:17.029 -> Free heap: 231364
22:17:19.401 -> Enter WifiConnected
22:17:19.401 -> Connected to Wifi with IP: 192.168.2.137, SSID: mywifiname, BSSID: B0:95:75:9F:FE:CF, RSSI: -49
22:17:19.438 -> Enter MQTTDisconnected
22:17:19.438 -> Connecting MQTT: 192.168.2.166, 1883
22:17:19.475 -> end of setupEnter MQTTConnected
22:17:19.586 -> Connected as atomecho
22:17:19.586 -> Enter Idle
22:17:19.586 -> Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
22:17:19.586 -> Core 0 register dump:
22:17:19.586 -> PC : 0x4015618e PS : 0x00060b30 A0 : 0x800d11ec A1 : 0x3ffde570
22:17:19.586 -> A2 : 0x00000000 A3 : 0x3ffde5e0 A4 : 0x00000200 A5 : 0x3ffde5ac
22:17:19.623 -> A6 : 0x00000064 A7 : 0x00000000 A8 : 0x00000001 A9 : 0x0000000b
22:17:19.623 -> A10 : 0x3ffb96ac A11 : 0x3ffde58f A12 : 0x00000000 A13 : 0x00000008
22:17:19.623 -> A14 : 0x00060b23 A15 : 0x00000000 SAR : 0x00000000 EXCCAUSE: 0x0000001c
22:17:19.623 -> EXCVADDR: 0x00000014 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000
22:17:19.660 ->
22:17:19.660 -> ELF file SHA256: 0000000000000000
22:17:19.660 ->
22:17:19.660 -> Backtrace: 0x4015618e:0x3ffde570 0x400d11e9:0x3ffde5a0 0x400d2174:0x3ffde5d0 0x40089bce:0x3ffdea50
22:17:19.660 ->
22:17:19.660 -> Rebooting...Got a bit further.
Should I change some of the settings to get it to continuously stream audio? I'm assuming I should not use hotward detection.
The Connected as null is a bit strange. I tried adding siteid to the config file, but that didn't seem to solve it.
22:41:39.027 -> Rebooting...
22:41:39.027 -> ets Jun 8 2016 00:22:57
22:41:39.027 ->
22:41:39.027 -> rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
22:41:39.027 -> configsip: 188777542, SPIWP:0xee
22:41:39.027 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
22:41:39.027 -> mode:DIO, clock div:1
22:41:39.027 -> load:0x3fff0018,len:4
22:41:39.061 -> load:0x3fff001c,len:1044
22:41:39.061 -> load:0x40078000,len:10124
22:41:39.061 -> load:0x40080400,len:5856
22:41:39.061 -> entry 0x400806a8
22:41:39.386 -> Boo⸮M5Atom initializing...Loading configuration
22:41:39.574 -> {
22:41:39.574 -> "mqtt_host": "192.168.2.166",
22:41:39.574 -> "mqtt_port": 1883,
22:41:39.574 -> "mqtt_user": "",
22:41:39.574 -> "mqtt_pass": "",
22:41:39.574 -> "sideid": "atomecho",
22:41:39.574 -> "mqtt_valid": true,
22:41:39.574 -> "mute_input": false,
22:41:39.574 -> "mute_output": false,
22:41:39.574 -> "amp_output": 0,
22:41:39.574 -> "brightness": 30,
22:41:39.574 -> "hotword_brightness": 100,
22:41:39.574 -> "hotword_detection": 0,
22:41:39.574 -> "volume": 100,
22:41:39.611 -> "gain": 5
22:41:39.611 -> }
22:41:39.611 -> Creating I2Stask
22:41:39.611 -> Enter WifiDisconnected
22:41:39.611 -> Total heap: 293084
22:41:39.611 -> Free heap: 230916
22:41:43.910 -> Enter WifiConnected
22:41:43.910 -> Connected to Wifi with IP: 192.168.2.137, SSID: sterrenkijker_nomap, BSSID: B0:95:75:9F:FE:CF, RSSI: -50
22:41:43.945 -> Enter MQTTDisconnected
22:41:43.945 -> Connecting MQTT: 192.168.2.166, 1883
22:41:43.945 -> end of setupConnect failed, retry
22:41:48.932 -> Audio connected: 1, Async connected: 0
22:41:48.932 -> Enter MQTTDisconnected
22:41:48.932 -> Connecting MQTT: 192.168.2.166, 1883
22:42:20.105 -> Connect failed, retry
22:42:20.105 -> Audio connected: 0, Async connected: 0
22:42:20.105 -> Enter MQTTDisconnected
22:42:20.105 -> Connecting MQTT: 192.168.2.166, 1883
22:42:38.203 -> Connect failed, retry
22:42:38.203 -> Audio connected: 0, Async connected: 0
22:42:38.203 -> Enter MQTTDisconnected
22:42:38.240 -> Connecting MQTT: 192.168.2.166, 1883
22:42:56.724 -> Connect failed, retry
22:42:56.724 -> Audio connected: 0, Async connected: 0
22:42:56.724 -> Enter MQTTDisconnected
22:42:56.724 -> Connecting MQTT: 192.168.2.166, 1883
22:43:15.229 -> Connect failed, retry
22:43:15.229 -> Audio connected: 0, Async connected: 0
22:43:15.229 -> Enter MQTTDisconnected
22:43:15.229 -> Connecting MQTT: 192.168.2.166, 1883
22:43:33.736 -> Connect failed, retry
22:43:33.736 -> Audio connected: 0, Async connected: 0
22:43:33.772 -> Enter MQTTDisconnected
22:43:33.772 -> Connecting MQTT: 192.168.2.166, 1883
22:43:52.241 -> Connect failed, retry
22:43:52.279 -> Audio connected: 0, Async connected: 0
22:43:52.279 -> Enter MQTTDisconnected
22:43:52.279 -> Connecting MQTT: 192.168.2.166, 1883
22:44:02.380 -> Connect failed, retry
22:44:02.380 -> Audio connected: 1, Async connected: 0
22:44:02.380 -> Enter MQTTDisconnected
22:44:02.380 -> Connecting MQTT: 192.168.2.166, 1883
22:44:02.492 -> Enter MQTTConnected
22:44:02.492 -> Connected as null
22:44:02.492 -> going from mqtt connected to idle
22:44:02.492 -> Enter Idle
22:44:02.492 -> still in idle
22:44:02.492 -> end of idle
22:44:02.492 -> Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
22:44:02.492 -> Core 0 register dump:
22:44:02.492 -> PC : 0x40157776 PS : 0x00060d30 A0 : 0x800d161c A1 : 0x3ffde570
22:44:02.492 -> A2 : 0x00000000 A3 : 0x3ffde5e0 A4 : 0x00000200 A5 : 0x3ffde5ac
22:44:02.525 -> A6 : 0x00000064 A7 : 0x00000000 A8 : 0x0000000b A9 : 0x00000068
22:44:02.525 -> A10 : 0x3ffb20d0 A11 : 0x3ffde58f A12 : 0x00000000 A13 : 0x00000008
22:44:02.525 -> A14 : 0x00060d23 A15 : 0x00000000 SAR : 0x00000000 EXCCAUSE: 0x0000001c
22:44:02.525 -> EXCVADDR: 0x00000014 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000
22:44:02.525 ->
22:44:02.560 -> ELF file SHA256: 0000000000000000
22:44:02.560 ->
In case you're curious, I've uploaded my current code here. I will look into it more this weekend. https://github.com/flatsiedatsie/voco_mini_sat
Romkabouter
commented
3 years ago Should I change some of the settings to get it to continuously stream audio?
Yes, the connected as null seems to imply that the config.siteid is empty. You should be able to browse to 192.168.2.137 and check the siteid.
Romkabouter
commented
3 years ago You have "sideid": "atomecho", in your config, that is why it is not filled most probably. Should be siteid in this file: https://github.com/flatsiedatsie/voco_mini_sat/blob/main/data/config.json
That setting is everywhere, so it might also be the cause of the crashes, but I cannot be sure about that
flatsiedatsie
commented
3 years ago You have "sideid": "atomecho", in your config, that is why it is not filled most probably.
I added that after I noticed that the value was null.
I didn't realise I should visit the device's webpage.
flatsiedatsie
commented
3 years ago Ah, I now see I made a type in the config file. Thanks.
This is where I'm at now. It connects to wifi, I can open the server config page (cool). It says audio is connected, but MQTT isn't. When it does suddenly connect, it crashes.
07:57:27.321 -> Boo�M5Atom initializing...Loading configuration
07:57:27.541 -> {
07:57:27.541 -> "siteid": "ATOMECHO",
07:57:27.541 -> "mqtt_host": "192.168.2.166",
07:57:27.541 -> "mqtt_port": 1883,
07:57:27.541 -> "mqtt_user": "",
07:57:27.541 -> "mqtt_pass": "",
07:57:27.541 -> "mqtt_valid": true,
07:57:27.541 -> "mute_input": false,
07:57:27.541 -> "mute_output": false,
07:57:27.541 -> "amp_output": 0,
07:57:27.541 -> "brightness": 30,
07:57:27.541 -> "hotword_brightness": 100,
07:57:27.541 -> "hotword_detection": 0,
07:57:27.541 -> "volume": 100,
07:57:27.541 -> "gain": 5
07:57:27.541 -> }
07:57:27.541 -> Creating I2Stask
07:57:27.541 -> Enter WifiDisconnected
07:57:27.541 -> Total heap: 293072
07:57:27.577 -> Free heap: 230748
07:57:32.604 -> Enter WifiConnected
07:57:32.604 -> Connected to Wifi with IP: 192.168.2.137, SSID: sterrenkijker_nomap, BSSID: B0:95:75:9F:FE:CF, RSSI: -58
07:57:32.641 -> Enter MQTTDisconnected
07:57:32.641 -> Connecting MQTT: 192.168.2.166, 1883
07:57:32.678 -> end of setupConnect failed, retry
07:57:37.604 -> Audio connected: 0, Async connected: 0
07:57:37.641 -> Enter MQTTDisconnected
07:57:37.641 -> Connecting MQTT: 192.168.2.166, 1883
07:57:55.890 -> Connect failed, retry
07:57:55.890 -> Audio connected: 0, Async connected: 0
07:57:55.890 -> Enter MQTTDisconnected
07:57:55.890 -> Connecting MQTT: 192.168.2.166, 1883
07:58:14.407 -> Connect failed, retry
07:58:14.407 -> Audio connected: 0, Async connected: 0
07:58:14.407 -> Enter MQTTDisconnected
07:58:14.407 -> Connecting MQTT: 192.168.2.166, 1883
07:58:22.184 -> Connect failed, retry
07:58:22.184 -> Audio connected: 1, Async connected: 0
07:58:22.184 -> Enter MQTTDisconnected
07:58:22.184 -> Connecting MQTT: 192.168.2.166, 1883
07:58:28.027 -> Connect failed, retry
07:58:28.027 -> Audio connected: 1, Async connected: 0
07:58:28.027 -> Enter MQTTDisconnected
07:58:28.027 -> Connecting MQTT: 192.168.2.166, 1883
07:58:33.045 -> Connect failed, retry
07:58:33.045 -> Audio connected: 1, Async connected: 0
07:58:33.045 -> Enter MQTTDisconnected
07:58:33.045 -> Connecting MQTT: 192.168.2.166, 1883
07:58:38.026 -> Connect failed, retry
07:58:38.026 -> Audio connected: 1, Async connected: 0
07:58:38.026 -> Enter MQTTDisconnected
07:58:38.026 -> Connecting MQTT: 192.168.2.166, 1883
07:58:43.031 -> Connect failed, retry
07:58:43.031 -> Audio connected: 1, Async connected: 0
07:58:43.031 -> Enter MQTTDisconnected
07:58:43.031 -> Connecting MQTT: 192.168.2.166, 1883
07:58:48.044 -> Connect failed, retry
07:58:48.044 -> Audio connected: 1, Async connected: 0
07:58:48.044 -> Enter MQTTDisconnected
07:58:48.044 -> Connecting MQTT: 192.168.2.166, 1883
07:58:53.599 -> Connect failed, retry
07:58:53.599 -> Audio connected: 1, Async connected: 0
07:58:53.599 -> Enter MQTTDisconnected
07:58:53.599 -> Connecting MQTT: 192.168.2.166, 1883
07:58:58.593 -> Connect failed, retry
07:58:58.593 -> Audio connected: 1, Async connected: 0
07:58:58.593 -> Enter MQTTDisconnected
07:58:58.629 -> Connecting MQTT: 192.168.2.166, 1883
07:59:03.593 -> Connect failed, retry
07:59:03.593 -> Audio connected: 1, Async connected: 0
07:59:03.631 -> Enter MQTTDisconnected
07:59:03.631 -> Connecting MQTT: 192.168.2.166, 1883
07:59:08.602 -> Connect failed, retry
07:59:08.602 -> Audio connected: 1, Async connected: 0
07:59:08.602 -> Enter MQTTDisconnected
07:59:08.602 -> Connecting MQTT: 192.168.2.166, 1883
07:59:13.614 -> Connect failed, retry
07:59:13.614 -> Audio connected: 1, Async connected: 0
07:59:13.614 -> Enter MQTTDisconnected
07:59:13.614 -> Connecting MQTT: 192.168.2.166, 1883
07:59:18.606 -> Connect failed, retry
07:59:18.606 -> Audio connected: 1, Async connected: 0
07:59:18.606 -> Enter MQTTDisconnected
07:59:18.606 -> Connecting MQTT: 192.168.2.166, 1883
07:59:23.610 -> Connect failed, retry
07:59:23.610 -> Audio connected: 1, Async connected: 0
07:59:23.610 -> Enter MQTTDisconnected
07:59:23.610 -> Connecting MQTT: 192.168.2.166, 1883
07:59:28.628 -> Connect failed, retry
07:59:28.628 -> Audio connected: 1, Async connected: 0
07:59:28.628 -> Enter MQTTDisconnected
07:59:28.628 -> Connecting MQTT: 192.168.2.166, 1883
07:59:33.618 -> Connect failed, retry
07:59:33.618 -> Audio connected: 1, Async connected: 0
07:59:33.618 -> Enter MQTTDisconnected
07:59:33.618 -> Connecting MQTT: 192.168.2.166, 1883
07:59:38.617 -> Connect failed, retry
07:59:38.617 -> Audio connected: 1, Async connected: 0
07:59:38.617 -> Enter MQTTDisconnected
07:59:38.617 -> Connecting MQTT: 192.168.2.166, 1883
07:59:43.615 -> Connect failed, retry
07:59:43.615 -> Audio connected: 1, Async connected: 0
07:59:43.615 -> Enter MQTTDisconnected
07:59:43.615 -> Connecting MQTT: 192.168.2.166, 1883
07:59:48.624 -> Connect failed, retry
07:59:48.624 -> Audio connected: 1, Async connected: 0
07:59:48.624 -> Enter MQTTDisconnected
07:59:48.624 -> Connecting MQTT: 192.168.2.166, 1883
07:59:53.618 -> Connect failed, retry
07:59:53.618 -> Audio connected: 1, Async connected: 0
07:59:53.618 -> Enter MQTTDisconnected
07:59:53.618 -> Connecting MQTT: 192.168.2.166, 1883
07:59:58.612 -> Connect failed, retry
07:59:58.612 -> Audio connected: 1, Async connected: 0
07:59:58.612 -> Enter MQTTDisconnected
07:59:58.612 -> Connecting MQTT: 192.168.2.166, 1883
08:00:03.612 -> Connect failed, retry
08:00:03.612 -> Audio connected: 1, Async connected: 0
08:00:03.612 -> Enter MQTTDisconnected
08:00:03.612 -> Connecting MQTT: 192.168.2.166, 1883
08:00:08.639 -> Connect failed, retry
08:00:08.639 -> Audio connected: 1, Async connected: 0
08:00:08.639 -> Enter MQTTDisconnected
08:00:08.639 -> Connecting MQTT: 192.168.2.166, 1883
08:00:13.608 -> Connect failed, retry
08:00:13.608 -> Audio connected: 1, Async connected: 0
08:00:13.608 -> Enter MQTTDisconnected
08:00:13.646 -> Connecting MQTT: 192.168.2.166, 1883
08:00:18.632 -> Connect failed, retry
08:00:18.632 -> Audio connected: 1, Async connected: 0
08:00:18.632 -> Enter MQTTDisconnected
08:00:18.632 -> Connecting MQTT: 192.168.2.166, 1883
08:00:23.607 -> Connect failed, retry
08:00:23.607 -> Audio connected: 1, Async connected: 0
08:00:23.645 -> Enter MQTTDisconnected
08:00:23.645 -> Connecting MQTT: 192.168.2.166, 1883
08:00:28.645 -> Connect failed, retry
08:00:28.645 -> Audio connected: 1, Async connected: 0
08:00:28.645 -> Enter MQTTDisconnected
08:00:28.645 -> Connecting MQTT: 192.168.2.166, 1883
08:00:33.632 -> Connect failed, retry
08:00:33.632 -> Audio connected: 1, Async connected: 0
08:00:33.632 -> Enter MQTTDisconnected
08:00:33.632 -> Connecting MQTT: 192.168.2.166, 1883
08:00:38.609 -> Connect failed, retry
08:00:38.645 -> Audio connected: 1, Async connected: 0
08:00:38.645 -> Enter MQTTDisconnected
08:00:38.645 -> Connecting MQTT: 192.168.2.166, 1883
08:00:43.633 -> Connect failed, retry
08:00:43.633 -> Audio connected: 1, Async connected: 0
08:00:43.633 -> Enter MQTTDisconnected
08:00:43.633 -> Connecting MQTT: 192.168.2.166, 1883
08:00:43.818 -> Enter MQTTConnected
08:00:43.818 -> Connected as ATOMECHO
08:00:43.818 -> going from mqtt connected to idle
08:00:43.818 -> Enter Idle
08:00:43.818 -> still in idle
08:00:43.818 -> end of idle
08:00:43.818 -> Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
From there it enters a reboot loop, probably because the router doesn't let it onto the network again quickly enough after the crash.
08:04:49.497 -> Connection Failed! Retry...
08:04:59.478 -> Connection Failed! Retry...
08:05:09.497 -> Connection Failed! Rebooting...Perhaps something in the MQTT correspondence is not sitting well.
Romkabouter
commented
3 years ago It says Audio connected: 1, Async connected: 0
This indicates that the streamer audio is connected, but the asynch client has a problem. I cannot tell why
flatsiedatsie
commented
3 years ago What does the async client do? Is that the webserver?
Could this setting be related?
#define CONFIG_ASYNC_TCP_RUNNING_CORE 1
flatsiedatsie
commented
3 years ago Maybe I have to transplant some more settings from this file into the Arduino file.
Romkabouter
commented
3 years ago What does the async client do? Is that the webserver?
Could this setting be related?
#define CONFIG_ASYNC_TCP_RUNNING_CORE 1
Absolutely, by default that ask runs on core 0. Where wifi and other tasks run as well.
flatsiedatsie
commented
3 years ago So I2Stask should run on core 1? I guess the audio streaming has core 1 all to itself?
In the code I see it running on core 0. I changed it to 1 here:
if (i2sHandle == NULL) {
Serial.println("Creating I2Stask");
xTaskCreatePinnedToCore(I2Stask, "I2Stask", 30000, NULL, 3, &i2sHandle, 1); // core was 0, changed it to 1
} else {
Serial.println("We already have a I2Stask");
}No, the I2Stask should run on core 0. This is because the default is core 1 but the I2Stask should have as much cpu power as possible. The MQTT task should run on core 1, but that is used internally be that client. Therefore you should define it.
flatsiedatsie
commented
3 years ago Removing #define CONFIG_ASYNC_TCP_RUNNING_CORE 1 made it much more stable.
However, it seemed to run even better with I2Stask running on core 1 :-D
For my understanding:
xEventGroupSetBits(audioGroup, STREAM); in the idle entry phase, so it would seem so.flatsiedatsie
commented
3 years ago Voco is receiving something if I press the button. If memory serves, this shows that is was ready to receive audio, but it didn't catch any intent, and then switches back to normal mode.
2021-05-31 18:38:00.197 INFO : voco: MQTT message to topic hermes/hotword/toggleOn received on: azrxidia a.k.a. hostname thuis
2021-05-31 18:38:00.197 INFO : voco: +
2021-05-31 18:38:00.198 INFO : voco: {"siteId":"ATOMECHO","sessionId":null}
2021-05-31 18:38:00.198 INFO : voco: +
2021-05-31 18:38:00.199 INFO : voco: In unmute. current_control_name: Headphone
2021-05-31 18:38:00.199 INFO : voco: No intent received
2021-05-31 18:38:00.200 INFO : voco: siteId was in /toggleOn payload: ATOMECHOI can also get the Echo to receive the response from Voco to play a sound (which it doesn't do yet).
18:37:17.020 -> Connected as ATOMECHO
18:37:17.020 -> Connected to asynch MQTT!
18:37:17.020 -> going from mqtt connected to idle
18:37:17.020 -> Enter Idle
18:37:17.020 -> still in idle
18:37:17.020 -> end of idle
18:37:25.252 -> Incoming MQTT message. Topic: hermes/hotword/toggleOff
18:37:25.252 -> toggleOff message was for us
18:37:25.252 -> SessionId in toggleOff
18:37:25.252 -> Hotword detected event
18:37:25.252 -> Enter HotwordDetected
18:37:50.999 -> Incoming MQTT message. Topic: hermes/hotword/toggleOn
18:37:50.999 -> toggleOn message was for us. Going to idle mode.
18:37:50.999 -> Enter Idle
18:37:50.999 -> still in idle
18:37:50.999 -> end of idleEnter MQTTDisconnected
18:37:50.999 -> Incoming MQTT message. Topic:
18:37:51.032 -> hermes/voco/ATOMECHO/play
18:37:51.032 -> Connecting MQTT: 192.168.2.166, 1883
18:38:01.011 -> Connect failed, retry
I2Stask is the audio streamer? It core 0 reserved for it? And all the other things run on core 1? Yes
Is the voco code currently set to permanently stream audio to the main controller after booting up? In theory the main controller could then detect a hotword in the stream. It currently can't get it to detect the hotword, so perhaps it's not streaming? Yes, it is a permanent stream. You can check it by connecting to the broker and subscribe to hermes/audioServer/#
What happens/should happen if I press the Atom Echo's button? Currently it triggers a "hotword detected". I've tried to then speak a command, but it hasn't recognised this yet. The button triggers the hotword, so it changes state. If there is no stream, no command will be caught.
Based on 2 and 3, I think your code is not streaming.
Romkabouter
commented
3 years ago I see you are using 18:37:51.032 -> hermes/voco/ATOMECHO/play. Did you change the topics in General.hpp as well? Because Snips uses hermes/audioServer/atomecho
flatsiedatsie
commented
3 years ago No I left all that as it was. I only added that one extra subscription to see if it was getting anything in return. And it was. So MQTT seemed to be working.
I just checked, it the code does reach the point where it streams audio data onto the network continuously.
Romkabouter
commented
3 years ago Do you see any messages on the audioFrame topic? hermes/audioServer/atomecho/audioFrame.
I don't know it topics are case sensitive, you might want to try all lowercase just to be sure.
I see this in the log as well: 18:37:50.999 -> still in idle 18:37:50.999 -> end of idleEnter MQTTDisconnected
Apparently, the code disconnects. Is there some logging on the broker?
flatsiedatsie
commented
3 years ago Coïncidentally I was just checking this. I don't seen any data arrive on mosquitto_sub -t 'hermes/audioServer/ATOMECHO/#'.
I checked if the correct path was set in Arduino, and it seems ok: hermes/audioServer/ATOMECHO/audioFrame
14:15:54.191 -> Enter WifiDisconnected
14:15:54.191 -> Total heap: 295944
14:15:54.191 -> Free heap: 233724
14:15:56.444 -> Enter WifiConnected
14:15:56.444 -> Connected to Wifi with IP: 192.168.2.137, SSID: sterrenkijker_nomap, BSSID: B0:95:75:9F:FE:CF, RSSI: -45
14:15:56.444 -> Enter MQTTDisconnected
14:15:56.444 -> Connecting MQTT: 192.168.2.166, 1883
14:15:56.444 -> asyncclient connect was called
14:15:56.444 -> also reconnecting to audio
14:15:56.519 -> hermes/audioServer/ATOMECHO/audioFrame
14:15:56.519 -> end of setup
14:15:56.519 -> BOTH CONNECTED
14:15:56.519 -> Enter MQTTConnected
14:15:56.519 -> Connected as ATOMECHO
14:15:56.555 -> Connected to asynch MQTT!
14:15:56.555 -> going from mqtt connected to idle
14:15:56.555 -> Enter Idle
14:15:56.555 -> still in idle
14:15:56.555 -> end of idle. Stream was set to true.
14:15:56.555 -> Total heap: 293988
14:15:56.555 -> Free heap: 158844
14:15:56.555 -> Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.I checked that it was reaching the end of the i2stask too, where it copies the data into a variable.
Could the issue lie with the MQTT part not liking this message? Perhaps the larger MQTT buffer isn't set correctly. Or maybe it's just swamped? From what I could find online, that error could mean an out of memory issue.
flatsiedatsie
commented
3 years ago I managed to receive audio! I modified the PubSubClient to have a larger buffer.
flatsiedatsie
commented
3 years ago An example of the Snips output:
hermes/audioServer/azrxidia/audioFrame RIFF4WAVEfmt ?>}timc???ydata?????????j?F?\?Q?`???????????????????????!?-?3?_???y?q?????????
#JkpgQSL4????????????????X?5????????x??{?????????????%?=?U?q??????????????? ????????|?[?F?=?7????
???????????????????????(?7?7?Y?n???????????????1J┌????????????????┬±D=5???????????????????????┴???????????????????#L┼─??????????
??????⎺─│≥┬V[??????????
? <␉┌°⎻␍S@0=G▒⎺␉Y.????␉A??????????
␊⎼└␊⎽/▒┤␍␋⎺S␊⎼┴␊⎼/▒≥⎼│␋␍␋▒/▒┤␍␋⎺F⎼▒└␊ RIFF4WAVE°└├ ?>£├␋└└???≤␍▒├▒??????│?▒?┴?????????????????????????)4,O␍Y??????┤NM<7??????$%#EXSH=>+????????????
????????????≥?▒?▒?±?E?N?T?_?┴?⎼?┌?[?.?"??????????????????????7?I?@?\????????@▒?????'4D[±␌V▒??⎺±QQC>"????±.??????????????????????≥?????≠???????????????????????1G7$
????????????
17
-T␊┴?????????????
!<<@8?H␌XX─┌┼├\?????┴┼IE@ ??
␊⎼└␊⎽/??????????vbWytlP4ELZXiv|????????????????????lD@0/4????≤␍0-2'?????????????????????⎼????????????K°⎺??????????
????%=\{???????????(7L8@KB????cBDN( ??????????????u?b?P?V?K?L?5?+?%?#? ?.?*?<?S??o?Q?e?h?T?"??
?!?5?H?i?e?Q?`?d?w?f?l?????????w?e?^?W?]?U?g?h?[?H?D?,?*??!?:?8?>?,?O?S?d?t???????????u?????????????????????????????j?f?K?k?J?0?
?
?????????????j?`?a?2?/?*?And the Atom Echo's output
hermes/audioServer/ATOMECHO/audioFrame RIFF,WAVEfmt ?>}data?!?!?&?&?"?)?+???? ???&?!?3??4?$?/?*?@?<?9?G?J?N???M?D?M?G???E???<?=?3?B?2?<?/?0?&?&?????? ?%??"?#?)?(?#??&?"?%?)?.?0?7?+?<?2?<?7?B???A?7?6?9?1?<?2?6?/?9?/?9?-?2?%? ?#? ?!? ?%?'?(?%?+?'?.?(?*?*?1?2?2?4?<?/?B?C?H?D?L?O?X?R?V?Q?S?T?Q?Q?O?[?[?V?U?O?N?P?G?I?G???D?8?;?-?4?0?(?+?&?+??,??,??!?#?!??*??'??%????#??-??0??0?'?0?4?4?3?-?5?0?1?/?6?0?0?4?/?,?0?-?&?&?$?&?&?%?!?-?%?+?'?,?.?-?,?1?-?8?&?6?4?9?-?<?A?8?D?N?L?I?I?Q?Q?E?F?H?>?=?>?:?@?9?>?,?9?/?:?&?)?'?&?!??$????? ????????
hermes/audioServer/azrxidia/audioFrame RIFF4WAVEfmt ?>}tim????ydata??
?*?3?Y?d?J?m?{?l?p?~???????????????%?F?3?[?p?p?????????????@FI+????????????????????????????????????????????????????????????????????????~?k?g?\?g?T?T?b?Q?[?l?g?R?T?V?G?N?X?l?n????~?|?~???????????????3Gdv?????????????????????kV^K/D?RMB+??????????????????????????????????????????????≥?┼?T?I?A?5?7?'?? ??
??&?$?6?'?6?3?(?,?(?????????????????????????????????????????????????(?/?G?F?<?3?(??1???
????????????????⎻?D?>?H?Z?┐?π?┤?⎻?
␊⎼└␊⎽/▒┤␍␋⎺S␊⎼┴␊⎼/ATOMECHO/▒┤␍␋⎺F⎼▒└␊ RIFF,WAVE°└├ ?>£␍▒├▒(?!?%?,?1?4?9?9?@?8???<?D???N?P?U?W?◆?Q?[?R?]?T?P?[?Y?]?W?V?P?O?O?S?G?L?L?I?D?K?C?>???B?D?E?C?@?@?A???=?E?=???;?????@?9?@?>?C?B?E?D?J?F?F?@?D???=?>?@?@?6???9?D?9?5?:?8?5?%?/?$?-?.?)?/?-?/?'?)?/?5?3?1?8?2?6?>?E?;?E?C?G?M?J?I?O?G?W?G?P?I?O?U?M?N?R?L?J?E?C???5?C?1?B?@?4?5?3?=?8?-?:?/?4?)?+?)?,?(?5?:?*?5?,?5?*?)?6?8?4?:?7?@?5?F?C?E?J?D?L?H?V?K?P?L?V?V?J?N?N?U?O?S?M?O?G?E?H?G?H?I?D?L?=?E?<?G?I?C?J?>?G?F?K?T?I?U?O?S?T?W?Q?V?Y?P?]?\?␉?^?▒?◆?Y?]?X?Y?X?X?[?^?_?P?]?T?T?J?Q?Y?R?R?P?P?P?Q?O?M?N?R?Q?M?V?R?[?S?]?\?T?W?T?
␊⎼└␊⎽/▒┤␍␋⎺S␊⎼┴␊⎼/▒≥⎼│␋␍␋▒/▒┤␍␋⎺F⎼▒└␊ RIFF4WAVE°└├ ?>£├␋└????≤␍▒├▒⎺?┤???????????????????!?4?K?␋?????????????????????????????????????#????????????????????????????????????????
??????????????????????'0??????+<4Aab2
&)IPe??????????
./62V[]bn??????????????vI.? %19
?????wx^ZY]Yot???????????????yp?z?????????????????????????????????????????????????????bK???????????
)4AIB
hermes/audioServer/ATOMECHO/audioFrame RIFF,WAVEfmt ?>}data'?,?/?.?4?'?+?&?3?"?-?/?;?.?0?1?.?1?,?0?0?,?.?'?/?3?:?4?2?6?@?;?9?9?9?<?:???I?7?=?B?C?8?6???;?9?;?:?;?>?>?B???3?:?9???8?;?A?;?6?D?5?;?7???=?9?>?>?=?;?C?9?@?G?I?L?D?=?E?A?H? ?)??!?!?&?&?"?)?+?+?(?.?'?*?+?(?)?7?6?4?7?:???8?;?A?<?=?=?@?C?6?;?5?<?7?>?.?6?4?/?0?*??!??*? ??? ???????????
????%?$?,?"?)?1?.?-?1?1?=?.?9?5?=?;?@?A?B?>?J?G???B?D???:?=?4?;?5?=?:?>?7?9?2?9?8?/?=?:?=?/?8?<?6?=?=?=?6?5?9?7?9?:?<?3?8?8?=?@?C?;?;?C?=?=???>?8???8?9?5?9?6?7?1???9?:?;?2?8?-?2?(?.?,?(?/?.?0?5?A?:?8?9?D?:?1?:?7?
hermes/audioServer/azrxidia/audioFrame RIFF4WAVEfmt ?>}tim????ydata6$FENQN3BHSijs??????????????svaUJOFA+9>GO5$??
????????????u?h?i?C?C?M?N?T?7?R?f?t?n?h?r?a?k?o?s?n?o?????????v?n?h?|?d???????????????????????????!!'')&$;?DB????????????????????????
?????????????????????????????????????????????????????????????????????????? DCA_o????????????????????????rvZTLKZhov?caaZZ[IREVUT^dbh`hxg??????????????????????
hermes/audioServer/azrxidia/audioFrame RIFF4WAVEfmt ?>}tim??ydata??????v?mb?zv????????????X^\Xo??vlo?s>H<??????????????????????????????????????
???&JHCW[aO@HM\XTD/BH=????????????????????????????????????????????????????????????????????????
??????????????????─?_?[?␊?◆?◆?L?4?9?>?A?1?$?)?,??!?&?:?<?<?7?3???????
?*???S?␊?P?^?␋?W?\?⎻?─?≠?????????????????????????????????????????⎼?@?6?▒?␌?⎺??????≥?┘?\???$?!?9?J?V?£?⎻?M?V?G?@?N?◆?└?R?@?U?E?!??'??????????????????????????????????? ?#???
␊⎼└␊⎽/▒┤␍␋⎺S␊⎼┴␊⎼/▒≥⎼│␋␍␋▒/▒┤␍␋⎺F⎼▒└␊ RIFF4WAVE°└├ ?>£├␋└͊??≤␍▒├▒??5?V?F?Z?▒?┴?????????????????????????????????????????????????????│?·?≠???????????????┬?┼?[?␋???─???≤???????????????????????????????????????????????????????????????????????????????????????????????
???????????+@N[\±┐VG41'?071UCYA-)32-????????3G8,.)!7
?
!66K/ ???? 1*%=I?F.%-2&!1
????%&*
????????????????????????????????z?k?p?o?h?y?t?m?^?a?h?n?{???????????hermes/audioServer/azrxidia/audioFrame RIFF4WAVEfmt ?>}timc???ydata
hermes/audioServer/ATOMECHO/audioFrame RIFF,WAVEfmt ?>}data
I see a difference between the two strings, where the Snips one has }timc???ydata where the Atom has }data only?
Romkabouter
commented
3 years ago I managed to receive audio! I modified the PubSubClient to have a larger buffer.
Ah ok, this is the ("MQTT_MAX_PACKET_SIZE", 2000) setting in this file ;)
https://github.com/Romkabouter/ESP32-Rhasspy-Satellite/blob/master/PlatformIO/load_settings.py
Snips has an additional header, found here: https://github.com/snipsco/hermes-protocol/blob/develop/hermes/src/ontology/audio_server.rs#L25..L87
I have no such extra bytes, which is why the header is different, I have tested this the header with Snips and that worked for me, so I think it has some additional functionality for snips but is not causing issues if those bytes are not there.
This is a continuation of discussion here.
I've managed to get Snips to recognise the wake-word, but only right after booting the Atom Echo.
Snips does recognise that there is audio input.
While in idle mode, Snips Watch indicates that audio is being heard.
If I press the button to start a session, a session is created, and the dialogue manager listens to the stream from the Atom Echo. But the voice input is not recognised as a voice command:
If I don't speak into the Raspberry Pi version, then things look a bit different.
So what would support the idea that some MQTT message is missing.
As an aside, I also noticed that the wave header is slightly different:
ESP32
USB microphone on Raspberry Pi:
I also check the output of the various commands in Mosquitto to find out which exact message was missing.
----Atom Echo button----
----Atom Echo hotword detected----