Vaquita-DSPG Crashes While Idle

HowardHonig commented 3 years ago

After about 5 minutes, Vaquita will crash and fall into gdb with the message "Entering gdb stub now".

I've enabled RainMaker. All code is the original, that is not modified. Before the crash, everything seems to work well and is really quite impressive.
When RainMaker is not enabled it also crashes but takes a little longer.

Any ideas?

The last printed messages from the serial port are:

[0;32mI (2970557) opus_decoder: Total decoded frames = 240[0m [0;32mI (2970557) opus_decoder: Closed[0m [0;32mI (2970567) [audio_codec]: Codec opus_decoder Event Stopped[0m [2968 seconds]: [http_transport]: Event data: {"events":[{"header":{"name":"SpeakerClosed","messageId":"4377508f-c7be-66d2-a922-bbc1119a0733"},"payload":{"offset":69280}}]} [0;32mI (2970587) [http_transport]: Sending to topic: $aws/alexa/ais/v1/517f70d2-c7f5-44af-8486-eab66f93135e/event[0m [0;32mI (2970597) [basic_player]: Waiting for decoder to stop line 116[0m [basic_player]: Codec event: Codec Stopped [0;32mI (2970607) opus_decoder: Closed[0m [0;32mI (2970617) [audio_codec]: Codec opus_decoder Event Stopped[0m [basic_player]: Codec event: Codec Stopped [0;32mI (2970627) [audio_codec]: Closing codec opus_decoder without decoding any data[0m [attachment]: Free ais_attachment_3 [dialog]: Speech done [2970 seconds]: [http_transport]: Free Memory Internal: 51836, External: 2480240 [directive_proc]: Json data: {"directives":[{"header":{"name":"SetAttentionState","messageId":"dd02e871-f5ca-4847-a105-952f0e738d45"},"payload":{"state":"IDLE"}}]} [0;32mI (2972877) [directive_proc]: Name: SetAttentionState [0m [dialog]: Stream finished [dialog]: Event STREAM_CLOSED unsupported in SPEAKING state [0;32mI (2972897) [app_va_cb]: Dialog state is: 8[0m [dialog]: Speaking state timeout. Releasing focus [0;32mI (2975397) [focus_manager]: Release channel: Dialog SpeechSynthesizer[0m [0;32mI (2975397) [focus_manager]: Releasing SpeechSynthesizer[0m [0;32mI (2975397) [sys_playback]: Release[0m [dialog]: Entering VA_IDLE [auth-delegate]: Queue receive returned false for 3000 seconds. It timed out. Getting new credentials. Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled. Core 0 register dump: PC : 0x4015ff24 PS : 0x00060e30 A0 : 0x80160897 A1 : 0x3f829410 A2 : 0x0000003c A3 : 0x00000000 A4 : 0x3f983b1c A5 : 0x3f829540 A6 : 0x3fff498c A7 : 0x00000000 A8 : 0x00000004 A9 : 0x3f8293d0 A10 : 0x00000000 A11 : 0x00000fff A12 : 0x00001005 A13 : 0x00000001 A14 : 0x00000008 A15 : 0x3f807574 SAR : 0x00000018 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000004 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff ELF file SHA256: 5659eb0ee866b947 Backtrace: 0x4015ff21:0x3f829410 0x40160894:0x3f8294e0 0x4011345c:0x3f829510 0x400daa6d:0x3f829530 0x400dab69:0x3f8295a0 0x400dae0c:0x3f8295c0 Entering gdb stub now. $T0b#e6

chiragatal commented 3 years ago

Could you share the IDF branch/commit you are using? Could you also share another log of the device crashing?

HowardHonig commented 3 years ago

Thank you, IDF version I couldn't figure out how to tell the version number of IDF but I originally loaded it from the instructions:

$ git clone --recursive https://github.com/espressif/esp-idf.git $ cd esp-idf; git checkout release/v4.0; git submodule init; git submodule update --init --recursive;

Here is the output of the export.bat file: Setting IDF_PATH: C:\Users\howar\Documents\Alexa\iot\esp-vaquita\esp-idf

Adding ESP-IDF tools to PATH... C:\Users\howar.espressif\tools\xtensa-esp32-elf\esp-2020r3-8.4.0\xtensa-esp32-elf\bin C:\Users\howar.espressif\tools\esp32ulp-elf\2.28.51.20170517\esp32ulp-elf-binutils\bin C:\Users\howar.espressif\tools\cmake\3.13.4\bin C:\Users\howar.espressif\tools\openocd-esp32\v0.10.0-esp32-20210721\openocd-esp32\bin C:\Users\howar.espressif\tools\mconf\v4.6.0.0-idf-20190628\ C:\Users\howar.espressif\tools\ninja\1.9.0\ C:\Users\howar.espressif\tools\idf-exe\1.0.1\ C:\Users\howar.espressif\tools\ccache\3.7\ C:\Users\howar.espressif\python_env\idf4.0_py3.9_env\Scripts C:\Users\howar\Documents\Alexa\iot\esp-vaquita\esp-idf\tools

and when I do a git describe from the esp-idf directory: v4.0.3-144-gd64bf0dfbe

I could try updating to v4.1.12 if you think that will help.

Crash Logs This was captured 3x with the same result. It is taking a little longer to crash today, about 1/2 hr or so. It enters gdb. When gdb is exited, it looks like vaquito tries to boot up but doesn't. Hitting a reset does boot vaquito. And here is the log of the last crash. Note that the SSDP is my AV amp and I know that isn't supported: '' 40e4::urn:schemas-upnp-org:service:RenderingControl:1 W (2486257) SSDP: --end-- I (2487267) SSDP: ---SSDP List---- I (2487267) SSDP: 1.id=, ip=http://192.168.1.232:8080/MediaRenderer/desc.xml, name=uuid:5f9ec1b3-ed59-1900-4530-00a0dea340e4::urn:schemas-upnp-org:service:ConnectionManager:1 W (2487277) SSDP: --end-- I (2487567) SSDP: ---SSDP List---- I (2487567) SSDP: 1.id=, ip=http://192.168.1.232:8080/MediaRenderer/desc.xml, name=uuid:5f9ec1b3-ed59-1900-4530-00a0dea340e4::urn:schemas-upnp-org:service:AVTransport:1 W (2487577) SSDP: --end-- W (2538097) LSSDP: remove timeout SSDP neighbor: (http://192.168.1.232:8080/MediaRenderer/desc.xml) (50522ms) I (2538097) SSDP: ---SSDP List---- W (2538097) SSDP: Empty W (2598387) httpd_uri: httpd_uri: URI '/rootDesc.xml' not found W (2598387) httpd_txrx: httpd_resp_send_err: 404 Not Found - This URI does not exist W (2719017) httpd_uri: httpd_uri: URI '/rootDesc.xml' not found W (2719017) httpd_txrx: httpd_resp_send_err: 404 Not Found - This URI does not exist W (2778647) httpd_uri: httpd_uri: URI '/rootDesc.xml' not found W (2778647) httpd_txrx: httpd_resp_send_err: 404 Not Found - This URI does not exist W (2838877) httpd_uri: httpd_uri: URI '/rootDesc.xml' not found W (2838877) httpd_txrx: httpd_resp_send_err: 404 Not Found - This URI does not exist [auth-delegate]: Queue receive returned false for 3000 seconds. It timed out. Getting new credentials. Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled. Core 0 register dump: PC : 0x4015ff24 PS : 0x00060d30 A0 : 0x80160897 A1 : 0x3f829410 0x4015ff24: esp_tcp_connect at C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-idf/components/esp-tls/esp_tls.c:163 (inlined by) esp_tls_low_level_conn at C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-idf/components/esp-tls/esp_tls.c:616

A2 : 0x0000003c A3 : 0x00000000 A4 : 0x3f8519ec A5 : 0x3f829540 A6 : 0x3fff4984 A7 : 0x00000000 A8 : 0x00000004 A9 : 0x3f8293d0 A10 : 0x00000000 A11 : 0x00000fff A12 : 0x00001005 A13 : 0x00000001 A14 : 0x00000008 A15 : 0x3f807574 SAR : 0x00000018 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000004 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff

ELF file SHA256: 5659eb0ee866b947

Backtrace: 0x4015ff21:0x3f829410 0x40160894:0x3f8294e0 0x4011345c:0x3f829510 0x400daa6d:0x3f829530 0x400dab69:0x3f8295a0 0x400dae0c:0x3f8295c0 0x4015ff21: esp_tcp_connect at C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-idf/components/esp-tls/esp_tls.c:163 (inlined by) esp_tls_low_level_conn at C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-idf/components/esp-tls/esp_tls.c:616

0x40160894: esp_tls_conn_new at C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-idf/components/esp-tls/esp_tls.c:717

0x4011345c: http_connection_new at C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-va-sdk/components/httpc/httpc.c:156

0x400daa6d: get_auth_response_with_header at /Users/ganesh/work/gitlab/esp-alexa/components/voice_assistant/common/src/auth_delegate.c:83

0x400dab69: get_auth_response at /Users/ganesh/work/gitlab/esp-alexa/components/voice_assistant/common/src/auth_delegate.c:141

0x400dae0c: refresh_auth_task at /Users/ganesh/work/gitlab/esp-alexa/components/voice_assistant/common/src/auth_delegate.c:303

Entering gdb stub now. $T0b#e6GNU gdb (crosstool-NG esp-2020r3) 8.1.0.20180627-git Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-host_w64-mingw32 --target=xtensa-esp32-elf". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from c:\users\howar\documents\alexa\iot\esp-vaquita\esp-va-sdk\examples\amazon_aia\build\aia.elf...done. Remote debugging using COM7 Ignoring packet error, continuing... warning: unrecognized item "timeout" in "qSupported" response Remote replied unexpectedly to 'vMustReplyEmpty': vMustReplyEmpty (gdb) ''

chiragatal commented 3 years ago

From your logs, it seems to crash after:

Queue receive returned false for 3000 seconds. It timed out. Getting new credentials.

The device connects to the server and gets new access tokens. And the crash seems to be when connecting. Let me check and get back. What I can think of, is it could be because of lack of memory or lack of sockets.

Have you also checked with the default repository?

HowardHonig commented 3 years ago

Thanks again. Not sure what you identify as the default repository. Just tried building with the recent tools version v4.3.1 of the tools and got a linking error, undefined reference. I did change the project.cmake file to add target_link_libraries(${project_elf} "-Wl,--start-group") statement.

Got the below link error message. Could not find where xTaskNofity is defined, except in a library in the recommended downloaded but not in v4.3.1 tools. Did find a number of places in which the function is called.

FAILED: aia.elf cmd.exe /C "cd . && C:\Users\howar.espressif\tools\xtensa-esp32-elf\esp-2020r3-8.4.0\xtensa-esp32-elf\bin\xtensa-esp32-elf-g++.exe -mlongcalls -Wno-frame-address @CMakeFiles\aia.elf.rsp -o aia.elf && cd ." c:/users/howar/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-va-sdk/components/voice_assistant/lib/libaia.a(dialog.o):(.literal.dialog_sm_thread+0x30): undefined reference to xTaskNotify' c:/users/howar/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: C:/Users/howar/Documents/Alexa/iot/esp-vaquita/esp-va-sdk/components/voice_assistant/lib/libaia.a(dialog.o): in functiondialog_sm_thread': /Users/ganesh/work/gitlab/esp-alexa/components/voice_assistant/common/src/dialog.c:324: undefined reference to `xTaskNotify' collect2.exe: error: ld returned 1 exit status ninja: build stopped: subcommand failed. ninja failed with exit code 1

chiragatal commented 3 years ago

The esp-idf version or branch should be release/v4.0 for the amazon_aia example. It is not expected to work with other IDF versions since the libraries built are version specific. Can you try with the release/v4.0 branch for esp-idf? (From the logs in previous comments, I think you were already using this branch and getting the crashes on this branch. We have a planned release for this week with release/v4.2 version and it has a few stability fixes for AIA.)

By 'default repository', I mean without any changes (apart from the patches required to compile) that you may have done.

HowardHonig commented 3 years ago

Thank you chiragatal.
I deleted and reinstalled all the need support drivers, libraries, and packages. Made sure it was version 4.0 of the esp-idf. Still crashes in the same way. Did not try the "future aia/beta" version. Looked for version esp-va-sdk 4.2 and it is an older release so I'll wait for the "planned release for this week of v4.2".

I also eliminated the SSDP sources in the local network, except for the other Alexa's which also sends out SSDP responses, still crashed. Tried a couple of times to delete the build directory and did an "idf.py fullclean" to be thorough.

What I don't understand is why I'm seeing this problem since this board is not new. Doesn't anyone else have this same problem? If not, perhaps my board is defective and I'm willing to buy another one to see if it is. If not, I'll wait for the new release but need more information on which driver and support package is being released so I can download and try it when ready.

chiragatal commented 3 years ago

The new release would be for esp-va-sdk. It will use esp-idf v4.2 and will not use esp-aws-iot at all. When released, just installing/updating the esp-idf drivers (running install.sh) should be enough.

espressif / esp-va-sdk

Vaquita-DSPG Crashes While Idle #151