Closed StuartIanNaylor closed 2 years ago
I received my ESP Box today. I've not been able to get the wake word to activate at all. I've tried both the 0.1.1 and 0.2.1 firmware, and also tried waking it with the voice commands in this demo video without success.
Tried a Skainet example which uses the wake word Alexa and this works fine.
Compiled the ESP_Box demo to use wakeword model alexa8 instead of hiesp8 and now wakes easily.
Tried a Skainet example which uses the wake word Alexa and this works fine.
Compiled the ESP_Box demo to use wakeword model alexa8 instead of hiesp8 and now wakes easily.
Maybe the Hi_ESP_Q8 model as I did get it to work but learning how to position myself and control my speech for wake word recognition took considerable time. But maybe your recompile as multinet in the supplied firmware seemed equally buggy as after saying 'turn white' anything was recognised as 'turn white'.
I need to get back up to speed with esp and install the IDF and get to grips with the framework so will also compile and test but currently the supplied firmware gives an extremely bad impression of the box capabilities which in use is near useless.
@s60sc how did you find the multinet commands?
https://github.com/espressif/esp-sr/blob/509cf82658cf2b645aa9ae89b6764beea48e7eae/include/esp_map.h#L23 Also did you change to 34mm as with the box mic arrangement?
The multiword model worked fine for me. I did not change the mic arrangement. I will do more work on the ESP-BOX in the new year. I have uploaded a version of the firmware with the Alexa wakeword in this repository
Hi, we have updated a new "Hi,ESP" model by adding more samples.
You can try it by pulling the latest esp-sr release/v1.0 branch.
The model of "Hi,ESP" is same as "Alexa". The main difference is the sample. "Alexa" samples come from different countries and accents so "Alexa" model feels more accurate. We will continue to improve the performance of "Hi,ESP" model.
Tried the skainet en_speech_commands_recognition example after updating the esp-sr component, installed on the ESP-Box. The new hiesp8 model crashes the ESP-Box without even logging an exception. The alexa8 model works fine.
hiesp8 version:
model_name: hiesp8 model_data: /srmodel/hiesp8/wn8_data
HUFZIP MC Quantized wakeNet8: WakeNet8_v3h8_hiesp_5_0.59_0.60, mode:2, p:1, (Jan 4 2022 15:08:13)
alexa8 version:
model_name: alexa8 model_data: /srmodel/alexa8/wn8_data
HUFZIP MC Quantized wakeNet8: wakeNet8_v5h8_alexa_5_0.57_0.59, mode:2, p:1, (Jan 4 2022 15:08:13)
Core dump if that helps:
Quantized8 Multinet5: MN5Q8_v1_english_8_0.9_0.90, (Dec 30 2021 14:40:34)
------------detect start------------
Initiating core dump!
I (4849) esp_core_dump_uart: Press Enter to print core dump to UART...
I (4856) esp_core_dump_uart: Print core dump to uart...
Core dump started (further output muted)
Received 16 kB...
Core dump finished!
espcoredump.py v0.4-dev
===============================================================
==================== ESP32 CORE DUMP START ====================
Crashed task handle: 0x3fcb0500, name: 'detect', GDB name: 'process 1070269696'
================== CURRENT THREAD REGISTERS ===================
exccause 0x1c (LoadProhibitedCause)
excvaddr 0x10
epc1 0x4202b8ef
epc2 0x0
epc3 0x0
epc4 0x0
epc5 0x0
epc6 0x0
eps2 0x0
eps3 0x0
eps4 0x0
eps5 0x0
eps6 0x0
pc 0x4201be38 0x4201be38 <dl_layer_glu_i16+120>
lbeg 0x4037cd84 1077398916
lend 0x4037cda7 1077398951
lcount 0x0 0
sar 0x1a 26
ps 0x60b20 396064
threadptr <unavailable>
br <unavailable>
scompare1 <unavailable>
acclo <unavailable>
acchi <unavailable>
m0 <unavailable>
m1 <unavailable>
m2 <unavailable>
m3 <unavailable>
expstate <unavailable>
f64r_lo <unavailable>
f64r_hi <unavailable>
f64s <unavailable>
fcr <unavailable>
fsr <unavailable>
a0 0x820083a7 -2113895513
a1 0x3fcb67b0 1070294960
a2 0x3fce5634 1070487092
a3 0x3fca8b98 1070238616
a4 0xfffffff1 -15
a5 0xfffffff6 -10
a6 0x0 0
a7 0x3d830390 1031996304
a8 0xfffffff1 -15
a9 0x3fce5968 1070487912
a10 0x3fce5634 1070487092
a11 0xfffffff0 -16
a12 0x3d830400 1031996416
a13 0x3da1af90 1034006416
a14 0x0 0
a15 0x7bb55c8 129717704
==================== CURRENT THREAD STACK =====================
#0 0x4201be38 in dl_layer_glu_i16 (args=0x3fce5968, in=<optimized out>, out=0x3fce6844) at ../components/dl_lib/layer/dl_layer_glu.c:107
#1 0x420083a7 in wakeNet7_model_run_step1 (model=0x3fce42b0) at ../components/wakenet/wakenet8_quantized.c:736
#2 0x42008fa8 in model_detect (model=0x3fce42b0, samples=<optimized out>) at ../components/wakenet/wakenet8_quantized.c:1002
#3 0x4200a362 in afe_fetch (afe=0x3fce195c, out=0x3fcebd58) at ../components/esp_audio_front_end/esp_afe_sr_2mic.c:464
#4 0x42007d74 in detect_Task (arg=0x3fce195c) at ../main/main.c:110
#5 0x4038b9a8 in vPortTaskWrapper (pxCode=0x42007cec <detect_Task>, pvParameters=0x3fce195c) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130
======================== THREADS INFO =========================
Id Target Id Frame
* 1 process 1070269696 0x4201be38 in dl_layer_glu_i16 (args=0x3fce5968, in=<optimized out>, out=0x3fce6844) at ../components/dl_lib/layer/dl_layer_glu.c:107
2 process 1070240844 0x400559e0 in ?? ()
3 process 1070238956 0x4206813e in esp_pm_impl_waiti () at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:182
4 process 1070234096 0x400559e0 in ?? ()
5 process 1070229428 xQueueSemaphoreTake (xQueue=0x3fca6358, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/queue.c:1563
6 process 1070227788 0x40389758 in xQueueSemaphoreTake (xQueue=0x3fca5cf0, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38
7 process 1070269348 0x403895b5 in xQueueReceive (xQueue=0x3fce1708, pvBuffer=0x3fce16f4, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38
==================== THREAD 1 (TCB: 0x3fcb0500, name: 'detect') =====================
#0 0x4201be38 in dl_layer_glu_i16 (args=0x3fce5968, in=<optimized out>, out=0x3fce6844) at ../components/dl_lib/layer/dl_layer_glu.c:107
#1 0x420083a7 in wakeNet7_model_run_step1 (model=0x3fce42b0) at ../components/wakenet/wakenet8_quantized.c:736
#2 0x42008fa8 in model_detect (model=0x3fce42b0, samples=<optimized out>) at ../components/wakenet/wakenet8_quantized.c:1002
#3 0x4200a362 in afe_fetch (afe=0x3fce195c, out=0x3fcebd58) at ../components/esp_audio_front_end/esp_afe_sr_2mic.c:464
#4 0x42007d74 in detect_Task (arg=0x3fce195c) at ../main/main.c:110
#5 0x4038b9a8 in vPortTaskWrapper (pxCode=0x42007cec <detect_Task>, pvParameters=0x3fce195c) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130
==================== THREAD 2 (TCB: 0x3fca944c, name: 'IDLE') =====================
#0 0x400559e0 in ?? ()
#1 0x4038bc59 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/include/freertos/portmacro.h:571
#2 vPortExitCritical (mux=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:318
#3 0x4202c09e in esp_task_wdt_reset () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/task_wdt.c:330
#4 0x4202c0aa in idle_hook_cb () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/task_wdt.c:80
#5 0x42003974 in esp_vApplicationIdleHook () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/freertos_hooks.c:51
#6 0x40389e30 in prvIdleTask (pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/tasks.c:3974
#7 0x4038b9a8 in vPortTaskWrapper (pxCode=0x40389e24 <prvIdleTask>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130
==================== THREAD 3 (TCB: 0x3fca8cec, name: 'IDLE') =====================
#0 0x4206813e in esp_pm_impl_waiti () at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:182
#1 0x42003982 in esp_vApplicationIdleHook () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/freertos_hooks.c:63
#2 0x40389e30 in prvIdleTask (pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/tasks.c:3974
#3 0x4038b9a8 in vPortTaskWrapper (pxCode=0x40389e24 <prvIdleTask>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130
==================== THREAD 4 (TCB: 0x3fca79f0, name: 'esp_timer') =====================
#0 0x400559e0 in ?? ()
#1 0x4038bc59 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/include/freertos/portmacro.h:571
#2 vPortExitCritical (mux=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:318
#3 0x4038b1fd in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/tasks.c:5388
#4 0x420073bb in timer_task (arg=0x0) at C:/Utilities/ESP32_IDF_V44R/components/esp_timer/src/esp_timer.c:384
#5 0x4038b9a8 in vPortTaskWrapper (pxCode=0x420073ac <timer_task>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130
==================== THREAD 5 (TCB: 0x3fca67b4, name: 'ipc1') =====================
#0 xQueueSemaphoreTake (xQueue=0x3fca6358, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/queue.c:1563
#1 0x4038212b in ipc_task (arg=0x1) at C:/Utilities/ESP32_IDF_V44R/components/esp_ipc/src/esp_ipc.c:59
#2 0x4038b9a8 in vPortTaskWrapper (pxCode=0x403820fc <ipc_task>, pvParameters=0x1) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130
==================== THREAD 6 (TCB: 0x3fca614c, name: 'ipc0') =====================
#0 0x40389758 in xQueueSemaphoreTake (xQueue=0x3fca5cf0, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38
#1 0x4038212b in ipc_task (arg=0x0) at C:/Utilities/ESP32_IDF_V44R/components/esp_ipc/src/esp_ipc.c:59
#2 0x4038b9a8 in vPortTaskWrapper (pxCode=0x403820fc <ipc_task>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130Exception in thread Thread-1:
Traceback (most recent call last):
File "threading.py", line 932, in _bootstrap_inner
File "threading.py", line 870, in run
File "subprocess.py", line 1366, in _readerthread
OSError: [Errno 22] Invalid argument
WARNING: Attempt to terminate the GDB process failed, because it is already terminated. Skip
==================== THREAD 7 (TCB: 0x3fcb03a4, name: 'feed') =====================
#0 0x403895b5 in xQueueReceive (xQueue=0x3fce1708, pvBuffer=0x3fce16f4, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38
#1 0x42027698 in i2s_read (i2s_num=I2S_NUM_1, dest=<optimized out>, size=6400, bytes_read=0x3fcb021c, ticks_to_wait=4294967295) at C:/Utilities/ESP32_IDF_V44R/components/driver/i2s.c:2219
#2 0x42007ca5 in feed_Task (arg=0x3fce195c) at ../main/main.c:67
#3 0x4038b9a8 in vPortTaskWrapper (pxCode=0x42007c58 <feed_Task>, pvParameters=0x3fce195c) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130
======================= ALL MEMORY REGIONS ========================
Name Address Size Attrs
.rtc.text 0x600fe000 0x0 RW
.rtc.dummy 0x600fe000 0x0 RW
.rtc.force_fast 0x600fe000 0x0 RW
.rtc.data 0x50000000 0x10 RW A
.rtc_noinit 0x50000010 0x0 RW
.rtc.force_slow 0x50000010 0x0 RW
.iram0.vectors 0x40378000 0x403 R XA
.iram0.text 0x40378404 0x16407 R XA
.dram0.data 0x3fc9e810 0x37f4 RW A
.noinit 0x3fca2004 0x0 RW
.flash.text 0x42000020 0x6ae27 R XA
.flash.appdesc 0x3c070020 0x100 R A
.flash.rodata 0x3c070120 0x212e4 RW A
.flash.rodata_noload 0x3c091404 0x0 RW
.iram0.text_end 0x4038e80b 0x0 RW
.iram0.bss 0x4038e80c 0x0 RW
.dram0.heap_start 0x3fca4358 0x0 RW
.coredump.tasks.data 0x3fcb0500 0x158 RW
.coredump.tasks.data 0x3fcb66f0 0x304 RW
.coredump.tasks.data 0x3fca944c 0x158 RW
.coredump.tasks.data 0x3fca98c0 0x2e4 RW
.coredump.tasks.data 0x3fca8cec 0x158 RW
.coredump.tasks.data 0x3fca91c0 0x284 RW
.coredump.tasks.data 0x3fca79f0 0x158 RW
.coredump.tasks.data 0x3fca7750 0x298 RW
.coredump.tasks.data 0x3fca67b4 0x158 RW
.coredump.tasks.data 0x3fca6530 0x27c RW
.coredump.tasks.data 0x3fca614c 0x158 RW
.coredump.tasks.data 0x3fca5ec0 0x284 RW
.coredump.tasks.data 0x3fcb03a4 0x158 RW
.coredump.tasks.data 0x3fcb00e0 0x2bc RW
===================== ESP32 CORE DUMP END =====================
===============================================================
Done!
Coredump checksum='315a1d99'
I (6357) esp_core_dump_uart: Core dump has been written to uart.
have fixed the bug of wakenet8_hiesp, pls try again.
Now works, and hiesp8 model is significantly improved.
Updated factory demo firmwares in this repository
Seems to be now fixed so will close
Hi downloaded https://github.com/espressif/esp-box/releases/download/v0.2.1/ESP-Box_Demo_EN_V0.2.1.bin and flashed.
I have spent over ten minutes several reboots to get only 3 recognitions with worry that my neighbours may be thinking I have gone mad.
If you do manage the wake word then the command turn on/off light works great. Actually getting the wake word to activate is another matter in a silent room.
I have a feeling it could be your dataset and now it actually has a english male voice it doesn't fit well or this firmware / model is not good. I have no problems recording some samples if you tell me format and sampling rate... Maybe community members could contribute if its merely a esp32-s3-box dataset accent problem.
Actually with more playing seems you need to be 2-3 meters away (further the better) for it to work relatively well if you pull the unit out on place close on your desk it doesn't work well maybe AGC? Maybe its as simple as the MAP of mic spacing is 50mm whilst the box is 34mm, dunno but currently does not seem to be well.
The hardware design and software of the esp-box is brilliant and is really polished but the results currently of ESP_SR wow I really don't want to comment and due to the module being a .a with no source code there is little I can do, but wait fingers crossed. ESP_DL lacks LSTM/GRU support which for audio is likely to be restrictive but always struggle what the xtensa hifi / esp32-s3 are capable of. Maybe TFLite-Micro might come to the esp32-s3's rescue with vector optimisation as what is available and that its closed source is a huge stumbling block for anyone wanting to do ML on the S3