Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0). (IDFGH-10904)

rlake-mysa commented 1 year ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

4.4.5

Operating System used.

macOS

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32 - Custom Board

Power Supply used.

External 3.3V

What is the expected behavior?

The application to not encounter an interrupt wdt timeout.

What is the actual behavior?

Random panics with Interrupt wdt timeouts.

Steps to reproduce.

Unsure of steps to reproduce at this time. Attempting to build an example project that exhibits the same behaviour.

Debug Logs.

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x4008364c  PS      : 0x00050035  A0      : 0x4008c12d  A1      : 0x3ffbe950  
0x4008364c: _xt_lowint1 at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/xtensa_vectors.S:1114

0x4008c12d: i2c_isr_handler_default at /Users/robert/espressif/esp-idf/release/v4.4.5/components/driver/i2c.c:553

A2      : 0x3ffcc73c  A3      : 0x3ffbe048  A4      : 0x4008364c  A5      : 0x3ffbe910  
0x4008364c: _xt_lowint1 at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/xtensa_vectors.S:1114

A6      : 0x3ffcc73c  A7      : 0x00000000  A8      : 0x00000001  A9      : 0x40092e9a  
0x40092e9a: _frxt_int_enter at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/portasm.S:119

A10     : 0x3ff000dc  A11     : 0x00000001  A12     : 0x00000002  A13     : 0x3ffbe8f0  
A14     : 0x3ffbf5a0  A15     : 0x3ffbe910  SAR     : 0x00000017  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  
Core  0 was running in ISR context:
EPC1    : 0x400d5833  EPC2    : 0x00000000  EPC3    : 0x40094eb8  EPC4    : 0x40094eb8
0x400d5833: uart_hal_write_txfifo at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/uart_hal_iram.c:35

0x40094eb8: i2c_ll_get_intsts_mask at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/esp32/include/hal/i2c_ll.h:244
 (inlined by) i2c_hal_get_intsts_mask at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/i2c_hal_iram.c:73

0x40094eb8: i2c_ll_get_intsts_mask at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/esp32/include/hal/i2c_ll.h:244
 (inlined by) i2c_hal_get_intsts_mask at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/i2c_hal_iram.c:73

Backtrace: 0x40083649:0x3ffbe950 0x4008c12a:0x3ffb6040 0x40093bca:0x3ffb6060 0x40084021:0x3ffb60a0 0x4011bc9e:0x3ffb60c0 0x400932b1:0x3ffb60f0
0x40083649: _xt_lowint1 at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/xtensa_vectors.S:1114

0x4008c12a: i2c_isr_handler_default at /Users/robert/espressif/esp-idf/release/v4.4.5/components/driver/i2c.c:551

0x40093bca: xQueueReceive at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/queue.c:1487

0x40084021: queue_recv_hlevel_wrapper at /Users/robert/espressif/esp-idf/release/v4.4.5/components/bt/controller/esp32/bt.c:816

0x4011bc9e: btdm_controller_task at ??:?

0x400932b1: vPortTaskWrapper at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/port.c:142

Core  1 register dump:
PC      : 0x4020178e  PS      : 0x00060b35  A0      : 0x800d5361  A1      : 0x3ffbc420  
0x4020178e: esp_pm_impl_waiti at /Users/robert/espressif/esp-idf/release/v4.4.5/components/esp_pm/pm_impl.c:832

A2      : 0x00000000  A3      : 0x00000000  A4      : 0x8009348d  A5      : 0x3fff58f0  
A6      : 0x00000003  A7      : 0x00060023  A8      : 0x80178fce  A9      : 0x3ffbc3f0  
A10     : 0x00000000  A11     : 0x3ffc3a18  A12     : 0x3ffc3b0c  A13     : 0x3ffb7dfc  
A14     : 0x3ffc3b0c  A15     : 0x00060023  SAR     : 0x0000001b  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  

Backtrace: 0x4020178b:0x3ffbc420 0x400d535e:0x3ffbc440 0x4020c00d:0x3ffbc460 0x400932b1:0x3ffbc480
0x4020178b: cpu_ll_waiti at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/esp32/include/hal/cpu_ll.h:183
 (inlined by) esp_pm_impl_waiti at /Users/robert/espressif/esp-idf/release/v4.4.5/components/esp_pm/pm_impl.c:830

0x400d535e: esp_vApplicationIdleHook at /Users/robert/espressif/esp-idf/release/v4.4.5/components/esp_system/freertos_hooks.c:63

0x4020c00d: prvIdleTask at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/tasks.c:3987 (discriminator 1)

0x400932b1: vPortTaskWrapper at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/port.c:142

More Information.

The panic randomly occurs, usually with the first 60-70s after launching and only happen on chip revision 1.1. The application executes fine using chip rev 1.0.

rlake-mysa commented 1 year ago

Here's another example of a crash, slightly different back trace:

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x40083647  PS      : 0x00050d35  A0      : 0x40083ced  A1      : 0x3ffbe950  
0x40083647: _xt_lowint1 at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/xtensa_vectors.S:1114

0x40083ced: queue_isr_handler at /Users/robert/espressif/esp-idf/release/v4.4.5/components/bt/controller/esp32/hli_api.c:170

A2      : 0x3ffcc73c  A3      : 0x3ffbe048  A4      : 0x4008bf24  A5      : 0x3ffbe910  
0x4008bf24: i2c_isr_handler_default at /Users/robert/espressif/esp-idf/release/v4.4.5/components/driver/i2c.c:484

A6      : 0x3ffcc73c  A7      : 0x00000000  A8      : 0x00000001  A9      : 0x40092e9a  
0x40092e9a: _frxt_int_enter at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/portasm.S:119

A10     : 0x00000000  A11     : 0x00060021  A12     : 0x00000000  A13     : 0x3ffbe8f0  
A14     : 0x3ffbf5a0  A15     : 0x3ffbe910  SAR     : 0x0000001e  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  
Core  0 was running in ISR context:
EPC1    : 0x400d5833  EPC2    : 0x00000000  EPC3    : 0x4008361c  EPC4    : 0x4008361c
0x400d5833: uart_hal_write_txfifo at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/uart_hal_iram.c:35

0x4008361c: _xt_lowint1 at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/xtensa_vectors.S:1114

0x4008361c: _xt_lowint1 at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/xtensa_vectors.S:1114

Backtrace: 0x40083644:0x3ffbe950 0x40083cea:0x3ffbc020 0x400d535e:0x3ffbc040 0x4020c00d:0x3ffbc060 0x400932b1:0x3ffbc080
0x40083644: _xt_lowint1 at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/xtensa_vectors.S:1114

0x40083cea: queue_isr_handler at /Users/robert/espressif/esp-idf/release/v4.4.5/components/bt/controller/esp32/hli_api.c:168

0x400d535e: esp_vApplicationIdleHook at /Users/robert/espressif/esp-idf/release/v4.4.5/components/esp_system/freertos_hooks.c:63

0x4020c00d: prvIdleTask at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/tasks.c:3987 (discriminator 1)

0x400932b1: vPortTaskWrapper at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/port.c:142

Core  1 register dump:
PC      : 0x4020178e  PS      : 0x00060e35  A0      : 0x800d5361  A1      : 0x3ffbc420  
0x4020178e: esp_pm_impl_waiti at /Users/robert/espressif/esp-idf/release/v4.4.5/components/esp_pm/pm_impl.c:832

A2      : 0x00000000  A3      : 0x00000000  A4      : 0x8009348d  A5      : 0x3fff58c0  
A6      : 0x00000003  A7      : 0x00060023  A8      : 0x80178fce  A9      : 0x3ffbc3f0  
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x8009348d  A13     : 0x3fff58f0  
A14     : 0x00000003  A15     : 0x00060023  SAR     : 0x0000001b  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  

Backtrace: 0x4020178b:0x3ffbc420 0x400d535e:0x3ffbc440 0x4020c00d:0x3ffbc460 0x400932b1:0x3ffbc480
0x4020178b: cpu_ll_waiti at /Users/robert/espressif/esp-idf/release/v4.4.5/components/hal/esp32/include/hal/cpu_ll.h:183
 (inlined by) esp_pm_impl_waiti at /Users/robert/espressif/esp-idf/release/v4.4.5/components/esp_pm/pm_impl.c:830

0x400d535e: esp_vApplicationIdleHook at /Users/robert/espressif/esp-idf/release/v4.4.5/components/esp_system/freertos_hooks.c:63

0x4020c00d: prvIdleTask at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/tasks.c:3987 (discriminator 1)

0x400932b1: vPortTaskWrapper at /Users/robert/espressif/esp-idf/release/v4.4.5/components/freertos/port/xtensa/port.c:142

rlake-mysa commented 1 year ago

I'm able to consistently produce the crash using a minimal program that I published here: https://github.com/rlake-mysa/esp32-1.1-crash.git.

The program is using WiFi + NimBLE and reading from an i2c sensor as fast as possible.

mythbuster5 commented 1 year ago

@rlake-mysa What if there is only i2c goes or there is only wifi&bt goes? Still wdt or both of them works well. Only mix them together fails?

kriegste commented 1 year ago

I have seen a similar crash with I2C access in a tight loop (without WiFi, without BT).

rlake-mysa commented 1 year ago

@mythbuster5 I'm testing now with only i2c (wifi & bt not enabled). I will report status later today regarding the outcome.

rlake-mysa commented 1 year ago

@mythbuster5 I'm testing now with only i2c (wifi & bt not enabled). I will report status later today regarding the outcome.

After testing for a few hours the device would only crash with I2C+WiFi+BLE.

moltek commented 11 months ago

Is there any update on this issue? I've got a similar problem. After restarting the device esp_restart() I have a continuous problem with Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0). And is creating boot-loop, only power cut helping. I also use WIFI, BLE, UART and I2C.

rlake-mysa commented 8 months ago

I think I have found a 'workaround' for this issue. I pinned the I2C task to core 0 - the same as the main task. There must be some constraint that requires I2C access to be done on the same core as the I2C driver was initialized. After making that change the application is stable.

hiruna commented 8 months ago

@rlake-mysa i just started having this issue today with the latest esp-idf (master) I2C driver.

I'm only trying to initialize the master bus with cfg:

i2c_master_bus_config_t i2c_master_conf = {
            .clk_source = I2C_CLK_SRC_DEFAULT,
            .i2c_port = I2C_NUM_0,
            .sda_io_num = GPIO_NUM_21,
            .scl_io_num = GPIO_NUM_22,
            .flags.enable_internal_pullup = true,
            .glitch_ignore_cnt = 9,
    };

i2c_new_master_bus function fails when i2c device is connected to power, weird, sometimes the function works and other times just causes the wdt panic

kaffepanna commented 7 months ago

@rlake-mysa i just started having this issue today with the latest esp-idf (master) I2C driver.

I'm only trying to initialize the master bus with cfg:
i2c_master_bus_config_t i2c_master_conf = {
            .clk_source = I2C_CLK_SRC_DEFAULT,
            .i2c_port = I2C_NUM_0,
            .sda_io_num = GPIO_NUM_21,
            .scl_io_num = GPIO_NUM_22,
            .flags.enable_internal_pullup = true,
            .glitch_ignore_cnt = 9,
    };
i2c_new_master_bus function fails when i2c device is connected to power, weird, sometimes the function works and other times just causes the wdt panic

Seeing the exact same thing

mythbuster5 commented 7 months ago

i2c_new_master_bus This function will cause panic? This is a non-blocking interface, where it might stuck in? I have never met before, @kaffepanna could you please give more clue?

mbratch commented 6 months ago

i2c_new_master_bus This function will cause panic? This is a non-blocking interface, where it might stuck in? I have never met before, @kaffepanna could you please give more clue?

@mythbuster5 see https://github.com/espressif/esp-idf/issues/12929. I am getting the same error with ESP-IDF 5.2.0. I have posted the stack trace in the comments of that issue.

FranckFreiburger commented 5 months ago

Hello, I also getting this error randomly.

ESP-IDF 5.2.1

I2C port configuration :

.i2c_port = 0,
.clk_source = I2C_CLK_SRC_DEFAULT,
.intr_priority = 0,
.sda_io_num = 21
.scl_io_num = 22
.glitch_ignore_cnt = 7
.flags.enable_internal_pullup = true

The Backtrace is :


I (851) gpio: GPIO[21]| InputEn: 1| OutputEn: 1| OpenDrain: 1| Pullup: 1| Pulldown: 0| Intr:0 
I (851) gpio: GPIO[22]| InputEn: 1| OutputEn: 1| OpenDrain: 1| Pullup: 1| Pulldown: 0| Intr:0 
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x40081ae0  PS      : 0x00060034  A0      : 0x80081cdc  A1      : 0x3ffb20c0  
A2      : 0x3ffefb58  A3      : 0x00000100  A4      : 0x3ffefe7c  A5      : 0x00060023  
A6      : 0xb33fffff  A7      : 0xb33fffff  A8      : 0x00000400  A9      : 0x3ffb20b0
A10     : 0x00000001  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x3ffdbac0  
A14     : 0x00000003  A15     : 0x00060323  SAR     : 0x0000000f  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000
Core  0 was running in ISR context:
EPC1    : 0x4013b177  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x40081ae0

Backtrace: 0x40081add:0x3ffb20c0 0x40081cd9:0x3ffb2100 0x40088539:0x3ffb2120 0x4012e70b:0x3ffdbb40 0x400d2f9b:0x3ffdbb90 0x400eedd9:0x3ffdbbe0 0x400ef6fb:0x3ffdbc10 0x400ef90d:0x3ffdbc40 0x40108d16:0x3ffdbc60 0x40109042:0x3ffdbc80 0x401090c9:0x3ffdbca0 0x401091a5:0x3ffdbd10 0x400e5cf2:0x3ffdbd40 0x400de930:0x3ffdbd60 0x400d1979:0x3ffdbd80 0x400d2a25:0x3ffdbdc0 0x4008c37c:0x3ffdbde0 0x4008f9b1:0x3ffdbe10

  #0  0x40081add:0x3ffb20c0 in i2c_master_isr_handler_default at C:\Users\Utilisateur\.platformio\packages\framework-espidf\components\driver\i2c/i2c_master.c:578
  #1  0x40081cd9:0x3ffb2100 in shared_intr_isr at C:/Users/Utilisateur/.platformio/packages/framework-espidf/components/esp_hw_support/intr_alloc.c:443
  #2  0x40088539:0x3ffb2120 in _xt_lowint1 at C:\Users\Utilisateur\.platformio\packages\framework-espidf\components\xtensa/xtensa_vectors.S:1240
  #3  0x4012e70b:0x3ffdbb40 in i2c_new_master_bus at C:\Users\Utilisateur\.platformio\packages\framework-espidf\components\driver\i2c/i2c_master.c:865 (discriminator 2)
...

hape65 commented 3 months ago

hi i got this error on combination of I2C and Interrupt set Button action. It comes if i press button a few times than suddenly it gets this error: Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0)

Core 0 register dump: MEPC : 0x40805c24 RA : 0x408075e8 SP : 0x4080d150 GP : 0x4080b790
TP : 0x40809d64 T0 : 0x7fffffff T1 : 0x42001276 T2 : 0x00000002
S0/FP : 0xffffffff S1 : 0x00000001 A0 : 0x4087db0c A1 : 0x4087d4fc
A2 : 0x00000018 A3 : 0x00000018 A4 : 0x4087d4fc A5 : 0x4087d4fc
A6 : 0xfa000000 A7 : 0x00000003 S2 : 0x00000000 S3 : 0xffffffff
S4 : 0x4087db0c S5 : 0x00000001 S6 : 0x00000000 S7 : 0x00000000
S8 : 0x00000000 S9 : 0x00000000 S10 : 0x00000000 S11 : 0x00000000
T3 : 0x4080d6e8 T4 : 0x00000000 T5 : 0x00000001 T6 : 0x00000000
MSTATUS : 0x00001881 MTVEC : 0x40800001 MCAUSE : 0x00000018 MTVAL : 0x00004310
MHARTID : 0x00000000

thats the button definition:

void IRAM_ATTR isr() {
  if (button1.pressed) {
      button1.pressed = false;
    Serial.println("button1 pressed false");
  }
  else {
    button1.pressed = true;
    Serial.println("button1 pressed true");
  }
}

the I2C is a pca9685 with adafruit library #include

So i think there is a bug causing collision of i2c and interrupt. hape

kriegste commented 3 months ago

Remove the "Serial.println" from your ISR and try again. That should never be placed inside an ISR.

hape65 commented 3 months ago

@kriegste thats totally true a noob error :-). I have searched and found the explanation and that makes totally sense. If someone reads this and needs explanation here it is: https://forum.arduino.cc/t/serial-print-inside-interrupt/537369/10 That also let me see things different with main loop and interrupts.

espressif / esp-idf