LIFsCode / ELOC-3.0

Firmware for ELOC 3.0 Hardware
MIT License
2 stars 3 forks source link

Regular reboots when CPU running at 240Mhz #104

Open OOHehir opened 1 month ago

OOHehir commented 1 month ago

Following on from #101 is desirable to temporarily increase CPU clock frequency from 80 to 240 Mhz. Running the CPU at 80 & 160 Mhz appears satisfactory but running it at 240Mhz results in regular reboots. The messages occuring when the reboots are generally fairly conistent, interrupt WDT timeout on either cores:

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0).

Backtrace: 0x40094973:0x3ffbc910 0x40095913:0x3ffbc930 0x400972b1:0x3ffbc950

  #0  0x40094973:0x3ffbc910 in cpu_ll_waiti at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/hal/esp32/include/hal/cpu_ll.h:183
      (inlined by) esp_pm_impl_waiti at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/esp_pm/pm_impl.c:828
  #1  0x40095913:0x3ffbc930 in esp_vApplicationIdleHook at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/esp_system/freertos_hooks.c:63
  #2  0x400972b1:0x3ffbc950 in prvIdleTask at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/freertos/tasks.c:3987
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Backtrace: 0x40099b00:0x3ffbc8f0 0x40097124:0x3ffbc930 0x400972b4:0x3ffbc950

  #0  0x40099b00:0x3ffbc8f0 in spinlock_acquire at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/esp_hw_support/include/soc/spinlock.h:124
      (inlined by) xPortEnterCriticalTimeout at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/freertos/port/xtensa/port.c:301
  #1  0x40097124:0x3ffbc930 in vPortEnterCritical at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/freertos/port/xtensa/include/freertos/portmacro.h:578
      (inlined by) prvGetExpectedIdleTime at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/freertos/tasks.c:2468
  #2  0x400972b4:0x3ffbc950 in prvIdleTask at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/freertos/tasks.c:4010
Guru Meditation Error: Core  1 panic'ed (Interrupt wdt timeout on CPU1). 

Backtrace: 0x40094973:0x3ffbd080 0x40095913:0x3ffbd0a0 0x400972b1:0x3ffbd0c0

  #0  0x40094973:0x3ffbd080 in cpu_ll_waiti at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/hal/esp32/include/hal/cpu_ll.h:183
      (inlined by) esp_pm_impl_waiti at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/esp_pm/pm_impl.c:828
  #1  0x40095913:0x3ffbd0a0 in esp_vApplicationIdleHook at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/esp_system/freertos_hooks.c:63
  #2  0x400972b1:0x3ffbd0c0 in prvIdleTask at /home/owen/.platformio/packages/framework-espidf@3.40404.0/components/freertos/tasks.c:3987
OOHehir commented 1 month ago

Increase CONFIG_ESP_INT_WDT_TIMEOUT_MS to a large value (3000) does not improve the situation

OOHehir commented 1 month ago

@EDsteve It seems that the resets at 240Mhz centers around the CPU spending most of its time at idle at sleep. As the CPU is running faster it accomplishes its tasks quicker & hence spends more time asleep. Hence the interrupt WDT doesn't get fed/ reset & times out. This seems to be particularly because CONFIG_FREERTOS_USE_TICKLESS_IDLE was set to y (to enable enhanced power saving).

I've just created another branch . Will you try that? No resets/ crashes on my side so far. Please remember to delete sdkconfig.esp32dev-ei so that it gets rebuilt with the new sdkconfig values.

On another important point, it appears that the max CPU freq is not limited by the config settings:

  "config": {
    "secondsPerFile": 60,
    "cpuMaxFrequencyMHZ": 80,
    "cpuMinFrequencyMHZ": 10,

From my tests the "cpuMaxFrequencyMHZ" is disgregarded by the CPU & the CPU runs at whatever CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ is set to. I'll have to do more testing to confirm.

However at least you should be able to test models at 240Mhz with the current build.

LIFsCode commented 1 month ago

@EDsteve @OOHehir

I'm not too sure but sometimes the esp implementations of freeRTOS get messed up withe the timer configs: sometimes they represent ticks sometimes ms and thus this gets affected by CPU frequency.

CONFIG_FREERTOS_USE_TICKLESS_IDLE can be also disabled using the

"config": { "cpuEnableLightSleep": false }

Did the problems also arise when setting min & max freq to the same value?

OOHehir commented 1 month ago

@LIFsCode @EDsteve I've had mixed success changing the CPU freq.

  1. When trying to change the CPU frequency (from something different to that defined in the sdkconfig) during boot it appeared to fail
  2. When changing the CPU freq during 'routine' program execution I was able to cycle repeatedly from 80Mhz -> 160 -> 240 -> 80 etc without issue.
  3. If CONFIG_FREERTOS_USE_TICKLESS_IDLE=n was set no crashes/ resets were observed.

I'll have to repeat the above to ensure I hadn't made an error somewhere but perhaps there's some issue changing the CPU freq during boot?

@LIFsCode The documentation is a little vague about automatic light sleep. It states:

Automatic light sleep is based on FreeRTOS Tickless Idle functionality. If automatic light sleep is requested while the option CONFIG_FREERTOS_USE_TICKLESS_IDLE is not enabled in menuconfig, esp_pm_configure() will return the error ESP_ERR_NOT_SUPPORTED.

So automatic light sleep is only available when CONFIG_FREERTOS_USE_TICKLESS_IDLE=y ?

Did the problems also arise when setting min & max freq to the same value?

I haven't tried this yet, will hopefully get a chance over the weekend.

EDsteve commented 1 day ago

@LIFsCode @OOHehir

Did the problems also arise when setting min & max freq to the same value?

A few test results. If more info helps. Let me know what else i should test.

Min Max Results
10 240 Reboots when AI is on
80 240 Reboot loop after boot
240 240 No reboots even when AI is on
10 160 No reboots
OOHehir commented 10 hours ago

@EDsteve @LIFsCode I didn't have much success changing the CPU freq during runtime, sometimes it seemed to work OK, sometimes not. Perhaps there's some sort of complication with using the Arduino version of the IDF? I believe (not certain) that it's not possible to change the clock frequency during I2S sampling.

EDsteve commented 3 hours ago

@OOHehir A restart after changing the frequency is totaly acceptable if that makes things easier :) (My test results above are with a restart in between each change)