search

raspberrypi / pico-examples

BSD 3-Clause "New" or "Revised" License
2.87k stars 823 forks source link

picow_freertos_ping_sys: hardfault on pico-w #302

Closed plebioda closed 1 year ago

plebioda commented 1 year ago

I get the hardfault when trying to run picow_freertos_ping_sys on pico-w, the picow_freertos_ping_nosys works fine.

Starting FreeRTOS SMP on both cores:
Version: 7.95.49 (2271bb6 CY) CRC: b7a28ef3 Date: Mon 2021-11-29 22:50:27 PST Ucode Ver: 1043.2162 FWID 01-c51d9400
cyw43 loaded ok, mac 28:cd:c1:03:e8:e0
API: 12.2
Data: RaspberryPi.PicoW
Compiler: 1.29.4
ClmImport: 1.47.1
Customization: v5 22/06/24
Creation: 2022-06-24 06:55:08
Connecting to WiFi...
connect status: joining
connect status: no ip
connect status: link up
Connected.
ping: send 142.251.35.196

Backtrace:

Starting program: /home/plebioda/pico/pico-examples/build/pico_w/freertos/ping/picow_freertos_ping_sys.elf
[New Thread 2]

Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
isr_hardfault () at /home/plebioda/pico/pico-sdk/src/rp2_common/pico_standard_link/crt0.S:98
98      decl_isr_bkpt isr_hardfault
(gdb) bt
#0  isr_hardfault () at /home/plebioda/pico/pico-sdk/src/rp2_common/pico_standard_link/crt0.S:98
#1  <signal handler called>
#2  uxListRemove (pxItemToRemove=0x2000f4fc <ucHeap+9620>) at /home/plebioda/pico/FreeRTOS-Kernel/list.c:193
#3  0x10022800 in xTaskIncrementTick () at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:3635
#4  0x100200da in xPortSysTickHandler () at /home/plebioda/pico/FreeRTOS-Kernel/portable/ThirdParty/GCC/RP2040/port.c:507
#5  <signal handler called>
#6  0x10022e06 in prvMinimalIdleTask (pvParameters=0x0) at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4332
#7  0x1001fd70 in pxPortInitialiseStack (pxTopOfStack=0x1001fec7 <prvDisableInterruptsAndPortStartSchedulerOnCore+18>, pxCode=0x20040fd8, pvParameters=0x10) at /home/plebioda/pico/FreeRTOS-Kernel/portable/ThirdParty/GCC/RP2040/port.c:187
#8  0x00000000 in ?? ()
(gdb) t 2  
[Switching to thread 2 (Thread 2)]
#0  0x10022f28 in prvCheckTasksWaitingTermination () at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4690
4690    }
(gdb) bt
#0  0x10022f28 in prvCheckTasksWaitingTermination () at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4690
#1  0x10022e28 in prvIdleTask (pvParameters=0x0) at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4390
#2  0x1001fd70 in pxPortInitialiseStack (pxTopOfStack=0x1001ff21 <xPortStartScheduler+80>, pxCode=0x20041fc0, pvParameters=0xf) at /home/plebioda/pico/FreeRTOS-Kernel/portable/ThirdParty/GCC/RP2040/port.c:187
#3  0x100224ec in vTaskStartScheduler () at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:2888
#4  0x10000410 in vLaunch () at /home/plebioda/pico/pico-examples/pico_w/freertos/ping/picow_freertos_ping.c:63
#5  0x1000043e in main () at /home/plebioda/pico/pico-examples/pico_w/freertos/ping/picow_freertos_ping.c:80

In list.c:193 the pxItemToRemove->pxPrevious pointer is invalid:

(gdb) p *pxItemToRemove
$3 = {xItemValue = 31, pxNext = 0x2000f024 <ucHeap+8380>, pxPrevious = 0x1, pvOwner = 0x4, pvContainer = 0x1}

Using:

peterharperuk commented 1 year ago

Probably a stack overflow

plebioda commented 1 year ago

After increasing the configMINIMAL_STACK_SIZE to 512 the hardfault is gone but the example still does not work. It gets stuck after sending the first ping. Whenever I interrupt the execution with gbd both threads seem to be in the idle task:

(gdb) info thread
  Id   Target Id                                           Frame
  1    Thread 2 (Name: rp2040.core1, state: debug-request) prvMinimalIdleTask (pvParameters=0x0) at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4332
* 2    Thread 1 (Name: rp2040.core0, state: debug-request) prvCheckTasksWaitingTermination () at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4649
(gdb) bt
#0  prvCheckTasksWaitingTermination () at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4649
#1  0x10022e28 in prvIdleTask (pvParameters=0x0) at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4390
#2  0x1001fd70 in pxPortInitialiseStack (pxTopOfStack=0x1001ff21 <xPortStartScheduler+80>, pxCode=0x20041fc0, pvParameters=0xf) at /home/plebioda/pico/FreeRTOS-Kernel/portable/ThirdParty/GCC/RP2040/port.c:187
#3  0x100224ec in vTaskStartScheduler () at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:2888
#4  0x10000410 in vLaunch () at /home/plebioda/pico/pico-examples/pico_w/freertos/ping/picow_freertos_ping.c:63
#5  0x1000043e in main () at /home/plebioda/pico/pico-examples/pico_w/freertos/ping/picow_freertos_ping.c:80
(gdb) t 1
[Switching to thread 1 (Thread 2)]
#0  prvMinimalIdleTask (pvParameters=0x0) at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4332
4332                        if( listCURRENT_LIST_LENGTH( &( pxReadyTasksLists[ tskIDLE_PRIORITY ] ) ) > ( UBaseType_t ) configNUM_CORES )
(gdb) bt
#0  prvMinimalIdleTask (pvParameters=0x0) at /home/plebioda/pico/FreeRTOS-Kernel/tasks.c:4332
#1  0x1001fd70 in pxPortInitialiseStack (pxTopOfStack=0x1001fec7 <prvDisableInterruptsAndPortStartSchedulerOnCore+18>, pxCode=0x20040fd8, pvParameters=0x10) at /home/plebioda/pico/FreeRTOS-Kernel/portable/ThirdParty/GCC/RP2040/port.c:187
#2  0x00000000 in ?? ()
plebioda commented 1 year ago

I get the stack overflow notification from FreeRTOS for configCHECK_FOR_STACK_OVERFLOW=2 in 'CYW43 Worker'. When I change the configMINIMAL_STACK_SIZE to 512, 1024, 4096 the stack overflow occurs in 'ping_thread'.

Starting FreeRTOS SMP on both cores:
Version: 7.95.49 (2271bb6 CY) CRC: b7a28ef3 Date: Mon 2021-11-29 22:50:27 PST Ucode Ver: 1043.2162 FWID 01-c51d9400
cyw43 loaded ok, mac 28:cd:c1:03:e8:e0
API: 12.2
Data: RaspberryPi.PicoW
Compiler: 1.29.4
ClmImport: 1.47.1
Customization: v5 22/06/24
Creation: 2022-06-24 06:55:08
Connecting to WiFi...
connect status: joining
connect status: no ip
[core 0] Stack overflow in task 'CYW43 Worker'
[core 0] Waiting for debugger...

My patch:

diff --git a/pico_w/freertos/ping/CMakeLists.txt b/pico_w/freertos/ping/CMakeLists.txt
index a76991c..139adda 100644
--- a/pico_w/freertos/ping/CMakeLists.txt
+++ b/pico_w/freertos/ping/CMakeLists.txt
@@ -16,6 +16,7 @@ if (EXISTS ${PICO_LWIP_CONTRIB_PATH}/apps/ping/ping.c)
             pico_cyw43_arch_lwip_threadsafe_background
             pico_stdlib
             pico_lwip_iperf
+            hardware_exception
             FreeRTOS-Kernel-Heap4 # FreeRTOS kernel and dynamic heap
             )
     pico_add_extra_outputs(picow_freertos_ping_nosys)
@@ -40,6 +41,7 @@ if (EXISTS ${PICO_LWIP_CONTRIB_PATH}/apps/ping/ping.c)
             pico_cyw43_arch_lwip_sys_freertos
             pico_stdlib
             pico_lwip_iperf
+            hardware_exception
             FreeRTOS-Kernel-Heap4 # FreeRTOS kernel and dynamic heap
             )
     pico_add_extra_outputs(picow_freertos_ping_sys)
diff --git a/pico_w/freertos/ping/FreeRTOSConfig.h b/pico_w/freertos/ping/FreeRTOSConfig.h
index f9f075d..0de1ed3 100644
--- a/pico_w/freertos/ping/FreeRTOSConfig.h
+++ b/pico_w/freertos/ping/FreeRTOSConfig.h
@@ -76,7 +76,7 @@
 #define configAPPLICATION_ALLOCATED_HEAP        0

 /* Hook function related definitions. */
-#define configCHECK_FOR_STACK_OVERFLOW          0
+#define configCHECK_FOR_STACK_OVERFLOW          2
 #define configUSE_MALLOC_FAILED_HOOK            0
 #define configUSE_DAEMON_TASK_STARTUP_HOOK      0

diff --git a/pico_w/freertos/ping/picow_freertos_ping.c b/pico_w/freertos/ping/picow_freertos_ping.c
index e2e38a1..8260d6e 100644
--- a/pico_w/freertos/ping/picow_freertos_ping.c
+++ b/pico_w/freertos/ping/picow_freertos_ping.c
@@ -6,6 +6,7 @@

 #include "pico/cyw43_arch.h"
 #include "pico/stdlib.h"
+#include "hardware/exception.h"

 #include "lwip/ip4_addr.h"

@@ -63,10 +64,32 @@ void vLaunch( void) {
     vTaskStartScheduler();
 }

+#define log_core_num(fmt, ...) printf("[core %d] " fmt, portGET_CORE_ID(), ##__VA_ARGS__)
+
+static void waitForDebugger()
+{
+    log_core_num("Waiting for debugger...\n");
+    __breakpoint();
+}
+
+void vApplicationStackOverflowHook(TaskHandle_t xTask, char * pcTaskName)
+{
+    log_core_num("Stack overflow in task '%s'\n", pcTaskName);
+    waitForDebugger();
+}
+
+static void hardfault_handler()
+{
+    log_core_num("hardfault!!!\n");
+    waitForDebugger();
+}
+
 int main( void )
 {
     stdio_init_all();

+    exception_set_exclusive_handler(HARDFAULT_EXCEPTION, hardfault_handler);
+
     /* Configure the hardware ready to run the demo. */
     const char *rtos_name;
 #if ( portSUPPORT_SMP == 1 )
plebioda commented 1 year ago

Is there any chance to get some support here?

peterharperuk commented 1 year ago

It works for me. Can you try the develop branch? You probably need this https://github.com/raspberrypi/pico-examples/pull/309

peterharperuk commented 1 year ago

Closing this. Please reopen if you can repro on develop.

ArcaEge commented 11 months ago

I have the exact same issue.

peterharperuk commented 11 months ago

Does increasing the stack solve the problem?

ArcaEge commented 11 months ago

Sorry for not specifying, I got the hardfault in a part of my own code that is very similar to the example. It looks like it might be linked in some way with raspberrypi/pico-sdk#1526 as it was seemingly working at stack size 256 once I added #define configNUM_CORES configNUMBER_OF_CORES to the FreeRTOS config. It also worked at 512 when I commented out the asserts in cyw43_arch.c, so my guess is that fixing the assertion error also fixes the stack overflow in some way.

Edit: Turns out it's not the #define, it's that I had switched FreeRTOS to single-core mode while trying to debug it and forgot to switch back to dual-core SMP (whoops). It looks like it works at 256 when it's single-core but has to be 512 when using SMP.