search

PX4 / PX4-Autopilot

PX4 Autopilot Software
https://px4.io
BSD 3-Clause "New" or "Revised" License
8.47k stars 13.5k forks source link

OrangeCube: Check for stack overflow on each function call causes hardfault #17235

Open dgeorge83616 opened 3 years ago

dgeorge83616 commented 3 years ago

Describe the bug Debug serial console at boot:

[2021-03-25 13:50:01] [boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[2021-03-25 13:50:01] [boot] Fault Log is Armed
[2021-03-25 13:50:01] arm_hardfault: Hard Fault:
[2021-03-25 13:50:01] arm_hardfault:   IRQ: 3 regs: 0x37ffff04
[2021-03-25 13:50:01] arm_hardfault:   BASEPRI: 000000f0 PRIMASK: 00000000 IPSR: 00000003 CONTROL: 00000000
[2021-03-25 13:50:01] arm_hardfault:   CFAULTS: 00000400 HFAULTS: 40000000 DFAULTS: 00000000 BFAULTADDR: 00000000 AFAULTS: 00000000
[2021-03-25 13:50:01] arm_hardfault: PANIC!!! Hard fault: 40000000
[2021-03-25 13:50:01] up_assert: Assertion failed at file:armv7-m/arm_hardfault.c line: 134 task: l"

To Reproduce See below.

Expected behavior Normal boot.

Log Files and Screenshots

See slack thread here. @dagar suggested enabling stack checking to help diagnose, but first, tried updated master as discussed here.

Decided to verify stack checking would work on master, without any changes, other than enabling the checking and discovered the above.

Drone (please complete the following information):

Additional context What follows are captures from tmux log leading to discovering the fault.

After attempting to merge #16444 into master branch, and failing to cleanly resolve conflicts, I decided to just enable stack checking on the master branch.

dan@ubuntu:~/projects/px4-community$ git reset --hard HEAD                                                                                                                                                                                                                         
HEAD is now at beadae6dfc process_sensor_caldata.py: Fix 2nd barometer plot                                                                                                                                                                                                        
dan@ubuntu:~/projects/px4-community$ git status                                                                                                                                                                                                                                    
On branch master                                                                                                                                                                                                                                                                   
Your branch is up to date with 'origin/master'.                                                                                                                                                                                                                                    

nothing to commit, working tree clean                                                                                                                                                                                                                                              
dan@ubuntu:~/projects/px4-community$ git log                                                                                                                                                                                                                                       
commit beadae6dfc032079bd1d0d4300a8854e45eae39f (HEAD -> master, origin/master, origin/HEAD)                                                                                                                                                                                       
Author: Serhat Aksun <serhat.aksun@maxwell-innovations.com>                                                                                                                                                                                                                        
Date:   Mon Mar 22 14:42:35 2021 +0300

distclean the workspace.

dan@ubuntu:~/projects/px4-community$ make distclean                                                                                                                                                                                                                                
Cleared directory 'Tools/flightgear_bridge'                                                                                                                                                                                                                                        
warning: Could not unset core.worktree setting in submodule 'Tools/flightgear_bridge'                                                                                                                                                                                              
Cleared directory 'Tools/jMAVSim'                                                                                                                                                                                                                                                  
warning: Could not unset core.worktree setting in submodule 'Tools/jMAVSim'                                                                                                                                                                                                        
Cleared directory 'Tools/jsbsim_bridge'                                                                                                                                                                                                                                            
warning: Could not unset core.worktree setting in submodule 'Tools/jsbsim_bridge'                                                                                                                                                                                                  
Cleared directory 'Tools/sitl_gazebo'                                                                                                                                                                                                                                              
warning: Could not unset core.worktree setting in submodule 'Tools/sitl_gazebo'                                                                                                                                                                                                    
Cleared directory 'boards/atlflight/cmake_hexagon'
warning: Could not unset core.worktree setting in submodule 'boards/atlflight/cmake_hexagon'
Cleared directory 'mavlink/include/mavlink/v2.0'
Submodule 'mavlink/include/mavlink/v2.0' (https://github.com/mavlink/c_library_v2.git) unregistered for path 'mavlink/include/mavlink/v2.0'
Cleared directory 'platforms/nuttx/NuttX/apps'
Submodule 'platforms/nuttx/NuttX/apps' (https://github.com/PX4/NuttX-apps.git) unregistered for path 'platforms/nuttx/NuttX/apps'
Cleared directory 'platforms/nuttx/NuttX/nuttx'
Submodule 'platforms/nuttx/NuttX/nuttx' (https://github.com/PX4/NuttX.git) unregistered for path 'platforms/nuttx/NuttX/nuttx'
Cleared directory 'platforms/qurt/dspal'
warning: Could not unset core.worktree setting in submodule 'platforms/qurt/dspal'
Cleared directory 'src/drivers/gps/devices'
Submodule 'src/drivers/gps/devices' (https://github.com/PX4/PX4-GPSDrivers.git) unregistered for path 'src/drivers/gps/devices'
Cleared directory 'src/drivers/uavcan/libuavcan'
Submodule 'src/drivers/uavcan/libuavcan' (https://github.com/PX4/libuavcan.git) unregistered for path 'src/drivers/uavcan/libuavcan'
Cleared directory 'src/drivers/uavcan_v1/libcanard'
warning: Could not unset core.worktree setting in submodule 'src/drivers/uavcan_v1/libcanard'
Cleared directory 'src/drivers/uavcan_v1/public_regulated_data_types'
warning: Could not unset core.worktree setting in submodule 'src/drivers/uavcan_v1/public_regulated_data_types'
Cleared directory 'src/drivers/uavcannode_gps_demo/libcanard'
warning: Could not unset core.worktree setting in submodule 'src/drivers/uavcannode_gps_demo/libcanard'
Cleared directory 'src/drivers/uavcannode_gps_demo/public_regulated_data_types'
warning: Could not unset core.worktree setting in submodule 'src/drivers/uavcannode_gps_demo/public_regulated_data_types'
Cleared directory 'src/lib/ecl'
Submodule 'src/lib/ecl' (https://github.com/PX4/PX4-ECL.git) unregistered for path 'src/lib/ecl'
Cleared directory 'src/lib/matrix'
Submodule 'src/lib/matrix' (https://github.com/PX4/PX4-Matrix.git) unregistered for path 'src/lib/matrix'
Cleared directory 'src/modules/micrortps_bridge/micro-CDR'
warning: Could not unset core.worktree setting in submodule 'src/modules/micrortps_bridge/micro-CDR'
Removing Tools/px4airframes/__pycache__/
Removing boards/cubepilot/cubeorange/nuttx-config/console/.defconfig.un~
Removing build/
Removing msg/tools/__pycache__/
Removing platforms/nuttx/NuttX/tools/__pycache__/
Removing src/lib/parameters/px4params/__pycache__/
dan@ubuntu:~/projects/px4-community$ make cubepilot_cubeorange_console

snipped out the build log.

          BKPRAM:          0 GB         4 KB      0.00%
[254/254] Creating /home/dan/projects/px4-community/build/cubepilot_cubeorange_console/external/Build/bootloader_firmware/cubepilot_cubeorange_bootloader.px4
[1347/1349] Linking CXX executable cubepilot_cubeorange_console.elf                                                                       
Memory region         Used Size  Region Size  %age Used
        ITCM_RAM:          0 GB        64 KB      0.00%
           FLASH:     1900245 B      1920 KB     96.65%
       DTCM1_RAM:          0 GB        64 KB      0.00%
       DTCM2_RAM:          0 GB        64 KB      0.00%
        AXI_SRAM:       47752 B       512 KB      9.11%
           SRAM1:          0 GB       128 KB      0.00%
           SRAM2:          0 GB       128 KB      0.00%
           SRAM3:          0 GB        32 KB      0.00%
           SRAM4:          0 GB        64 KB      0.00%
          BKPRAM:          0 GB         4 KB      0.00%
[1349/1349] Creating /home/dan/projects/px4-community/build/cubepilot_cubeorange_console/cubepilot_cubeorange_console.px4
dan@ubuntu:~/projects/px4-community$

All good, so far. Now enable stack checking. Used make menuconfig to enable [*] Check for stack overflow on each function call.

dan@ubuntu:~/projects/px4-community$ cd build/cubepilot_cubeorange_console/NuttX/nuttx
dan@ubuntu:~/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx$ make menuconfig
LN: include/arch/board to /home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/../nuttx-config/include
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/libs/libxx'
make[1]: Nothing to be done for 'dirlinks'.
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/libs/libxx'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/boards'
make[1]: Nothing to be done for 'dirlinks'.
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/boards'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/openamp'
make[1]: Nothing to be done for 'dirlinks'.
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/openamp'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'
make[2]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps/platform'
LN: platform/board to /home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps/platform/dummy
make[2]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps/platform'
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'
make[1]: Nothing to be done for 'preconfig'.
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'

*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

dan@ubuntu:~/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx$ cd -                                                                                                                                                                                           
/home/dan/projects/px4-community

Rebuild

dan@ubuntu:~/projects/px4-community$ make cubepilot_cubeorange_console                                                                    
[0/1304] Performing build step for 'px4io_firmware'                 
ninja: no work to do.                                                                                                                    
[1/1298] Performing build step for 'bootloader_firmware'                                                                                 
ninja: no work to do.                          
[1296/1298] Linking CXX executable cubepilot_cubeorange_console.elf                                                                       
Memory region         Used Size  Region Size  %age Used                                                                                  
        ITCM_RAM:          0 GB        64 KB      0.00%             
           FLASH:     1950277 B      1920 KB     99.20%                                                                                  
       DTCM1_RAM:          0 GB        64 KB      0.00%                                                                                  
       DTCM2_RAM:          0 GB        64 KB      0.00%                                                                                  
        AXI_SRAM:       47752 B       512 KB      9.11%                                                                                  
           SRAM1:          0 GB       128 KB      0.00%                                                                                  
           SRAM2:          0 GB       128 KB      0.00%                                                                                  
           SRAM3:          0 GB        32 KB      0.00%                                                                                  
           SRAM4:          0 GB        64 KB      0.00%
          BKPRAM:          0 GB         4 KB      0.00%                                                                                  
[1298/1298] Creating /home/dan/projects/px4-community/build/cubepilot_cubeorange_console/cubepilot_cubeorange_console.px4                
dan@ubuntu:~/projects/px4-community$

Phew! Just fits into FLASH.

Verify stack check is the only difference from boards/cubepilot/cubeorange/nuttx/console/defconfig

dan@ubuntu:~/projects/px4-community$ cd build/cubepilot_cubeorange_console/NuttX/nuttx/                                                   
dan@ubuntu:~/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx$ make savedefconfig
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/libs/libxx'
make[1]: Nothing to be done for 'dirlinks'.
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/libs/libxx'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/boards'
make[1]: Nothing to be done for 'dirlinks'.    
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/boards'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/openamp'
make[1]: Nothing to be done for 'dirlinks'.
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx/openamp'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'
make[2]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps/platform'
LN: platform/board to /home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps/platform/dummy
make[2]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps/platform'
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'
make[1]: Entering directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'
make[1]: Nothing to be done for 'preconfig'.  
make[1]: Leaving directory '/home/dan/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/apps'
dan@ubuntu:~/projects/px4-community/build/cubepilot_cubeorange_console/NuttX/nuttx$ cd -
dan@ubuntu:~/projects/px4-community$ diff boards/cubepilot/cubeorange/nuttx-config/console/defconfig build/cubepilot_cubeorange_console/NuttX/nuttx/defconfig
38a39
> CONFIG_ARMV7M_STACKCHECK=y
207d207
< CONFIG_TTY_SIGINT_CHAR=0x03

What's with the CONFIG_TTY_SIGINT_CHAR=0x03 difference? Not sure why but savedefconfig cut it, even though it remained in .config.

dan@ubuntu:~/projects/px4-community$ grep TTY_SIGINT_CHAR boards/cubepilot/cubeorange/nuttx-config/console/defconfig build/cubepilot_cubeorange_console/NuttX/nuttx/defconfig
boards/cubepilot/cubeorange/nuttx-config/console/defconfig:CONFIG_TTY_SIGINT_CHAR=0x03
dan@ubuntu:~/projects/px4-community$ grep TTY_SIGINT_CHAR build/cubepilot_cubeorange_console/NuttX/nuttx/.config
CONFIG_TTY_SIGINT_CHAR=0x03
dan@ubuntu:~/projects/px4-community$

Longer listing of workspace log for double checking:

commit beadae6dfc032079bd1d0d4300a8854e45eae39f (HEAD -> master, origin/master, origin/HEAD)
Author: Serhat Aksun <serhat.aksun@maxwell-innovations.com>
Date:   Mon Mar 22 14:42:35 2021 +0300

    process_sensor_caldata.py: Fix 2nd barometer plot

    fixed the figure numbers to avoid plot conflicts

commit 5a1eb3f9eff6510c0eb6f93b9338c65832b80150
Author: Daniel Agar <daniel@agar.ca>
Date:   Thu Mar 25 00:25:29 2021 -0400

    Update submodule GPSDrivers to latest Thu Mar 25 00:41:54 UTC 2021

commit 98e1e25d93217f76b567c9fe2401e62ed132fd28
Author: PX4 BuildBot <bot@px4.io>
Date:   Thu Mar 25 00:41:48 2021 +0000

    Update submodule sitl_gazebo to latest Thu Mar 25 00:41:48 UTC 2021

        - sitl_gazebo in PX4/Firmware (7f2566dd8a2158325a6ad57dcb5250e1056165e9): https://github.com/PX4/PX4-SITL_gazebo/commit/1f339cdf5c5da27097e58ec2beb2793bf6057c7d
        - sitl_gazebo current upstream: https://github.com/PX4/PX4-SITL_gazebo/commit/3e5fed04d8e574b10e17e446d2938346bc6152ca
        - Changes: https://github.com/PX4/PX4-SITL_gazebo/compare/1f339cdf5c5da27097e58ec2beb2793bf6057c7d...3e5fed04d8e574b10e17e446d2938346bc6152ca

        3e5fed0 2021-03-22 JaeyoungLim - Multiple airspeed sensors in SITL with multi HIL_SENSOR instances (#731)

commit 7f2566dd8a2158325a6ad57dcb5250e1056165e9
Author: David Sidrane <David.Sidrane@Nscdg.com>
Date:   Wed Mar 24 16:29:40 2021 -0700

    NuttX STM32H7 16 bit SPI fixes + ADIS16470 updates for proper 16 bit mode in burst

commit 92910de81d66480a9f22cd9237200fbacac10122
julianoes commented 1 year ago

I'm just trying to reproduce this but I'm not sure how this would possibly fit into flash? :astonished: 

/usr/lib/gcc/arm-none-eabi/10.3.1/../../../arm-none-eabi/bin/ld: cubepilot_cubeorange_default.elf section `.text' will not fit in region `FLASH'
/usr/lib/gcc/arm-none-eabi/10.3.1/../../../arm-none-eabi/bin/ld: region `FLASH' overflowed by 938840 bytes