search

meshtastic / firmware

Meshtastic device firmware
https://meshtastic.org
GNU General Public License v3.0
3.37k stars 825 forks source link

TCP Connections to 1.1.20 firmware and greater causes device crash upon interface close. #598

Closed ScriptBlock closed 3 years ago

ScriptBlock commented 3 years ago

See thread: https://meshtastic.discourse.group/t/python-api-rebooting-device-on-every-connection/1981/3

For repro/video of crash.

Here's the link to the serial crash detail. Appears to begin about line 154. https://drive.google.com/file/d/1GnGpgWgNZzkTA9GYdWZRMFFH2Syl_JfQ/view?usp=sharing

ScriptBlock commented 3 years ago 
CORRUPT HEAP: Bad head at 0x3ffe9874. Expected 0xabba1234 got 0x3ffe9f5c
abort() was called at PC 0x40085b03 on core 1

ELF file SHA256: 0000000000000000

Backtrace: 0x4008ff5f:0x3ffd1410 0x400902e5:0x3ffd1430 0x40085b03:0x3ffd1450 0x40085c1d:0x3ffd1490 0x4011a4e7:0x3ffd14b0 0x40114acd:0x3ffd1770 0x4010b959:0x3ffd17c0 0x400943af:0x3ffd17f0 0x40086151:0x3ffd1810 0x400859f9:0x3ffd1830 0x4000bec7:0x3ffd1850 0x401a69a9:0x3ffd1870 0x400d4ae1:0x3ffd1890 0x400dbd15:0x3ffd18b0 0x400d5679:0x3ffd18d0 0x400d56b2:0x3ffd1920 0x400d4d1e:0x3ffd1940 0x400eb6b1:0x3ffd1960 0x400d9dc3:0x3ffd1990 0x400fa77d:0x3ffd19b0

Rebooting...
geeksville commented 3 years ago

Serial log of the failure:

08:21:40 [ApiServer] Incoming connection from 192.168.10.240
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x593d8c16 Fr0x58 To0xff, WantAck0, HopLim3 Portnum=3 WANTRESP)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 30 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x593d8c17 Fr0x58 To0x58, WantAck0, HopLim3 Portnum=3)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 28 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x593d8c18 Fr0x58 To0x58, WantAck0, HopLim3 Portnum=3)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 28 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x593d8c19 Fr0x58 To0xff, WantAck0, HopLim3 Portnum=4 WANTRESP)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 74 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x593d8c1a Fr0x58 To0x58, WantAck0, HopLim3 Portnum=4)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 72 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x593d8c1b Fr0x58 To0x58, WantAck0, HopLim3 Portnum=4)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 72 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x21d38a8a Fr0x98 To0x58, WantAck0, HopLim3 Portnum=3 rxSNR=-3.75)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 57 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x21d38a8b Fr0x98 To0x58, WantAck0, HopLim3 Portnum=4 rxSNR=-2)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 71 bytes
08:21:40 [ApiServer] getFromRadio, state=0
08:21:40 [ApiServer] phone downloaded packet (id=0x21d38a8c Fr0x98 To0xff, WantAck0, HopLim2 Portnum=3 rxSNR=-2.5)
08:21:40 [ApiServer] encoding toPhone packet to phone variant=2, 57 bytes
Trigger powerFSM 9
Trigger powerFSM 11
08:21:40 [ApiServer] Client wants config, nonce=42
08:21:40 [ApiServer] Reset nodeinfo read pointer
08:21:40 [ApiServer] getFromRadio, state=2
08:21:40 [ApiServer] encoding toPhone packet to phone variant=3, 45 bytes
08:21:40 [ApiServer] getFromRadio, state=3
08:21:40 [ApiServer] encoding toPhone packet to phone variant=6, 41 bytes
08:21:40 [ApiServer] getFromRadio, state=4
08:21:40 [ApiServer] Sending nodeinfo: num=0x28979058, lastseen=0, id=!246f28979058, name=Unknown 9058
08:21:40 [ApiServer] encoding toPhone packet to phone variant=4, 54 bytes
08:21:40 [ApiServer] getFromRadio, state=4
08:21:40 [ApiServer] Sending nodeinfo: num=0xabf84098, lastseen=1609056949, id=!2462abf84098, name=Node 98
08:21:40 [ApiServer] encoding toPhone packet to phone variant=4, 75 bytes
08:21:40 [ApiServer] getFromRadio, state=4
08:21:40 [ApiServer] Done sending nodeinfos
08:21:40 [ApiServer] getFromRadio, state=5
08:21:40 [ApiServer] encoding toPhone packet to phone variant=8, 2 bytes
[D][WiFiClient.cpp:511] connected(): Disconnected: RES: 0, ERR: 128
08:21:40 [ApiServer] Client dropped connection, closing TCP server
08:21:40 [ApiServer] Telling client we have new packets 0
CORRUPT HEAP: Bad head at 0x3ffea40c. Expected 0xabba1234 got 0x3ffeaaf4
abort() was called at PC 0x400859df on core 1

ELF file SHA256: 0000000000000000

Backtrace: 0x4008ff5f:0x3ffd1470 0x400902e5:0x3ffd1490 0x400859df:0x3ffd14b0 0x40085af9:0x3ffd14f0 0x4011b50f:0x3ffd1510 0x40115af5:0x3ffd17d0 0x4010c7a9:0x3ffd1820 0x400943af:0x3ffd1850 0x40086151:0x3ffd1870 0x40085ba5:0x3ffd1890 0x4000bec7:0x3ffd18b0 0x401a79b9:0x3ffd18d0 0x400d4ae1:0x3ffd18f0 0x400dbf81:0x3ffd1910 0x400d573d:0x3ffd1930 0x400d5776:0x3ffd1980 0x400d4dde:0x3ffd19a0 0x400ec4c9:0x3ffd19c0 0x400d9fe7:0x3ffd19f0 0x400fb5d9:0x3ffd1a10

Rebooting...
geeksville commented 3 years ago

that stack trace means!

~/development/meshtastic/meshtastic-esp32$ bin/exception_decoder.py -e .pio/build/tbeam/firmware.elf ex
stack:
0x4008ff5f: invoke_abort at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
0x400902e5: abort at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
0x400859df: lock_acquire_generic at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:143
0x40085af9: _lock_acquire_recursive at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:171
0x4011b50f: _vfiprintf_r at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/vfprintf.c:865
0x40115af5: _fopen_r at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/fopen.c:141
0x4010c7a9: atexit at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/atexit.c:66
0x400943af: multi_heap_free at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:305
0x40086151: heap_caps_free at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:354
0x40085ba5: _free_r at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/newlib/syscalls.c:42
0x401a79b9: operator new[](unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_opv.cc:33
0x400d4ae1: StreamAPI::~StreamAPI() at /home/kevinh/development/meshtastic/meshtastic-esp32/src/mesh/StreamAPI.h:30
0x400dbf81: PhoneAPI::onNotify(unsigned int) at /home/kevinh/development/meshtastic/meshtastic-esp32/src/mesh/PhoneAPI.cpp:288
0x400d573d: WiFiServerPort::runOnce() at /home/kevinh/development/meshtastic/meshtastic-esp32/src/esp32/WiFiServerAPI.cpp:64
0x400d5776: non-virtual thunk to WiFiServerPort::runOnce() at ??:?
0x400d4dde: concurrency::OSThread::run() at /home/kevinh/development/meshtastic/meshtastic-esp32/src/concurrency/OSThread.cpp:41
0x400ec4c9: ThreadController::runOrDelay() at /home/kevinh/development/meshtastic/meshtastic-esp32/.pio/libdeps/tbeam/Thread/ThreadController.cpp:153
0x400d9fe7: loop() at /home/kevinh/development/meshtastic/meshtastic-esp32/src/main.cpp:572
0x400fb5d9: app_main at /home/kevinh/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:28
~/development/meshtastic/meshtastic-esp32$