Sending API requests while the wheel is streaming sometimes crashes the uC

felixchenier commented 7 months ago

Even with somewhat low sample rates, if the wheel is streaming and we make get_system_state requests maybe 2-3 times per second, it always happen that the uC crashes and reboots. I had to comment line 84 of python/nextwheel/monitor.py to avoid those crashes when monitoring the wheel state.

doumdi commented 6 months ago

@felixchenier Is this possible to connect the MCU to the computer and get the serial data sent when this happens ?

I suspect we are running out of memory.

doumdi commented 6 months ago

@felixchenier I have created a new branch with the latest version of the SDK. To be tested: https://github.com/felixchenier/NextWheel/tree/update-sdk

felixchenier commented 6 months ago

@doumdi I uploaded the new firmware with the new SDK, but we still have:

Intermittent jumps in data transfer
Crash after too many API requests.

I run a streaming at ADC=240Hz, IMU=240Hz, with about 2-3 API requests per second to know the current status. It crashed after about 3 minutes with this crash log (compiled in Verbose mode (Debug level 5). It confirms a heap issue.

16:07:45.287 > [135774][V][WebServer.cpp:296] handleClient(): New client: client.localIP()=192.168.1.167
16:07:45.293 > [135776][V][Parsing.cpp:122] _parseRequest(): method: GET url: /system_state search: 
16:07:45.299 > [135779][V][Parsing.cpp:226] _parseRequest(): headerName: Host
16:07:45.304 > [135785][V][Parsing.cpp:227] _parseRequest(): headerValue: 192.168.1.167
16:07:45.309 > [135791][V][Parsing.cpp:226] _parseRequest(): headerName: User-Agent
16:07:45.320 > [135797][V][Parsing.cpp:227] _parseRequest(): headerValue: python-requests/2.31.0
16:07:45.326 > [135805][V][Parsing.cpp:226] _parseRequest(): headerName: Accept-Encoding
16:07:45.331 > [135811][V][Parsing.cpp:227] _parseRequest(): headerValue: gzip, deflate, br
16:07:45.337 > [135818][V][Parsing.cpp:226] _parseRequest(): headerName: Accept
16:07:45.342 > [135824][V][Parsing.cpp:227] _parseRequest(): headerValue: */*
16:07:45.348 > [135830][V][Parsing.cpp:226] _parseRequest(): headerName: Connection
16:07:45.354 > [135835][V][Parsing.cpp:227] _parseRequest(): headerValue: keep-alive
16:07:45.359 > [135841][V][Parsing.cpp:254] _parseArguments(): args: 
16:07:45.365 > [135846][V][Parsing.cpp:237] _parseRequest(): Request: /system_state
16:07:45.370 > [135852][V][Parsing.cpp:238] _parseRequest():  Arguments: 
16:07:45.376 > get: /system_state
16:07:45.382 > {
16:07:45.382 >  "recording":    0,
16:07:45.382 >  "streaming":    1,
16:07:45.382 >  "filename":     ""
16:07:45.382 > }
16:07:45.594 > [136079][V][WebServer.cpp:296] handleClient(): New client: client.localIP()=192.168.1.167
16:07:45.600 > [136082][V][Parsing.cpp:122] _parseRequest(): method: GET url: /system_state search: 
16:07:45.605 > [136086][V][Parsing.cpp:226] _parseRequest(): headerName: Host
16:07:45.610 > [136091][V][Parsing.cpp:227] _parseRequest(): headerValue: 192.168.1.167
16:07:45.616 > [136098][V][Parsing.cpp:226] _parseRequest(): headerName: User-Agent
16:07:45.627 > [136104][V][Parsing.cpp:227] _parseRequest(): headerValue: python-requests/2.31.0
16:07:45.632 > [136111][V][Parsing.cpp:226] _parseRequest(): headerName: Accept-Encoding
16:07:45.638 > [136117][V][Parsing.cpp:227] _parseRequest(): headerValue: gzip, deflate, br
16:07:45.644 > [136124][V][Parsing.cpp:226] _parseRequest(): headerName: Accept
16:07:45.649 > [136132][V][Parsing.cpp:227] _parseRequest(): headerValue: */*
16:07:45.654 > [136136][V][Parsing.cpp:226] _parseRequest(): headerName: Connection
16:07:45.661 > [136142][V][Parsing.cpp:227] _parseRequest(): headerValue: keep-alive
16:07:45.666 > [136148][V][Parsing.cpp:254] _parseArguments(): args: 
16:07:45.671 > [136153][V][Parsing.cpp:237] _parseRequest(): Request: /system_state
16:07:45.677 > [136159][V][Parsing.cpp:238] _parseRequest():  Arguments: 
16:07:45.682 > get: /system_state
16:07:45.688 > {
16:07:45.688 >  "recording":    0,
16:07:45.688 >  "streaming":    1,
16:07:45.688 >  "filename":     ""
16:07:45.688 > }
16:07:45.893 > [136380][V][WebServer.cpp:296] handleClient(): New client: client.localIP()=192.168.1.167
16:07:45.899 > [136381][V][Parsing.cpp:122] _parseRequest(): method: GET url: /system_state search: 
16:07:45.904 > [136387][V][Parsing.cpp:226] _parseRequest(): headerName: Host
16:07:45.909 > [136390][V][Parsing.cpp:227] _parseRequest(): headerValue: 192.168.1.167
16:07:45.915 > [136397][V][Parsing.cpp:226] _parseRequest(): headerName: User-Agent
16:07:45.926 > [136403][V][Parsing.cpp:227] _parseRequest(): headerValue: python-requests/2.31.0
16:07:45.931 > [136410][V][Parsing.cpp:226] _parseRequest(): headerName: Accept-Encoding
16:07:45.937 > [136416][V][Parsing.cpp:227] _parseRequest(): headerValue: gzip, deflate, br
16:07:45.943 > [136423][V][Parsing.cpp:226] _parseRequest(): headerName: Accept
16:07:45.948 > [136429][V][Parsing.cpp:227] _parseRequest(): headerValue: */*
16:07:45.954 > [136435][V][Parsing.cpp:226] _parseRequest(): headerName: Connection
16:07:45.959 > [136441][V][Parsing.cpp:227] _parseRequest(): headerValue: keep-alive
16:07:45.965 > [136447][V][Parsing.cpp:254] _parseArguments(): args: 
16:07:45.970 > [136452][V][Parsing.cpp:237] _parseRequest(): Request: /system_state
16:07:45.976 > [136458][V][Parsing.cpp:238] _parseRequest():  Arguments: 
16:07:45.982 > get: /system_state
16:07:45.987 > {
16:07:45.987 >  "recording":    0,
16:07:45.987 >  "streaming":    1,
16:07:45.987 >  "filename":     ""
16:07:45.987 > }
16:07:46.439 > WARNING : Free heap size: 45056
16:07:46.486 > 
16:07:46.487 > abort() was called at PC 0x4015938b on core 1
16:07:46.487 > 
16:07:46.487 > 
16:07:46.487 > Backtrace: 0x40083c35:0x3ffaee00 0x4008dbd9:0x3ffaee20 0x40093ba1:0x3ffaee40 0x4015938b:0x3ffaeec0 0x401593d2:0x3ffaeee0 0x401595c1:0x3ffaef00 0x40159678:0x3ffaef20 0x400e1916:0x3ffaef40 0x400d4180:0x3ffaef60 0x400e1206:0x3ffaefe0
16:07:46.800 > 
16:07:46.800 >   #0  0x40083c35:0x3ffaee00 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:408
16:07:46.800 >   #1  0x4008dbd9:0x3ffaee20 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:137
16:07:46.800 >   #2  0x40093ba1:0x3ffaee40 in abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/abort.c:46
16:07:46.800 >   #3  0x4015938b:0x3ffaeec0 in __cxxabiv1::__terminate(void (*)()) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:47
16:07:46.800 >   #4  0x401593d2:0x3ffaeee0 in std::terminate() at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:57
16:07:46.800 >   #5  0x401595c1:0x3ffaef00 in __cxa_allocate_exception at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:300
16:07:46.800 >   #6  0x40159678:0x3ffaef20 in operator new(unsigned int) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/new_op.cc:54
16:07:46.800 >   #7  0x400e1916:0x3ffaef40 in IMUDataFrame::clone() const at lib/NextWheel/src/data/IMUDataFrame.cpp:23
16:07:46.800 >   #8  0x400d4180:0x3ffaef60 in SensorTask::sendData(DataFrame const&) at lib/NextWheel/src/tasks/SensorTask.h:50
16:07:46.800 >       (inlined by) IMUSensorTask::run(void*) at src/tasks/IMUSensorTask.cpp:82
16:07:46.800 >   #9  0x400e1206:0x3ffaefe0 in Task::runTask(void*) at lib/NextWheel/src/Task.cpp:64
16:07:46.800 > 
16:07:46.800 > 
16:07:46.800 > 
16:07:46.800 > 
16:07:46.800 > ELF file SHA256: eeb499b31378c10a
16:07:46.800 > 
16:07:46.800 > Rebooting...

felixchenier / NextWheel

Sending API requests while the wheel is streaming sometimes crashes the uC #27