search

sparkfun / SparkFun_DataLogger

Documentation and firmware for the SparkFun DataLogger IoT line of products.
https://docs.sparkfun.com/SparkFun_DataLogger/
18 stars 4 forks source link

Crashes faster than 100ms Logging Interval #14

Closed jkrein79 closed 6 months ago

jkrein79 commented 10 months ago

The firmware 01.01.01 crashes when the logging rate is set faster than 100ms logging interval. The error is:

Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandle d.

Core 1 register dump:
PC : 0x40223bc6 PS : 0x00060b30 A0 : 0x800ec90f A1 : 0x3f fb2250
A2 : 0x00000000 A3 : 0x3ffc9730 A4 : 0x3ffb941c A5 : 0x00 0000ff
A6 : 0x3ffb9454 A7 : 0x3ffc8c04 A8 : 0x80223bcc A9 : 0x3f fb2230
A10 : 0x3ffc54a8 A11 : 0x3ffc53e8 A12 : 0x3ffc51e8 A13 : 0x00 000000
A14 : 0x3ffb8a6c A15 : 0x3ffd58b4 SAR : 0x0000000c EXCCAUSE: 0x00 00001c
EXCVADDR: 0x0000011b LBEG : 0x4008563d LEND : 0x40085645 LCOUNT : 0x00 000027

Backtrace: 0x40223bc3:0x3ffb2250 0x400ec90c:0x3ffb2270 0x401602bc:0x3ffb2290

gigapod commented 10 months ago

Hi @jkrein79 -

A couple of questions to help us reproduce this issue (not seeing this crash with an initial test):

Thanks, -Kirk

jkrein79 commented 10 months ago
  1. DataLogger IoT 9DOF
  2. Zero devices attached. Logging IMU and Mag and Fuel Gauge
  3. No IOT service.
  4. Wifi is not connected
  5. 32 gig SD card is installed

This is just usb c connected and serial open. I've tried turning off serial logging and it still happens every few seconds (sd card write size I'm guessing?) It also occurs without an SD card present.

Thank you for your prompt response!

gigapod commented 10 months ago

Thanks for the details.

Don't see the crash - but we'll need to dig deeper than my simple check.

This is just a guess - If you disable the LED (under application settings) does that stop the crash? Every time a log event happens, it triggers the LED to flash in a different task - Wondering if this is overwhelming that system.

Also, does after you start the board, does the crash happen immediately, or after it runs a bit?

jkrein79 commented 10 months ago

Disabling the LED appears to fix the issue. I think the crashing depended on the timer rate, but it would generally print out 30-40 lines and then crash.

With the LED off, it logged up to 10 ms delay!

Sounds like it would be a simple fix to add a semaphore around that or opensource the code? :)

Thank you!

gigapod commented 10 months ago

Fantastic - glad that fixed the issue and it's cranking out data at 10ms intervals!

The LED logic is actually involved - it has a stake stack - when a new state/blink is requested, it's added to the stack. When a state is finished, it pops and reverts to it's previous state.

The blink delta right now is 100ms, so the high logging rate overflowed the stack ... which is checked, but maybe the error condition response, the rtos timer had issues or some other race condition caused the problem

Well get it fixed - planning a small release mid December.

Thanks for helping us find this!

-Kirk

gigapod commented 9 months ago

Update - this was fixed and will be part of our next release that was pushed to late January.

Logic to prevent a race condition greatly improved. Tested down to 2ms log event loops (as fast as we could get it).

falkorichter commented 9 months ago

2ms - impressive!

gigapod commented 6 months ago

Fixed: The fix for this is included in v1.2

1.2 Pre Release