search

sparkfun / OpenLog_Artemis

The OpenLog Artemis is an open source datalogger the comes preprogrammed to automatically log IMU, GPS, serial data, and various pressure, humidity, and distance sensors. All without writing a single line of code!
https://www.sparkfun.com/products/15846
Other
86 stars 45 forks source link

Hard faults on OLA and 'poisoned' ISM330DHCX #163

Closed decamun closed 1 year ago

decamun commented 1 year ago

Subject of the issue

It seems that the ISM330DHCX sensors I'm using have been 'poisoned.' They were working with one OLA, but after a mysterious hard fault they no longer work, even with a second brand new OLA.

Your workbench

Steps to reproduce

This hard fault first popped up on a different Openlog Artemis I was using with slightly modified firmware, and a quick mux with two ISM330DHCX sensors connected (to different ports on the mux).

I did disconnect the pull up resistor and swap the address connection on one of the ISM330DHCX sensors at some point during this process. However that was before the first occurrence of the hard fault, and even the unmodified ISM330DHCX sensor no longer works.

Finally, it's worth noting that the two ICM IMUs and the two Lidars had worked in the past with the old OLA with the same firmware. The hard fault only appeared when both groups of sensors were connected.

Expected behavior

I would expect the sensors to connect and log as normal, however it appears that the ISM330DHCX sensors I'm using have been 'poisoned'.

Actual behavior

Log using first OLA:

Artemis OpenLog v2.5 Logging to: dataLog00090.TXT Finding the next available log file. This could take a long time if the SD card contains many existing log files. Logging to: serialLog00000.TXT SD card online Data logging online Serial logging online IMU offline - or not present Logging analog pins at max data rate Identifying Qwiic Muxes... Identifying Qwiic Devices... Multiplexers found. Scanning sub nets... Unknown device at address (0x7E)(Mux:0x70 Port:0) Autodetect complete IMU-ISM330DHCX online at address 0x6B.0x70.0 Multiplexer online at address 0x70 Content-Type: text/csv

rtcTime,VIN,aX,aY,aZ,gX,gY,gZ,dataRdy,output_Hz,

Menu: Main Menu 1) Configure Terminal Output 2) Configure Time Stamp 4) Configure Serial Logging 5) Configure Analog Logging 6) Detect / Configure Attached Devices 7) Configure Power Options h) Print Sensor Helper Text (and return to logging) s) SD Card File Transfer r) Reset all settings to default q) Quit: Close log files and power down x) Return to logging ---- Sent utf8 encoded message: "x" ---- 00:00:40.80,5.31,-5.73,-7.20,795.81,385.0,-647.50,-411.25,1,0.0,

++ MbedOS Fault Handler ++

FaultType: HardFault

Context: R0: 1000C168 R1: 8 R2: 0 R3: 6F546F67 R4: 1000C168 R5: 61A80 R6: 0 R7: 100088A0 R8: 1 R9: 0 R10: 0 R11: 0 R12: 100088D8 SP : 10008890 LR : 280BD PC : 3C4E4 xPSR : 1000000 PSP : 10008828 MSP : 1005FF70 CPUID: 410FC241 HFSR : 40000000 MMFSR: 0 BFSR : 82 UFSR : 0 DFSR : 0 AFSR : 0 BFAR : 6F546F77 Mode : Thread Priv : Privileged Stack: PSP

-- MbedOS Fault Handler --

++ MbedOS Error Info ++ Error Status: 0x80FF013D Code: 317 Module: 255 Error Message: Fault exception Location: 0x3C4E4 Error Value: 0x10007A68 Current Thread: main Id: 0x10005E44 Entry: 0x4E381 StackSize: 0x1000 StackMem: 0x10007AC8 SP: 0x10008890 For more info, visit: https://mbed.com/s/error?error=0x80FF013D&tgt=SFE_ARTEMIS_ATP -- MbedOS Error Info --

Log using second (brand new) OLA:

SD init failed (second attempt). Is card present? Formatted? Please ensure the SD card is formatted correctly using https://www.sdcard.org/downloads/formatter/ Config file read failed: SD offline Artemis OpenLog v2.4 SD card offline Datalogging offline Serial logging offline IMU online Identifying Qwiic Muxes... Identifying Qwiic Devices... Known I2C address but device failed identification at address 0x6B Unknown device at address (0x7E) Autodetect complete Device config file read failed: SD offline Content-Type: text/csv

rtcDate,rtcTime,aX,aY,aZ,gX,gY,gZ,mX,mY,mZ,imu_degC,output_Hz, 01/01/2000,00:24:32.37,-49.32,-48.34,-985.35,0.86,-0.85,0.53,51.75,6.15,-24.15,27.50,100.0, 01/01/2000,00:24:32.46,-68.85,-37.60,-973.14,0.65,-6.76,3.12,51.15,6.30,-24.45,27.55,18.182, 01/01/2000,00:24:32.56,-63.96,-50.29,-983.40,-1.34,-0.52,1.76,51.15,7.50,-25.80,27.69,14.286, 01/01/2000,00:24:32.66,-67.38,-48.34,-972.66,-1.75,0.76,-1.73,51.0,6.15,-23.85,27.98,12.903, 01/01/2000,00:24:32.76,-61.04,-42.48,-988.77,2.42,-1.37,-0.26,50.55,6.15,-24.15,27.84,12.195, 01/01/2000,00:24:32.86,-60.55,-57.13,-982.91,-1.33,-1.51,-1.27,51.15,5.85,-25.20,27.89,11.765, 01/01/2000,00:24:32.96,-59.57,-51.27,-982.91,-1.26,-4.16,0.08,51.30,6.45,-23.85,27.79,11.475, 01/01/2000,00:24:33.06,-59.57,-46.39,-981.45,0.10,0.05,-0.91,49.80,5.70,-24.60,27.69,11.268, 01/01/2000,00:24:33.16,-57.13,-37.60,-977.05,-0.42,-8.01,1.62,51.30,6.15,-24.60,28.03,11.111, 01/01/2000,00:24:33.26,-62.01,-47.36,-983.89,-2.08,3.40,-2.95,50.70,7.80,-26.25,27.65,10.989, 01/01/2000,00:24:33.36,-54.69,-50.29,-987.30,-0.04,-1.57,3.02,50.70,7.80,-25.20,28.03,10.891, 01/01/2000,00:24:33.46,-72.75,-56.64,-980.96,1.31,-5.37,0.44,50.40,6.75,-25.80,27.89,10.811, 01/01/2000,00:24:33.56,-69.34,-51.76,-971.68,-2.59,-1.50,-1.02,51.90,5.85,-25.50,27.98,10.833, 01/01/2000,00:24:33.66,-65.43,-46.88,-1001.46,0.07,-7.59,0.25,49.50,6.90,-25.80,27.55,10.769, 01/01/2000,00:24:33.76,-46.39,-40.04,-978.03,-1.22,-1.53,0.14,51.30,6.90,-24.60,27.79,10.714, 01/01/2000,00:24:33.86,-71.78,-55.18,-993.65,0.75,-5.41,1.35,51.15,6.75,-24.75,27.89,10.667, 01/01/2000,00:24:33.96,-62.01,-58.11,-974.12,2.57,-2.66,0.17,51.15,6.75,-25.20,27.84,10.625, 01/01/2000,00:24:34.06,-70.80,-45.41,-988.28,-1.89,1.34,-0.10,51.15,5.85,-24.45,27.69,10.588, 01/01/2000,00:24:34.16,-60.06,-53.71,-992.19,1.05,-5.63,-0.43,51.0,8.10,-25.35,27.89,10.556, 01/01/2000,00:24:34.26,-64.45,-47.36,-980.96,-2.07,-2.58,1.73,51.15,5.85,-25.20,27.69,10.526, ---- Closed the serial port COM3 ----

decamun commented 1 year ago

Update: strangely the first OLA, running modified v2.5 firmware, has seemingly recovered from the hard fault it suffered. It's now capable of identifying and reading from both ISM330DHCX sensors without the presence of a mux.

However, there appears to be a phantom device online at (0x7E).

In order to recover from the hard fault, I selected option (r) in the serial menu.

Log below:

SD init failed (second attempt). Is card present? Formatted? Please ensure the SD card is formatted correctly using https://www.sdcard.org/downloads/formatter/ Config file read failed: SD offline Artemis OpenLog v2.5 SD card offline Datalogging offline Serial logging offline IMU online Identifying Qwiic Muxes... Identifying Qwiic Devices... Unknown device at address (0x7E) Autodetect complete Device config file read failed: SD offline IMU-ISM330DHCX online at address 0x6A IMU-ISM330DHCX online at address 0x6B Content-Type: text/csv

rtcDate,rtcTime,aX,aY,aZ,gX,gY,gZ,mX,mY,mZ,imu_degC,aX,aY,aZ,gX,gY,gZ,dataRdy,aX,aY,aZ,gX,gY,gZ,dataRdy,output_Hz, 01/01/2000,00:00:45.71,-221.19,-33.20,-2000.0,1.83,-0.37,0.56,-9.45,27.45,-3.90,25.78,4.39,-25.13,899.38,166.25,245.0,-78.75,1,7.81,-130.05,856.68,17.50,-542.50,-498.75,1,100.0,

decamun commented 1 year ago

On the other hand, the brand new OLA is still not capable of initializing the unmodified ISM330DHCX

PaulZC commented 1 year ago

Hi Devin (@decamun ),

I think you've somehow managed to corrupt the Artemis' flash memory? The message Logging analog pins at max data rate is worrying. That's not a default option - and it means the code is only logging the analog pin voltages as fast as possible.

Please check the compiler messages. If your code is using more that 50% of the available program storage space, then you are probably hitting the bootloader issue we discussed in #154.

You new OLA is running firmware 2.4 - which doesn't support the ISM330. Support for that was added in v2.5. So that explains why that combination isn't working.

I've also seen the device at 0x7E issue. I don't know why, but the ISM330 does seem to have something inside it which responds to that address.

Best wishes, Paul

PaulZC commented 1 year ago

One more thing that might help:

The ICM20948 uses I2C addresses 0x68 and 0x69. That's only 1 bit different from the ISM330 at 0x6A and 0x6B. If the voltages on your I2C bus aren't quite right - maybe due to the pull-ups - then it might be possible for both devices to respond simultaneously, causing all kinds of badness...

PaulZC commented 1 year ago

Ah. Another thing... If you added extra settings in your custom code, but haven't increased the software version number, then the original settings from the original 2.4 or 2.5 will still be in EEPROM. The data won't then align with your custom code and its settings. That would also cause all kinds of badness and might explain the Logging analog pins message. The r option resets all the settings - including your new additional settings - and so everything becomes aligned again.

PaulZC commented 1 year ago

Hi Devin (@decamun ),

I'm closing this as I believe we understand why it is happening: the settings not being updated in EEPROM (a 'reset' will fix that); and your new OLA being 2.4 and not supporting the ISM330.

Please re-open if you need more help with this.

Best wishes, Paul