sparkfun / SparkFun_BNO080_Arduino_Library

An Arduino Library for the BNO080 IMU combination triple axis accelerometer/gyro/magnetometer packaged with an ARM Cortex M0+ running powerful algorithms.
Other
81 stars 62 forks source link

discussing about the I2C hanging issues #72

Open jerabaul29 opened 3 years ago

jerabaul29 commented 3 years ago

This is moslty just for discussing / brainstorming.

There is this really annoying issue with the BNO080 hanging when put on I2C to the Artemis.

Any feedback you received about connecting the BNO080 through UART or SPI instead? Does that work without hanging? Of course I would prefer I2C, but if needed, I can use something else.

It is very late, will try tomorrow, but would be great to know if there are some known issues there too / if investigating this would waste my time. In this case, I would stick to I2C and the hard reset solution ^^ .

jerabaul29 commented 3 years ago

I closed issues #68 and #69 in favor of this discussion.

I spent quite a bit of time investigating these issues, so I thought I would give a small update here. That may be useful for other users who hit the same issues.

Because of some issues with querying several data feeds with the current SF code (see #70 ), I use the Adafruit code. There is a bit of a hack to get it to work on the Artemis board, see the recipe https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/main/recipes/recipe_IMU_BNO_Adafruit/recipe_IMU_BNO_Adafruit.ino comment at the start to explain how to get it to work.

I use this script to follow the advices of @PaulZC and restart the board if connection fails (of course the issue is that I then loose all the RAM information; a solution would be to dump user data to Flash through the "EEPROM" library):

#include "Arduino.h"
#include "WDT.h"

#include <Adafruit_BNO08x.h>

Adafruit_BNO08x bno08x{};
sh2_SensorValue_t sensorValue;

unsigned long micros_last;
unsigned long micros_crrt;

void setReports(void) {
  if (!bno08x.enableReport(SH2_ROTATION_VECTOR, 500000UL)) {
    Serial.println("Could not enable rotation vector");
  }
}

APM3_WDT wdt;

void resetArtemis(void)
{
  Wire.end(); //Power down I2C

  SPI.end(); //Power down SPI

  power_adc_disable(); //Power down ADC. It it started by default before setup().

  Serial.end(); //Power down UART

  //Force the peripherals off
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM0);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM1);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM2);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM3);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM4);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM5);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_ADC);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART0);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART1);

  while (1) // That's all folks! Artemis will watchdog reset in 1.25 seconds
    ;
}

void setup(void) {
  wdt.start();

  Serial.begin(115200);
  while (!Serial){
    delay(10);
  }

  Serial.println();
  Serial.println("------ booted ------");
  Serial.println();

  // initialize
  for (int i=0; i<60;i++){
    wdt.restart();
    if (bno08x.begin_I2C(0x4B, &Wire, 0)){
      Serial.println(F("found the Artemis!"));
      break;
    }
    else{
      Serial.print("Failed to find BNO08x chip "); Serial.println(i+1);
      if (i == 19){
        Serial.println(F("reset Artemis"));
        resetArtemis();
      }
      Wire.end();
      delay(50);
      Wire.begin();
      delay(50);

      delay(500);
    }
  }

  Wire.setClock(400000);
  delay(50);

  setReports();

  micros_last = micros();

  Serial.println("----- Reading events -----");
  delay(100);
}

void loop() {
  Serial.println(F("start reading"));

  while(true){
    if (bno08x.wasReset()) {
      Serial.print("sensor was reset ");
      setReports();
      continue;
    }

    if (!bno08x.getSensorEvent(&sensorValue)) {
      continue;
    }
    else{
      wdt.restart();

      switch (sensorValue.sensorId) {
      case SH2_ROTATION_VECTOR:
        micros_crrt = micros();
        Serial.print(F("micros: ")); Serial.print(micros_crrt); Serial.println(F(" | micros elapsed: ")); Serial.println(micros_crrt-micros_last);
        micros_last = micros_crrt;

        Serial.print("Rotation Vector - r: ");
        Serial.print(sensorValue.un.rotationVector.real);
        Serial.print(" i: ");
        Serial.print(sensorValue.un.rotationVector.i);
        Serial.print(" j: ");
        Serial.print(sensorValue.un.rotationVector.j);
        Serial.print(" k: ");
        Serial.println(sensorValue.un.rotationVector.k);

        // TODO: may want to put a small delay here

        break;
      }
    }
  }
}

I noticed a few interesting things. I think there may be 2 different bugs that cause hanging. Indeed, I get 2 "flavors" of hanging.

jerabaul29 commented 3 years ago

@PaulZC I was thinking more about that. Any way that we could restart the I2C port, i.e. power it up and down, from the watchdog, without the need to restart the whole board? That may be a workaround to reset the port without loosing the data in RAM, right? Any idea this could be doable / is there a HAL call for that?

jerabaul29 commented 3 years ago

I am trying to reset the I2C hardware from within the interrupt with something like this:


volatile bool force_wait {true};  // no delay() in interrupt; force reading and writing a volatile variable
volatile int remaining_watchdog_tries {10};

// Interrupt handler for the watchdog
extern "C" void am_watchdog_isr(void)
{
  wdt.clear();

  if (remaining_watchdog_tries > 0){
    wdt.restart();
    remaining_watchdog_tries -= 1;
    Serial.print(F("retry I2C "));

      Wire.end();
      am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM0);

      // hand-made delay, to make sure power has time to go down
      for (int i=0; i<1000000; i++){
        force_wait = !force_wait;
      }

      am_hal_pwrctrl_periph_enable(AM_HAL_PWRCTRL_PERIPH_IOM0);
      Wire.begin();

  }
  else{
    // reset
    while (true){;}
  }
}
jerabaul29 commented 3 years ago

I even tried to disable and enable all the periphs (IOM0 to end), but that does not help.

I do not know / have no way to know as far as I know (some method to get this information?) where the code hangs. I would say that now in around 90% of the cases I manage with the looping to get contact with the BNO, and in around 1 in 10, the code hangs when trying to begin the BNO, the watchdog kicks in, tries 10 times to restart the I2C, but with no success, leading to a full reset.

If there was a way to trigger a reset of the right I2C stuff from the watchdog, rather than by doing a full chip reset, that would practically solve the problem. I do not have the JTAG or similar debugging tool nor the experience to debug down to this level / get this to work. If you have any advices @PaulZC that would be amazing.

PaulZC commented 3 years ago

Hi JR, Like I said in this comment, I have already spent hours looking at this. But, I must admit, I haven't tried JTAG. That might be a way forward - but I really don't have time to look at it right now. If you want to dig into it, start looking in here: https://github.com/sparkfun/Arduino_Apollo3/blob/v1/cores/arduino/ard_sup/iomaster/ap3_iomaster.c#L759 That function is used for both I2C and SPI - not just SPI. If I recall correctly, the behavior is that the hardware gets into a state where am_hal_flash_delay_status_check always times out, preventing any further I2C transactions and causing the bus to hang. If you solve this, I definitely owe you a beer! Paul

jerabaul29 commented 3 years ago

@PaulZC just FYI: it looks like I "empirically" found a recipe that works all the time at starting the BNO with Artemis, at least on the Artemis RedBoard and AGT I have here for testing, and with the couple of BNOs I have here too ("works on my computer" symptom).

This recipe:

https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/main/recipes/recipe_IMU_quaternions/recipe_IMU_quaternions.ino

With in particular this snippet:

https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/5984c4c799a52c4979392e0e71386a56da78fb3a/recipes/recipe_IMU_quaternions/recipe_IMU_quaternions.ino#L36-L67

Has been working without any issues now for me several tens of times. Other similar snippets with different timing or checks fail regularly, but just this one seems to work.

I would be interested to hear if other users are able to use this snippet to get a reliable startup.

jerabaul29 commented 3 years ago

(you really do not want to know how many hours I spent having trial and errors to get to this one...).

PaulZC commented 3 years ago

Hi JR, Sorry, I should have mentioned that earlier. I did notice that calling Wire.setClock(400000); before attempting to start the BNO improved the performance. Why should 400k be more reliable than 100k? I really have no idea. In my mind it just confirms that there is something 'marginal' about the combination of the Artemis hardware with the BNO. Best wishes, Paul

jerabaul29 commented 3 years ago

Yes, there is something really fishy. I have been "test trying" for around 1 hours to get this snippet to crash in the same way other similar snippets do, "luckily without" success. I think there is a broken piece of code somewhere and that somehow having just this combination of delays allows to avoid it.

jerabaul29 commented 3 years ago

For the records, Teensy forum has a loooong discussion around that:

https://forum.pjrc.com/threads/58268-Sparkfun-BNO080-Teensy-3-6-gt-gt-4-0

Read through it now, and not completely sure what the conclusion was ^^ .

adamgarbo commented 3 years ago

Hi folks,

I just received my SparkFun BNO080 boards yesterday and spent a couple of hours trying to troubleshoot why they wouldn't initialize with Artemis-based boards before I found these issues on GitHub.

I'm still catching up, but it sounds like the main takeaway is that the Artemis and BNO080 are, for the moment, incompatible over I2C?

Thanks, @jerabaul29 and @PaulZC for your work in looking into this.

Cheers, Adam

jerabaul29 commented 3 years ago

Hi @adamgarbo ,

Yes, had the same kind of troubles unfortunately :( .

A few notes:

So my advice would be, if it is ok for your application, to use the RVC mode. As far as I found out, the only negative points for my use were that:

So it is in general quite ok to use, but it will make for a good story in my department, when I explain to my boss and colleagues that our instruments believe that they are robot cleaners ^^ .

These problems are very sad, because this is an amazing sensor. I would be very curious @PaulZC to know if using the BNO085 could fix this issue, and if Sparkfun would be ok shifting to these in the future :) .

PS: my current snippet for the I2C. Ugly, ad-hoc, hacky, but seems to do the trick around 90% of the time in my tests:


bool BNO080_Manager::_power_start(void){
  turn_bno_off();
  delay(1000);

  Wire1.begin();
  delay(50);
  Wire1.setClock(400000);
  delay(50);
  wdt.restart();

  delay(2000);
  wdt.restart();
  turn_bno_on();
  delay(5000);
  wdt.restart();

  bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1);
  delay(50);
  bno080_imu.enableDebugging(Serial);
  delay(50);
  wdt.restart();

  bno080_imu.enableRotationVector(imu_output_period);
  // bno080_imu.enableAccelerometer(imu_output_period);
  wdt.restart();

  delay(5000);
  wdt.restart();

  while (!bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1)){
    Serial.println(F("try to begin"));
    delay(100);
  }

  delay(5000);
  bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1);

  wdt.restart();

  Serial.println(F("bno set up, start measuring"));
  wdt.restart();

  return true;
}

bool BNO080_Manager::_first_measurement_start(void){
  //--------------------------------------------------
  Serial.println(F("wait for the first BNO output"));

  bool got_first_reading {false};

  // TODO: protect, this can hang
  for (int i=0; i<15;i++){
    if (bno080_imu.getReadings() != 0){
      got_first_reading = true;
      break;
    }
    Serial.println(F("receiving nothing, try to set output"));
    bno080_imu.enableRotationVector(imu_output_period);
    delay(50);
    bno080_imu.enableAccelerometer(imu_output_period);
    delay(50);
  }
  wdt.restart();

  if (!got_first_reading){
    Serial.println(F("could not get a reading"));
    return false;
  }

  // bno080_imu.enableAccelerometer(imu_output_period);
  bno080_imu.enableRotationVector(imu_output_period);
  wdt.restart();

  Serial.println(F("got first measurement, check has both quat and accel"));

  //--------------------------------------------------
  // check that we can get both a quat and an accel reading
  delay(100);

  Serial.println(F("get accel and quat"));

  while (true){
    if (bno080_imu.getReadings() == SENSOR_REPORTID_ACCELEROMETER){
      Serial.println("accel ok");
      break;
    }
    delay(1);
    // bno080_imu.enableAccelerometer(imu_output_period);
  }

  while (true){
    if (bno080_imu.getReadings() == SENSOR_REPORTID_ROTATION_VECTOR){
      Serial.println(F("quat ok"));
      break;
    }
    delay(1);
    bno080_imu.enableRotationVector(imu_output_period/2);
  }

  //--------------------------------------------------
  Serial.println(F("ignore first BNO measurements"));
  Serial.flush();
  delay(5000);
  wdt.restart();

  for (int i=0; i<50; i++){
    while (true){
      if (_perform_one_reading() != 0){
        break;
      }
      delay(1);
      wdt.restart();
    }
  }
  wdt.restart();

  Serial.println(F("ready"));
  Serial.flush();

  return true;
}

bool BNO080_Manager::start(void){
  bool start_succes {false};

  for (int i=0; i<5; i++){
    if (!_power_start()){
      stop();
      delay(1000);
      continue;
    };
    wdt.restart();

    if (_first_measurement_start()){
      start_succes = true;
      break;
    }
    else{
      stop();
      delay(1000);
    }
  }

  if (start_succes){
    Serial.println(F("success starting bno"));
  }
  else{
    Serial.println(F("no success starting bno"));
    stop();

    Serial.println(F("we reboot"));
    while(true){;}
  }

  return start_succes;
}
adamgarbo commented 3 years ago

Hi folks,

I tested the majority of BNO080 examples using v2.0.5 of the Apollo3 Core and a SparkFun Redboard Artemis/Redboard Artemis Nano. I would have also included theMicroMod Artemis, but v2.0.5 doesn't have this board definition yet.

I am happy to report that I was not able to reproduce any of the I2C hangs or sensor initialization failures with the examples I tested. Perhaps this bug is specific only to v1.2.x?

@jerabaul29, with SparkFun announcing the EOL of v1.x of the Apollo3 Core yesterday, I believe it makes sense for us to focus our efforts on v2.x. User contributions will be a big help in getting it to where it needs to be.

Edit: I may have spoken too soon. I left Example 17 running and it crashed after ~20 minutes multiple times. I was then unable to re-initialize the sensor until I reset/unplugged it. Yikes. I wonder if the Hillcrest Labs/CEVA would have any insight.

Cheers, Adam

jerabaul29 commented 3 years ago

Ok, interesting. Yes, there are reports of issues with the BNO080 all over the internet with a variety of boards. For my part, very curious to know if the BNO085 fixes these issues.

jerabaul29 commented 3 years ago

@PaulZC @nseidle I am planning to build a new batch of between 5 to 10 instruments based on Artemis Global Tracker and some variation of BNO IMU in the course of the next few months. Any hope you can check if upgrading to the BNO085 can offer a fix to this problem, or if there is another way to fix this issue? That would be extremely helpful.

PaulZC commented 3 years ago

Hi JR, Unfortunately we don't have a product based on the BNO085 and, to the best of my knowledge, we don't plan to release one any time soon. (Fundamentally, I don't have a BNO085 I can test.) We have just added support for the Digital Motion Processor on the ICM-20948. You can now read 6-axis or 9-axis Quaternion data from that chip and it works well with the Artemis on I2C. The release notes are here. You can put it into a low power state but, as @adamgarbo observed, the sleep current on our breakout is not as low as it could be. This has to be due to the level-shifting circuitry as we use the same sensor on OpenLog Artemis and can get the sleep current there below 100uA. Best wishes, Paul

jerabaul29 commented 3 years ago

Thanks for the detailed explanations :) .

If I understand right, the BNO085 is virtually exactly the same chip as the BNO080, except from a few fixes. It looks like for example Adafruit just swapped the BNO080 for a BNO085 in production, using the exact same breakout board and libraries. I will definitely buy one of their BNO085 to test a bit and see if it helps. The fact that these 2 chips seems to be very easily swappable, with the BNO085 being a drop-out backwards-compatible replacement, is what would make me very interested in seeing Sparkfun offering it.

jerabaul29 commented 3 years ago

Another thing is that I am interested a bit in the power consumption, but most important for me is accuracy. It seems that, on this point, the BNO08x is the best, right? Or do you know @PaulZC of an accelerometer with even better performance?

That starts to drift a bit from the original discussion, but if you have recommendation about what would be the "best possible accelerometer under at or under 50 to 100USD" in the sense of accuracy, I would be very interested; may be possible to have both an high accuracy accelerometer and an IMU to get acceleration and orientation on the same board for my part, i.e. would not be a worry to have 2 chips with a bit of overlap, if that would be the way to get best accuracy.

PaulZC commented 3 years ago

Hi JR, Indeed, we are going off-topic but let's keep talking... Have you got a feel for what you need in terms of: Gyro Angular Random Walk, Axis to Axis Alignment, Gyro In-Run Bias Stability, Accel In-Run Stability, Factory Calibration etc.? This parameter table from Analog Devices might help: https://www.analog.com/en/parametricsearch/11172#/ Cheers, Paul

jerabaul29 commented 3 years ago

Thanks for the link :) .

I think this will become much broader discussion, would you @PaulZC @adamgarbo be ok to continue discussing here? https://forum.sparkfun.com/viewtopic.php?f=139&t=55182&p=223524#p223524

PaulZC commented 3 years ago

Hi JR (@jerabaul29 ), I hope all is going OK for you. I think I might have cured this, or improved it at least? Can you please pull the release_candidate branch and give it a try? The BNO080 seems to start cleanly on Artemis using I2C whereas it used to fall over regularly (as you know). I'd be very interested to see if you see the same improvement? Thanks! Paul

jerabaul29 commented 3 years ago

Hi Paul,

That sounds amazing, many thanks :) .

I am exhausted after the fieldwork of last month and taking a bit of holidays. @adamgarbo do you think you could have a look at this? :) .

adamgarbo commented 3 years ago

Morning,

I just tested the BNO080 with a SparkFun Redboard Artemis on v2.0.5. It appears that if the sensor and/or system is power cycled, the initialization failures will still manifest themselves.

09:17:52.631 -> BNO080 Read Example
09:17:53.093 -> Rotation vector enabled
09:17:53.093 -> Output in form roll, pitch, yaw
09:17:53.336 -> 3.7,-0.8,-13.9
09:17:53.407 -> 3.7,-0.7,-13.9
09:17:53.440 -> 3.8,-0.7,-13.9
09:17:53.477 -> 3.9,-0.8,-13.9
09:17:53.546 -> 3.9,-0.7,-13.9
09:17:53.581 -> 4.0,-0.7,-13.9
09:17:53.648 -> 4.0,-0.7,-13.9
09:17:53.686 -> 3.9,-0.7,-13.9
09:17:53.757 -> 4.0,-0.6,-13.9
09:17:53.791 -> 3.9,-0.7,-14.0
09:17:53.828 -> 4.1,-0.7,-13.9
09:17:53.903 -> 4.7,-0.6,-14.0
09:17:53.938 -> 5.6,-0.1,-13.5
09:17:53.975 -> 6.0,0.3,-12.9
09:17:54.043 -> 6.1,0.5,-12.6
09:17:54.078 -> 6.8,0.9,-12.8
09:17:54.154 -> 6.5,0.9,-12.9
09:17:54.191 -> 6.9,1.1,-12.7
09:17:54.225 -> 7.0,1.1,-12.7
09:17:54.295 -> 6.9,0.9,-12.8
09:17:56.648 -> BNO080 Read Example
09:17:57.108 -> BNO080 not detected at default I2C address. Check your jumpers and the hookup guide. Freezing...

I also tried testing the Qwiic Power Switch with the Redboard Artemis but I'm not sure if this combination is compatible. I can observe the softReset forcing the QPS to shut down (LED turns off)? I haven't looked at the changes to the library yet, but is this causing a hard reset to the I2C bus?

09:21:00.470 -> BNO080 Read Example
09:21:07.061 -> softReset: first receivePacket
09:21:07.127 -> softReset: second receivePacket
09:21:07.127 -> softReset: complete
09:21:07.127 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7
09:21:18.263 -> softReset: first receivePacket
09:21:18.335 -> softReset: second receivePacket
09:21:18.335 -> softReset: complete
09:21:18.335 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7

Cheers, Adam

PaulZC commented 3 years ago

Hi Adam, Thanks for looking at this. The softReset has always been there - it is part of the .begin functionality - but the debug messages are new. It should have no effect on the Qwiic Power Switch. The I2C address is different - but only two bits different for the default address on the SF Breakout (0x41 vs 0x4B). I've tried hooking up a QPS here and I don't see the LED turn off. Sounds like something very strange is happening? How are your pull-ups configured? Also, I've been meaning to ask: how long do you wait, after turning the QPS on, before you call .begin? Thanks! Paul

adamgarbo commented 3 years ago

Hi Paul,

There is a 1-second delay after enabling the QPS and initializing the BNO080. QPS "IN" pullups enabled, "OUT" disabled. I just ran another series of tests with the SparkFun Redboard Artemis and QPS and had better results. The QPS was able to repeatably power cycle the BNO080. I'll run a longer test to see how it performs.

I did encounter a strange Mbed OS error on more than one occasion (shown below), but I don't know if it is related. It occurs when powering up the Redboard Artemis with the QPS/BNO080 attached or when uploading new code.

With the BNO080 connected directly to the Redboard Artemis, it does appear to be more stable, though pressing the "RESET" button will still produce the usual I2C initialization errors:

11:29:49.092 -> �BNO080 Read Example
11:29:49.542 -> BNO080 not detected at default I2C address. Check your jumpers and the hookup guide. Freezing...

Cheers, Adam

11:20:54.802 -> �BNO080 Read Example
11:20:55.295 -> 
11:20:55.295 -> ++ MbedOS Fault Handler ++
11:20:55.295 -> 
11:20:55.295 -> FaultType: HardFault
11:20:55.295 -> 
11:20:55.295 -> Context:
11:20:55.295 -> R0: 0
11:20:55.295 -> R1: 100070AB
11:20:55.295 -> R2: 1
11:20:55.295 -> R3: A3009150
11:20:55.295 -> R4: 100070AB
11:20:55.295 -> R5: 0
11:20:55.295 -> R6: 4
11:20:55.295 -> R7: 4
11:20:55.295 -> R8: 0
11:20:55.295 -> R9: 0
11:20:55.295 -> R10: 0
11:20:55.295 -> R11: 0
11:20:55.295 -> R12: 25231
11:20:55.295 -> SP   : 10007078
11:20:55.329 -> LR   : 125D9
11:20:55.329 -> PC   : 125E0
11:20:55.329 -> xPSR : 21000000
11:20:55.329 -> PSP  : 10007010
11:20:55.329 -> MSP  : 1005FF70
11:20:55.329 -> CPUID: 410FC241
11:20:55.329 -> HFSR : 40000000
11:20:55.329 -> MMFSR: 0
11:20:55.329 -> BFSR : 82
11:20:55.329 -> UFSR : 0
11:20:55.329 -> DFSR : 0
11:20:55.329 -> AFSR : 0
11:20:55.329 -> BFAR : A3009154
11:20:55.329 -> Mode : Thread
11:20:55.329 -> Priv : Privileged
11:20:55.329 -> Stack: PSP
11:20:55.329 -> 
11:20:55.329 -> -- MbedOS Fault Handler --
11:20:55.329 -> 
11:20:55.329 -> 
11:20:55.329 -> 
11:20:55.329 -> ++ MbedOS Error Info ++
11:20:55.329 -> Error Status: 0x80FF013D Code: 317 Module: 255
11:20:55.329 -> Error Message: Fault exception
11:20:55.329 -> Location: 0x125E0
11:20:55.329 -> Error Value: 0x100060C8
11:20:55.363 -> Current Thread: main Id: 0x100044B8 Entry: 0x25005 StackSize: 0x1000 StackMem: 0x10006128 SP: 0x10007078 
11:20:55.363 -> For more info, visit: https://mbed.com/s/error?error=0x80FF013D&tgt=SFE_ARTEMIS
11:20:55.363 -> -- MbedOS Error Info --
adamgarbo commented 3 years ago

Hi Paul,

I forgot to mention that the LED on the QPS turning off was due to a coding error on my part.

I just started a power-cycle test with the QPS and have already been able to observe some strange behaviour with the BNO080. The test cycles between 10 seconds ON/OFF. Below, you can see that the sensor fails to reinitialize several times, but then magically starts to work again. This seems to be consistent behaviour the longer the test runs.

Very strange!

11:44:07.180 -> -0.8,-7.8,-10.1
11:44:07.252 -> -0.8,-7.8,-10.1
11:44:07.287 -> -0.8,-7.8,-10.1
11:44:07.323 -> -0.8,-7.8,-10.1
11:44:07.393 -> -0.8,-7.8,-10.1
11:44:07.429 -> -0.7,-7.8,-10.1
11:44:07.503 -> -0.8,-7.8,-10.1
11:44:07.537 -> -0.7,-7.8,-10.1
11:44:07.575 -> -0.7,-7.8,-10.1
11:44:07.644 -> -0.7,-7.8,-10.1
11:44:07.682 -> -0.7,-7.8,-10.1
11:44:18.694 -> sendPacket(I2C): endTransmission returned: 2
11:44:18.835 -> softReset: first receivePacket
11:44:18.908 -> softReset: second receivePacket
11:44:18.908 -> softReset: complete
11:44:18.908 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7
11:44:41.087 -> sendPacket(I2C): endTransmission returned: 2
11:44:41.260 -> softReset: first receivePacket
11:44:41.293 -> softReset: second receivePacket
11:44:41.330 -> softReset: complete
11:44:41.330 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7
11:45:03.406 -> softReset: first receivePacket
11:45:03.477 -> softReset: second receivePacket
11:45:03.477 -> softReset: complete
11:45:03.477 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7
11:45:25.602 -> softReset: first receivePacket
11:45:25.638 -> softReset: second receivePacket
11:45:25.638 -> softReset: complete
11:45:25.638 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7
11:45:47.745 -> softReset: first receivePacket
11:45:47.816 -> softReset: second receivePacket
11:45:47.816 -> softReset: complete
11:45:47.816 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7
11:46:12.515 -> softReset: first receivePacket
11:46:12.515 -> softReset: second receivePacket
11:46:12.515 -> softReset: complete
11:46:12.515 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x18A498 SW Build Number: 0x172 SW Version Patch: 0x7
11:46:12.515 -> -0.7,-7.6,-8.7
11:46:12.515 -> -0.8,-7.8,-8.7
11:46:12.515 -> -0.8,-7.8,-8.7
11:46:12.515 -> -0.8,-7.8,-8.7
11:46:12.515 -> -0.8,-7.8,-8.7
11:46:12.515 -> -0.8,-7.8,-8.7
11:46:12.515 -> -0.8,-7.8,-8.7
11:46:12.515 -> -0.8,-7.8,-8.7
PaulZC commented 3 years ago

Well, it's progress. Kind of... ;-) I do see occasions where: if I reset the Artemis by re-starting the Serial Monitor, the SDA line sticks low, permanently. And then (of course) trying to .begin the BNO fails. I suspect the BNO is pulling it low. Maybe because it was interrupted part way through a data transfer ? I'm wondering if the BNO would behave better if we disabled the data stream or put it into modeSleep before turning the power off?

adamgarbo commented 3 years ago

I think the BNO080 is just too darn smart for its own good!

Can we also pause for a moment to appreciate the irony that the BNO080 is based on the Cortex-M0+ but throws a fit when we want it to work with other Cortex-M0+ or M4 processors?

PaulZC commented 3 years ago

"Interesting" thing: if you see the "SW Version Major" debug message then the BNO has begun successfully. You only see that if the PRODUCT_ID_RESPONSE is correct. But, yes, it then doesn't generate any data. Why does it sometimes ignore the setFeatureCommand which enables the sensor report? I think that's what I need to figure out next...

adamgarbo commented 3 years ago

Paul Stoffegren fixed (perhaps?) the same issue with the Teensy 4.0 (Cortex M7) and the BNO080 last year, though I'm not exactly sure what was fixed and if it's applicable to the M4 or M0+-based processors:

https://forum.pjrc.com/threads/58268-Sparkfun-BNO080-Teensy-3-6-gt-gt-4-0?p=226478&viewfull=1#post226478

Would SparkFun be eligible for support directly from CEVA/Hillcrest Labs?

PaulZC commented 3 years ago

Yeah. I'm pretty sure that's it. I've restructured the sleep example and it now seems to start reliably too. I was seeing the same thing as you. The begin was successful but then the setFeatureCommand failed periodically. Please kick the tyres if you can spare the time! Thanks! The good news (for you and JR (@jerabaul29 )) is that the current draw in sleep is now back down to the ~2mA LED background current. The previous version continued to pound the I2C bus while the BNO was asleep, raising the current draw by several mA!

Re: CEVA. Yes, but they're busy guys. We try not to bother them unless there's something really major we need to ask them about. (Like the SPI bus crash feature - if you don't read the data quickly enough). I guess this issue is getting up there though...!

All the best, Paul

adamgarbo commented 3 years ago

Hi Paul,

To confirm, are you using Apollo3 Core v2.0.5?

I've found that when the BNO080 is connected directly to a Redboard Artemis on v2.0.5, there is a significant improvement to stability. The BNO080 seems to be able to recover from multiple hard resets to the board or unplugging/plugging back in the USB-C cable.

However, when the QPS is added back, the BNO080 will crash after a number of power cycles and never recover. The QPS continues to operate (see below).

Also, no improvements with SAMD-based boards, though I'm not sure if the changes would affect this platform.

Cheers, Adam

...
15:18:40.030 -> 0.3,12.8,-111.5
15:18:40.068 -> 0.3,12.8,-111.5
15:18:40.137 -> 0.3,12.8,-111.5
15:18:40.170 -> 0.3,12.8,-111.5
15:18:40.240 -> 0.3,12.8,-111.5
15:18:40.274 -> 0.3,12.8,-111.5
15:18:40.340 -> 0.3,12.8,-111.5
15:18:47.071 -> sendPacket(I2C): endTransmission returned: 2
15:18:47.206 -> softReset: first receivePacket
15:18:47.312 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:47.383 -> softReset: second receivePacket
15:18:47.487 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:47.487 -> softReset: complete
15:18:47.487 -> sendPacket(I2C): endTransmission returned: 2
15:18:47.595 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:47.595 -> BNO080 not detected at default I2C address. Check your jumpers and the hookup guide. Freezing...
15:18:47.595 -> sendPacket(I2C): endTransmission returned: 2
15:18:48.696 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:48.808 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:48.916 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.027 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.136 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.241 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.344 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.450 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.559 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.662 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.771 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.874 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:49.978 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.080 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.182 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.320 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.425 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.529 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.638 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.739 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.843 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:50.946 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.054 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.155 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.262 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.366 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.471 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.577 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.682 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.787 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:51.893 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.000 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.109 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.218 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.327 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.433 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.542 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.645 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.753 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.856 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:52.957 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.093 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.193 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.294 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.399 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.502 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.603 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.709 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.813 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:53.922 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.037 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.141 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.245 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.351 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.460 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.568 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.673 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.776 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.909 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:54.985 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.097 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.201 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.304 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.443 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.519 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.629 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.763 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.866 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:55.975 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.082 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.183 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.288 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.395 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.509 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.622 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.723 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.824 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:56.933 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.042 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.151 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.257 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.365 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.475 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.549 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.656 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.790 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:57.895 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:58.005 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:58.108 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:58.221 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:58.325 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:58.436 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:58.508 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:18:58.614 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:19:09.606 -> sendPacket(I2C): endTransmission returned: 2
15:19:09.782 -> softReset: first receivePacket
15:19:09.888 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:19:09.923 -> softReset: second receivePacket
15:19:10.027 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:19:10.027 -> softReset: complete
15:19:10.027 -> sendPacket(I2C): endTransmission returned: 2
15:19:10.135 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:19:10.168 -> BNO080 not detected at default I2C address. Check your jumpers and the hookup guide. Freezing...
15:19:10.168 -> sendPacket(I2C): endTransmission returned: 2
15:19:11.265 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:19:11.365 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
15:19:11.472 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
...
PaulZC commented 3 years ago

I’m still using 1.2.1. Haven’t been brave enough to upgrade yet. But that time is coming soon! All the best, Paul

adamgarbo commented 3 years ago

Interesting! I found that when I downgraded to v1.2.1 and conducted the same tests, the sensor was significantly less stable than on v2.0.5.

What instrument are you using to monitor the I2C data? It might be something worth investing in.

Cheers, Adam

PaulZC commented 3 years ago

https://www.sparkfun.com/products/15033

jerabaul29 commented 3 years ago

Just picking things up: what is the current status on this? :) . @adamgarbo are you using the BNO in 'production' and does it work fine, or did you give up and go for another IMU?

adamgarbo commented 3 years ago

Hi @jerabaul29,

Unfortunately, I haven't had time to troubleshoot the BNO080 further. With how unreliable it's been, I think it'll be too risky to include in the instrumentation I'll be deploying this year, especially considering it can crash the entire I2C bus.

I'm currently exploring alternative IMU sensor options. Jim Remington has written a library for the LSM9DS1 that enables tilt-compensated compass measurements and sensor fusion (https://github.com/jremington/LSM9DS1-AHRS). I've been waiting for SparkFun to get the sensor back in stock (https://www.sparkfun.com/products/13944).

Please let me know if you have any suggestions.

jerabaul29 commented 3 years ago

Ok, thanks for the update!

I just ordered / will be looking at replacing it with the ICM-20948 .

adamgarbo commented 3 years ago

Fair warning that the ICM-20948 is a black box and has its own issues. Paul has done lots of great work with the library, but I haven't been able to get reliable measurements from it yet. It also has a high quiescent draw of ~4 mA due to the switching circuitry.

https://github.com/sparkfun/SparkFun_ICM-20948_ArduinoLibrary/issues/40

jerabaul29 commented 3 years ago

Ok, thanks for the warning. This is quite annoying - there are so many IMUs that 'should be a perfect fit' but looks like none of them is truly working flawlessly :( .

PaulZC commented 3 years ago

Hi JR, The release_candidate branch contains some changes which helped the BNO080 start correctly on Artemis / Apollo3 using v1.2.1 of the core. The key thing was inserting an extra delay after the soft reset. If you try to talk to the BNO080 too soon, it returns an invalid or corrupt packet where the length is huge. That was confusing the library and causing it to try to read far too many bytes from the BNO. The extra delay cures this. But the changes I've made seem to cause extra problems on the PORTENTA. I don't have PORTENTA hardware so I can't investigate further... All the best, Paul

jerabaul29 commented 3 years ago

That sounds great @PaulZC :) . So do you mean this new fix on dev branch is working 100% of the time? @adamgarbo you were naming some issues even with the newer fixes, does that apply to this dev branch code?

jerabaul29 commented 3 years ago

To continue this discussion @PaulZC , from my 'small egoistic' perspective I am only interested in using the BNO with the Artemis platform, but I really need high reliability, so Portenta is not a big deal for me :) Would even say that if there were an 'artemis tuned' branch that would work perfectly with specifically the Artemis, that would be excellent as far as I am concerned :)

PaulZC commented 3 years ago

The Portenta issue was - I think - a silly debug print mistake which I've just corrected: https://github.com/sparkfun/SparkFun_BNO080_Arduino_Library/issues/80#issuecomment-808867019 Please give release_candidate a go when you have time. Thanks!

jerabaul29 commented 3 years ago

Sounds good, I want do some detailed testing of the updated branch, hope to have a bit of time for that next week :) .

adamgarbo commented 3 years ago

Hi @jerabaul29,

I quickly tested the release-candidate branch with the Artemis on v1.2.1 and it appears to still have issues with initialization and power cycling (reset button).

I purchased a logic analyzer from SparkFun but don't have the time to work on this at the moment. Please let me know what you find!

09:02:23.510 -> BNO080 Read Example
09:02:24.004 -> BNO080 not detected at default I2C address. Check your jumpers and the hookup guide. Freezing...
09:02:29.106 -> 
09:02:29.106 -> BNO080 Read Example
09:02:29.524 -> BNO080 not detected at default I2C address. Check your jumpers and the hookup guide. Freezing...
09:02:32.671 -> 

Cheers, Adam

jerabaul29 commented 3 years ago

Ok, thanks, sad to hear that, will do some tests later this week (I hope, busy week).

adamgarbo commented 3 years ago

Hi @jerabaul29,

I have a test running with the BNO080 and MicroMod Artemis Processor + Data Logging Carrier Board. I am power cycling the sensor 5 seconds ON and 5 seconds OFF, reinitializing each time.

So far, it does appear to be more stable, but the important thing was to remove the while (1); in the example if the initialization fails. I've read elsewhere that the failure to initialize should be ignored and the sensor can be read from regardless.

When you enable debugging, you can observe there is a timeout, but the sensor does eventually initialize properly:

10:33:21.547 -> sendPacket(I2C): endTransmission returned: 4
10:33:21.690 -> softReset: first receivePacket
10:33:21.796 -> waitForI2C: I2C timeout when expecting 4 bytes. 0 were available
10:33:21.866 -> softReset: second receivePacket
10:33:21.866 -> softReset: complete
10:33:21.866 -> SW Version Major: 0x3 SW Version Minor: 0x2 SW Part Number: 0x98A498 SW Build Number: 0x172 SW Version Patch: 0x7

When I upload new code the sensor does still fail to initialize, frequently, but it appears you can just ignore this. @PaulZC have you found the same?

Cheers, Adam

PaulZC commented 3 years ago

Many thanks Adam, I haven't tried cycling the power anywhere near as thoroughly as you... Cheers, Paul

jerabaul29 commented 3 years ago

@adamgarbo : when doing some testing a couple of months ago I had had similar results that ignoring initialization was helping. However, I was then hitting another problem: I needed to enable BOTH the quaternion and the IMU output, and when some initialization was failing, I was sometimes not able to ask the chip for both of them, and I was getting either the quaternion of the IMU but not both.

Not sure if this works now or not, but that may be a tips for testing: to be really exhaustive, may be a good idea during test to check that you can both: