Sometimes MPU9250 doesn't initialize

[julianoes]

Sometimes it appears that the MPU9250 doesn't initialize correctly:

   2643172: HAP:61:  arg0 = 'df_mpu9250_wrapper' (main.cpp:85)
   2643172: HAP:61:  arg1 = 'start' (main.cpp:85)
   2643172: HAP:61:1367420 Added driver f0914e20 /dev/imu0 (DevMgr.cpp:145)
   2643182: HAP:60:px4_getpid() called from unknown thread context! (px4_qurt_tasks.cpp:343)
   2643182: HAP:60:px4_getpid() called from unknown thread context! (px4_qurt_tasks.cpp:343)
   2643254: HAP:61:1469621 Reset MPU9250 (MPU9250.cpp:89)
   2643332: HAP:48:[posix-uORB::Manager::process_add_subscription(339)] entering Manager_process_add_subscription: name:  (uORBManager.cpp:339)
   2643332: HAP:48:[posix-uORB::Manager::process_add_subscription(351)]DeviceNode(sensor_combined) not created yet (uORBManager.cpp:351)
   2644136: HAP:61:2571514 I2C transfer timed out (MPU9250_mag.cpp:381)
   2644136: HAP:61:2571559 error reading mag whoami reg: -1 (MPU9250_mag.cpp:265)
   2644136: HAP:61:2571587 MPU9250 mag not detected. (MPU9250_mag.cpp:81)
   2644136: HAP:61:2571612 mag initialization failed 1 tries (MPU9250_mag.cpp:191)
   2645024: HAP:61:3682052 I2C transfer timed out (MPU9250_mag.cpp:381)
   2645024: HAP:61:3682095 error reading mag whoami reg: -1 (MPU9250_mag.cpp:265)
   2645024: HAP:61:3682123 MPU9250 mag not detected. (MPU9250_mag.cpp:81)
   2645024: HAP:61:3682148 mag initialization failed 2 tries (MPU9250_mag.cpp:191)
   2645912: HAP:61:4791611 I2C transfer timed out (MPU9250_mag.cpp:381)
   2645912: HAP:61:4791652 error reading mag whoami reg: -1 (MPU9250_mag.cpp:265)
   2645912: HAP:61:4791680 MPU9250 mag not detected. (MPU9250_mag.cpp:81)
   2645912: HAP:61:4791705 mag initialization failed 3 tries (MPU9250_mag.cpp:191)
   2646800: HAP:61:5901914 I2C transfer timed out (MPU9250_mag.cpp:381)
   2646800: HAP:61:5901956 error reading mag whoami reg: -1 (MPU9250_mag.cpp:265)
   2646800: HAP:61:5901985 MPU9250 mag not detected. (MPU9250_mag.cpp:81)
   2646800: HAP:61:5902010 mag initialization failed 4 tries (MPU9250_mag.cpp:191)
   2647688: HAP:61:7012191 I2C transfer timed out (MPU9250_mag.cpp:381)
   2647688: HAP:61:7012233 error reading mag whoami reg: -1 (MPU9250_mag.cpp:265)
   2647688: HAP:61:7012262 MPU9250 mag not detected. (MPU9250_mag.cpp:81)
   2647688: HAP:61:7012289 mag initialization failed 5 tries (MPU9250_mag.cpp:191)
   2647696: HAP:61:7022334 failed to initialize mag! (MPU9250_mag.cpp:202)
   2647696: HAP:61:7022364 Magnetometer initialization failed (MPU9250.cpp:176)
   2647696: HAP:61:MPU9250 device id is: 4288784 (df_mpu9250_wrapper.cpp:291)
   2647697: HAP:60:7023769 FIFO overflow (MPU9250.cpp:359)

Also, trying to start the MPU9250 without mag won't work at that point:

   2783748: HAP:52:32776650 Added driver f0945a50 /dev/imu1 (DevMgr.cpp:145)
   2783828: HAP:52:32876989 Reset MPU9250 (MPU9250.cpp:89)
   2783829: HAP:52:32878215 initializing mpu9250 driver without mag support (MPU9250.cpp:110)
   2783829: HAP:60:32878633 IMU temperature initialized to: 34.997231 (MPU9250.cpp:466)
   2783833: HAP:60:32883110 FIFO overflow (MPU9250.cpp:359)
   2783833: HAP:60:32883357 FIFO overflow (MPU9250.cpp:359)
   2783834: HAP:52:MPU9250 device id is: 4288784 (df_mpu9250_wrapper.cpp:291)
   2784168: HAP:69673:gyro has timed out (sensors.cpp:2200)
   2784168: HAP:69673:failing over to second gyro (sensors.cpp:2214)

FYI: @jywilson thoughts?

[jywilson]

@julianoes @mcharleb Julian,

Perhaps if the mag is left powered on, it can't be reinitialized.

What I would recommend is to enhance the detect function in the mag class to power cycle the mag, in the same way that the get_sensitivity_adjustment method does. This might be enough to make it responsive again.

Jim W.

[julianoes]

Thanks @jywilson. I tried to do what you suggested but it didn't work because the I2C transfer itself always seemed to time out. I did some tries to reset the master I2C, and reset the I2C slave. Results were not really conclusive but I haven't seen the problem since. I'll give it some more testing this week.

[julianoes]

Unfortunately, the problem is back.

[jywilson]

Just to clarify, the mag only has init problems after stopping and restarting the flight stack. Is this correct?

It would probably make sense to modify the unit test to try and repro this issue, just to eliminate possible interactions with the flight stack.

I will try to look at this issue later this week and assign the problem to myself.

[julianoes]

Just to clarify, the mag only has init problems after stopping and restarting the flight stack. Is this correct?

Yes correct. I can reproduce it as well using the df_imu_test though, however, this one gets stuck and doesn't seem to return (and doesn't react to Ctrl+C).

I would appreciate your help, I haven't had any luck with all my tries so far.

[julianoes]

@jywilson Did you find anything here? Were you able to reproduce it?

[julianoes]

The failure is also reproducible on the RPi2 with Navio2.

[julianoes]

Fixed by #117.