Massive error in i2c

[gendebeat]

I flash the latest 1.3 RC2 , because the earlier have a problem to detect the magnetometer (BMP280) on my naze. i think this issue may be corrected. but now after 1 minute (aprox) the i2c error increase from 4 to a masive number (aprox +200 error per second) and Cycletime goes from 2000 to 11000. An of couse the FC freeze. thanks to all people who work to make inav/cleanfligth better

[digitalentity]

Sounds like a hardware error on the bus.

Please share details of your setup. What board, what sensors are connected? Do you have external compass connected to the board?

[digitalentity]

Looks like I2C timeout (#713, #720) it's not the only problem. I believe I2C driver requires a serious rewrite to properly handle all bus conditions. Or at least be simplified to the level of one used in F3. I'll look into it once I get INAV-1.3 released.

[gendebeat]

ok thanks, i had not seen the other entries. i believed that the problem with the magnetometer was already solved. im not a programer, sorry. My board is a NAZE32 rev6a 10DOF: MPU6500 BMP280 and external HMC5883 (on board disable by cut) Of course i try cleanfligth and fly good, and all sensor have very few i2c error.

[gendebeat]

Hello, im try the last release 1.3 and i have exactly the same problem. i2c error increase after 1 minute (or less). and the boad freeze because cycle time increase too. I think is not a hardware problem because with INAV 1.2.1 i have not this problem. I try CLEANFLIGTH and with the last release 1.14 i have this problem too. (earlier cleanfligth releases no problem),. i need to wait more time to try inav on my FC.. :(

[digitalentity]

@gendebeat this looks like hardware issue which current implementation is unable to cope with - looks like you're the only one having this problem.

I'm looking into making I2C driver more robust but that will take time.

[brumbad]

I confirm this with dof6/rev6a naze32 INAV-1.3, 1.2 and 1.2.1. Baro connected - gy-68 (BMP180). MC time to time startup correct, and in most case after short time become very slow with many i2c errors. This can lead to cpu consumption (600-700%) with cycle time about 16000-17000.

Baro connected to MC pin's with short (2-3cm) wire on board.

[digitalentity]

@gendebeat, @brumbad Can you please test this hex https://www.dropbox.com/s/tc48euihxlugib0/inav_1.5.0_NAZE_a0207d0.hex?dl=0 ?

It has massive changes to i2c bus driver, maybe it will solve the issue for you.

[gendebeat]

@digitalentity hello, this hex work correct!! i not flying, i wil try this weekend. But now i can say that there some i2c error, but dont freeze. CPU load is allways at 2-3%. cycle time 2000 thanks!

[digitalentity]

@gendebeat if i2c error count increases over time it's an indication of unstable hardware. I wouldn't trust that setup. But still, good to know that a single bus glitch won't cause a crash with 100% probability.

[digitalentity]

I'm closing this issue now

[AvengerIl]

Can confirm the latest 1.6-rc2, and 1.5.1 still have this problem on spracing f3 acro (clone) with external hmc5883 gyro. (so does betaflight 3.1.5). i2c errors go through the roof, cpu usage to 900%.

[digitalentity]

@AvengerIl It's very likely faulty hardware. I2C code in 1.6 is reasonably robust - if it doesn't work - 99% possibility that there is something wrong with the hardware.

[AvengerIl]

@digitalentity FC or the ublox n7 gps with compass module more likely to be at fault? any external mags that are known to be good?

[Redshifft]

@AvengerIl you could go for a Naza gps then you won't require the I2C (mag and gps are encoded in one serial stream) and they work well in iNAV

[AvengerIl]

@Redshifft thanks... do have one of those lying around... I tried a F1 NAZE32 rev6b with the same i2c mag and it worked fine, does not lock up... must be the spracing f3 clone w/ osd causing the problem.

too many options, too many decisions... i really want 8khz and dshot running.. AND GPS..

[Redshifft]

I hear an F4 calling you ;)

[Waltec]

Using Omnibus F3 AIO and Neo 6m after a minute or two I get I2C errors and mag unhealthy warning. All works okay at reboot.

I'm troubled by the statement that it must be a hardware (mag) issue as I've tried two different GPS' with the same result, eventual failure (as indicated by Inaz or EZGUI) of mag on GPS module. These modules are known workers on APM 2.5 and 2.6 boards.

I saw this failure once in flight as I was setting up my copter. It just turned and augered in from low altitude.

Any chance for a rethink on the cause? Why would I2C be error free for more than a minute or and then start getting errors?

Hardware problems seems a bit too convenient but i'll admit, I'm far from an expert at these things.

Thoughts or further insight?

[mgaller]

@Waltec I have the exact same problem with the same board and neo-m8n. Did you found a solution?

[Waltec]

Not yet. I've been pointed in a couple of directions and I have my own theory. The one thing I tried was eliminating the overclocking of the I2C that someone suggested. That had no effect that I could tell.

Its been suggested that the mag should have 3.3v vs the 5v that my gps/mag module specifies but I've not explored that yet...I don't have a 3.3v source on the copter at present. Tough to wrap my head around that one as my module(s) call for 5v, and, they've worked fine on other FCs various APMs that is). I'll explore that at some point but I've not had any real time to get back to my bench.

My theory though is that (as I read somewhere) since the I2C bus shares resources with the UART3 (I think it's #3 is what I've read) there is some sort of data collision happening. I'm working from memory here so bear with me...

I think it was dronetrest.com that spells out this sharing issue specific to using ppm receivers (I'm using a knockoff Spektrum satellite receiver which connects via different pins than a standard ppm receiver but I'm guessing that qualifies). I'm guessing that the ppm data and the mag data "collide" on the shared resource and cause the I2C errors. Again on dronettest I think it is suggested to recompile the Inav firmware but doing so while uncommenting the "avoid_using _uart3" or somesuch statement. I'm a bit over my head on all this to begin with but recompiling is something I've not done before,so, I haven't tested this fix yet. Not sure if I will because recompiling is a whole new set of learning and as best I can tell requires a Linux setup which I don't have right now. Add to that that I'm wondering what problems you might be inviting by changing the firmware... why isn't Uart3 being "avoided" in the first place?

In looking at the Inav wiki I see that the Omnibus F3 AIO isn't in the fully supported category so I'm wondering if that's not a contributing factor on this issue.

I'll post back if I did pursue any of these fixes but for now I've opted to swap in an SP F3 board thinking it might be the quicker path to getting my copter back in the air again.

Not sure I've furthered the cause any but thanks for the question/response. Hopefully I've suggested something to go after if you are so inclined. If you do chose to tinker further, and something works or shows promise, I'd enjoy hearing about it!

Regards and Good Luck, Waltec

Sent from my iPhone

On Apr 20, 2017, at 2:53 AM, Michael Galler notifications@github.com wrote:

@Waltec I have the exact same problem with the same board and neo-m8n. Did you found a solution?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Waltec]

I forgot to mention that along with the 3.3v question on the mag that there is also a question regarding having too many pull up resistors...one on the FC and one on the gps/mag module. I'm not entirely clear but to my understanding the FC having a resistor built in is what's causing the voltage problem since the module has one too.

Sent from my iPhone

On Apr 20, 2017, at 2:53 AM, Michael Galler notifications@github.com wrote:

@Waltec I have the exact same problem with the same board and neo-m8n. Did you found a solution?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[stronnag]

Strange, I have external mag on I2C (Beitian BN880 GPS) working OK on Omnibus F3 AIO. Using UART3 for serial RX, thusly:

serial 0 17 9600 38400 9600 38400
serial 1 2 19200 38400 0 115200
serial 2 64 115200 38400 0 115200

Tested on 1.6.1 / 1.6.3 (development) and 1.7.0-RC1).

I2C connected to the J3 (SCL / SDA),[not J22 (PWM7/8)]

[mgaller]

@Waltec Thanks to you i found the problem. The problem is the shared UART3, i dont use ppm but i use motor 5 and 6 and they are using the shared pins from UART3. If i switch from hexacopter to quadcopter the problem does not happen anymore. If i renable hexacopter problem occurs after 1 minute.

But i did not found any solution yet.

[mgaller]

@stronnag Big thanks for your hint with I2C connected to J3 instead of J22. No more I2C errors. But really weird that the relocated pins are "broken" and the original still working. @Waltec i had to swap SCL and SDA wires before soldering to J3.

[Waltec]

Great insight. Honestly, I tried those connections at one point and couldn't get it to work. User error I'm sure.

Sent from my iPhone

On Apr 20, 2017, at 2:27 PM, Michael Galler notifications@github.com wrote:

@stronnag Big thanks for your hint with I2C connected to J3 instead of J22. I dont get any I2C errors anymore. @Waltec i had to swap SCL and SDA bevor soldering to J3.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Waltec]

Thanks for the feedback.

Tried last night with two different gps modules. M6n ublox. Gps would show up blue in status bar but magnetometer remained red. Swapped wires in case pins were wrong but no joy.

I didn't have much time so couldn't research myself but...

1. Is there a way, cli statement maybe, to determine if mag is on the bus and communicating?

2. I'm recalling that when mag shows red I can't do a calibration (board was actually showing calibration complete which seemed strange) but I didn't check to see if the option was available. Perhaps I need to view the sensors tab and then go back to the calibration tab to see if the calibration process "resets".?

I'm expecting that when the mag is working (communicating at least) it at least goes blue in the status bar similar to what the gps does...could be wrong on that.

One thing that has me thinking I'm confused there is that I can never get a solo mag module to register on the status bar either. I have a couple of those lying around and whenever I connect them I don't see evidence that the board sees them. Complicating this is the fact that the mag module doesn't have a status light that I can see.

If anyone has any thoughts on this just let me know.

Regards

Sent from my iPhone

On Apr 20, 2017, at 2:27 PM, Michael Galler notifications@github.com wrote:

@stronnag Big thanks for your hint with I2C connected to J3 instead of J22. I dont get any I2C errors anymore. @Waltec i had to swap SCL and SDA bevor soldering to J3.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[peakhunt]

I am having the same issue with 1.6.1 on my chinese clone SPRacing F3. With cleanflight, although there are some i2c erros but not this much. So I downgraded iNav to 1.4 and the count stops increasing. Let me take a look at the source code on the coming weekend although I am new to this source code.

[Waltec]

That would be great. You mention the SP F3 where I'm having the problem too but I also have it on a F3 Omnibus AIO. Both boards get runaway I2c errors on 1.6.1. On the Omnibus others are able to use the dedicated sda scl pins for the mag but I can't get those to work.

Between the two board I probably have 6 controllers from various sources that I can't use. That I can have some success by downgrading makes me wonder about the latest code but again, I'm out of my depth there.

Sent from my iPhone

On Apr 28, 2017, at 2:53 AM, Hawk Kim notifications@github.com wrote:

I am having the same issue with 1.6.1 on my chinese clone SPRacing F3. With cleanflight, although there are some i2c erros but not this much. So I downgraded iNav to 1.4 and the count stops increasing. Let me take a look at the source code on the coming weekend although I am new to this source code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[peakhunt]

just replaced bus_i2c_stm32f30x.c in 1.6.1 tag with the same file in 1.4 tag, built a new image and flashed it. It looks like it's just working fine at least in iNav Configurator. I am not seeing any i2c error or unusable iNav Configurator due to excessive i2c error. I'm suspecting there is definitely something wrong with bus_i2c_stm32f30x.c since 1.5 release since this excessive i2c error occurs from 1.5 release in my experience.

[Waltec]

Pretty interesting. Given the breadth of boards that I'm having problems with it implies a firmware problem but this is unfamiliar territory for me.

Sent from my iPhone

On Apr 28, 2017, at 8:00 AM, Hawk Kim notifications@github.com wrote:

just replaced bus_i2c_stm32f30x.c in 1.6.1 tag with the same file in 1.4 tag, built a new image and flashed it. It looks like it's just working fine at least in iNav Configurator. I am not seeing any i2c error or unusable iNav Configurator due to excessive i2c error. I'm suspecting there is definitely something wrong with bus_i2c_stm32f30x.c since 1.5 release since this excessive i2c error occurs from 1.5 release in my experience.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[digitalentity]

SPRacingF3 from China running 1.7.0 for 5 hours straight. No issues.

[digitalentity]

For troubleshooting this I need either a failing board or a recording of SCL and SDA done with a logic probe at the moment of failure.

[peakhunt]

just a quick update. cut down the I2C clock from 400 KHz to 200 KHz and it looks like it's working. CPU load is around 4%. No errors so far. At 300 KHz, it doesn't work. At 100 Khz, it works but CPU load goes wayward like 70%. Now I am pretty sure this is a PCB issue. you know, impedance differs from one pcb to another. And No idea why i2c driver in 1.4 tag works. Maybe later.

This is the I2C_STANDARD_TIMING in bus_i2c_stm32f30x.c I am using.

define I2C_STANDARD_TIMING 0xA010031A

Just happy I don't have to throw away my precious chinese clone lol. Is it gonna fly?

[peakhunt]

did a short test flight in the morning with 200 KHz I2C clock. altitude hold worked near perfect. in gps hold, my quad kept circling although my magnetometer looks quite okay. maybe there are some configurations I missed. Anyway for those who are suffering from the same issue, here is a 1.6.1 with 200KHz I2C clock. Use it at your own risk!!!

[Waltec]

Konstantin,

Did anyone reply to you on this?

Once finished you'd return the board(s)? Are you looking for sp f3 and omnibus f3 aio? I've not looked...what country are you in?

Thanks,

Eric

Sent from my iPhone

On Apr 29, 2017, at 12:25 AM, Konstantin Sharlaimov notifications@github.com wrote:

For troubleshooting this I need either a failing board or a recording of SCL and SDA done with a logic probe at the moment of failure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[digitalentity]

@Waltec I'm in Russia. One russian guy is willing to send me his SPRacingF3 for investigation. I will gladly return the boards, but I have to warn you - the boards might not survive (I'll also be looking at hardware quality issues). What country are you in? I want to make sure that shipping the board back and forth won't cost more than a new board :smile:

[Waltec]

I'm in the US.  If I got a logic probe, could a novice give you useful information?

  From: Konstantin Sharlaimov <notifications@github.com>

To: iNavFlight/inav inav@noreply.github.com Cc: Waltec altawalt@yahoo.com; Mention mention@noreply.github.com Sent: Sunday, April 30, 2017 7:43 AM Subject: Re: [iNavFlight/inav] Massive error in i2c (#724)

@Waltec I'm in Russia. One russian guy is willing to send me his SPRacingF3 for investigation. I will gladly return the boards, but I have to warn you - the boards might not survive (I'll also be looking at hardware quality issues). What country are you in? I want to make sure that shipping the board back and forth won't cost more than a new board 😄 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Waltec]

I loaded this 200 khz i2c fix on my sp f3 board that wasn't showing either the mag or the barometer.  At boot up it immediately showed both.  Should be able to test fly later today.

  From: Hawk Kim <notifications@github.com>

To: iNavFlight/inav inav@noreply.github.com Cc: Waltec altawalt@yahoo.com; Mention mention@noreply.github.com Sent: Saturday, April 29, 2017 9:57 PM Subject: Re: [iNavFlight/inav] Massive error in i2c (#724)

did a short test flight in the morning with 200 KHz I2C clock. altitude hold worked near perfect. in gps hold, my quad kept circling although my magnetometer looks quite okay. maybe there are some configurations I missed. Anyway for those who are suffering from the same issue, here is a 1.6.1 with 200KHz I2C clock. Use it at your own risk!!!— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[brat002]

One more set of props were breaked after new try to fly in air my QR400+SPRacing F3 1.6.1. Controller is just stuck. This never was happened before with 1.3 firmware. Something really is wrong with new firmwares.

2017-04-30 18:07 GMT+03:00 Waltec notifications@github.com:

I loaded this 200 khz i2c fix on my sp f3 board that wasn't showing either the mag or the barometer. At boot up it immediately showed both. Should be able to test fly later today.

From: Hawk Kim notifications@github.com To: iNavFlight/inav inav@noreply.github.com Cc: Waltec altawalt@yahoo.com; Mention mention@noreply.github.com Sent: Saturday, April 29, 2017 9:57 PM Subject: Re: [iNavFlight/inav] Massive error in i2c (#724)

did a short test flight in the morning with 200 KHz I2C clock. altitude hold worked near perfect. in gps hold, my quad kept circling although my magnetometer looks quite okay. maybe there are some configurations I missed. Anyway for those who are suffering from the same issue, here is a 1.6.1 with 200KHz I2C clock. Use it at your own risk!!!— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/iNavFlight/inav/issues/724#issuecomment-298237538, or mute the thread https://github.com/notifications/unsubscribe-auth/ADyecNMr7CKFSkVbGdmUNrHDf8BxC8tYks5r1KOWgaJpZM4KfFnY .

[digitalentity]

@brat002 please consider sending your faulty FC to me for debugging. Once again - there is absolutely NO issues with genuine SPRacingF3 boards and with many Chinese clones as well. If I can't reproduce the issue I obviously can't troubleshoot it, can I?

[brooksymike]

I've just upgraded my SP Racing F3 EVO to a better GPS with external mag (HMC5883) and thought I'd add my experiences. Been having the same problem with the I2C errors which start to happen after about 1 minute and the mag then shows as unavailable. I tried to upgrade the firmware as detailed above but it says the firmware is corrupt. Maybe this is because my board is an EVO rather than a normal SP Racing F3? I'm not experienced in messing about with FC firmware so not sure how to go in and change the I2C clock manually. Maybe this fix wouldn't even apply to me as I've got a different board? Any thoughts appreciated?

[peakhunt]

yes it is only for SPRACINGF3 not for EVO. If you really wanna take the risk, I just uploaded 1.7.0 RC version with 200 KHz I2C clock speed for EVO here.

By the way I just switched to 1.7.0 RC version to sort out my position hold problem. Please share your experience so I can sort out my position hold problem.

[brooksymike]

I've just tried to install that firmware but I'm getting the same "firmware appears to be corrupt" message as I did with the normal SP racing F3 firmware. I did earlier successfully upload the standard version of 1.7.0 I found "inav_1.7.0_SPRACINGF3EVO.hex" (and I also tried version I found "inav_1.7.0_SPRACINGF3EVO_1SS.hex"). Don't know what the difference is but on both it still has the mag issue. Since posting the above I have also found this thread which gives info about powering from 3.3V and using pull up resistors: https://github.com/iNavFlight/inav/issues/1583 I have not tried this as I've only just found it so should probably go and give that a go.

[peakhunt]

hmmm honestly I have no idea what's going on. can you post a first hundreds bytes of the firmware file? You can just open it with notepad.

[brat002]

@digitalentity it is possible. Write me your address: brat002@gmail.com

[brooksymike]

I think I've done something stupid whilst trying to download. The Notepad file is full of rubbish compared to the other .hex files I downloaded. See at the bottom. I left clicked to download in Chrome like I normally would for any file but it just took me to the raw data, so I right slicked "Save target as:" Take it I need to left click and copy and paste into a notepad file. I've not downloaded anything from Github directly before (as I've just joined). Maybe I'm missing something.

"LC317" class="blob-code blob-code-inner js-file-line">:1013A80000F10E06AB88B38007464FF00109002014

:1013B800DFF8248198F81A30012B44460DD002D367

  <tr>
    <td id="L319" class="blob-num js-line-number" data-line-number="319"></td>
    <td id="LC319" class="blob-code blob-code-inner js-file-line">:1013C800022B1ED024E0424B1B7823B93846022159

:1013D80001221CF047FB012328E0FFF7A7FF40BBD1

  <tr>
    <td id="L321" class="blob-num js-line-number" data-line-number="321"></td>
    <td id="LC321" class="blob-code blob-code-inner js-file-line">:1013E80038466946FFF7B2FF9DF8003008B19B0701

:1013F80013D1002388F81A30B9F1000F0BD09946A1

[brooksymike]

Ok just got the modified EVO firmware installed. Problem with mag is still there but it is not consistently failing in the same length of time. At first power on it failed very quickly. Maybe 10 seconds. Rebooted it and this time it stayed good for at least a few minutes before I rebooted again. A few more times has revealed that it takes varying amounts of time to fail. Next I think I'll try powering the GPS / Mag with 3.3V and see if I can find any 1k resistors.

[brooksymike]

Just noticed that shortly after the compass fails and the I2C errors start counting up, the barometer starts jumping around all over the place. Don't particularly know what this means...............

[peakhunt]

how long is the cable for the external compass?

[brooksymike]

It's about 30cm. I've got the compass up on a vertical mount to keep it away from the other components.

[brooksymike]

Ok I managed to find some resistors of around 1K and I connected them between SCL and SDA and 3.3V as detailed in https://github.com/iNavFlight/inav/issues/1583. I also supplied the GPS with 3.3V. This looks like it's worked for me. I then tried connecting the GPS power to +5v instead of 3.3V and it also works. I haven't changed back to the normal firmware so I don't know if that's doing something as well but I don't have time to do that now. I also haven't tested it properly as all the connections were temporary so I can't fly it yet, but I will do. So for me at least it (appears so far) as though the pull up resistors are my problem rather that firmware issues. More testing required though.