I2C / Wire not reliable

[extrapixel]

after the other Wire/i2c issues have been closed ( #166 etc), and the speed got better, some I2C devices started working with the 101.

However, it's still far from really working. Basically, in practice I2C is not usable on the 101 - it's slow, unreliable. Code / hardware that works with all other Arduinos, does not work with the 101.

I have multiple I2C devices here which work on all official Arduinos, except the 101.

BTW I'm using the latest code from this repo.

We're the manufacturer of the Tentacle Shields, these are isolation shields for sensor circuits made by Atlas Scientific. E.g. the pH sensor circuit: https://www.atlas-scientific.com/_files/_datasheets/_circuit/pH_EZO_datasheet.pdf

Schematics of the shield mentioned above is here: https://github.com/whitebox-labs/tentacle-mini-oshw/blob/master/hardware/T2_mini_schematic.pdf

@bigdinotech your fix did help somewhat, but unfortunately Wire is still unreliable. If someone's working on this, I can send i2c devices to test, if needed.

[intel-larry]

Hi Patrick, I'm sorry to hear we're having issues interfacing with your devices. Of course, we do want everything to work seemlessly. I've personally tested a lot of I2C devices, ranging from various sensors to RTC boards and LCD displays and generally, they've been working. I looked on your website for example code and it looks mostly like plain-jane stuff that I'd think should work.

In 30 years of working with I2C, I can say, that it can be tricky to get things working together sometimes. We would really like to see your devices and would greatly appreciate a few samples so we can work on them in our lab.

[extrapixel]

@intel-larry thanks for looking into this, greatly appreciated! Could you use the contact form here https://www.whiteboxes.ch/contact/ to send me your shipping address?

[eriknyquist]

@extrapixel done!

[eriknyquist]

Hi @extrapixel, I received your I2C devices (and a lovely handwritten note!) this morning. Do you have a sketch that you could share with us, which demonstrates the performance difference between an Arduino 101 and an Arduino Uno?

[PaulStoffregen]

@eriknyquist - Did you see this? https://github.com/01org/corelibs-arduino101/issues/166#issuecomment-229851730 (if this link doesn't take you right to my posts, scroll around looking for a photo of stack of various Arduino boards... it's the only actual photograph on that mammoth-length conversation which isn't a screenshot or diagram)

In the simple test I ran, Arduino Uno and other boards took approx 1.5 ms. Arduino 101 was ten time slower, due to some sort of stall in the Wire library. I posted the complete code I used for testing on all boards. You can see 4 oscilloscope screenshots I posted, showing the long delay. @mjs513 also posted several tests with another chip.

I'll make the same offer for you that I made for @bigdinotech and @piroozn - just email me your shipping address and I'll send you the hardware for free. You can just plug that shield into your Arduino 101 or any other Arduino board and run the simple sketch to measure the time taken to read 12 bytes from that EEPROM chip. Truly, that shield is just sitting here collecting dust and precious bench space, so you'd practically be doing me a favor taking it off my hands! If you want it, just email me directly (paul at pjrc dot com) and I'll put it in the mail to you - anywhere in the world the US postal service can send.

[eriknyquist]

Absolutely @PaulStoffregen, send us that shield, extra testing hardware is always appreciated! I'll send you an email with a postal address.

In this particular case, @extrapixel has also sent us some specific hardware, and it would help me greatly if he could provide a sketch for his hardware that demonstrates the slowness.

[extrapixel]

@eriknyquist yes, we've got some code. It's not made especially for testing/comparing, but it should demonstrate the issue clearly. There's the "Tentacle setup sketch" to interactively talk to the stamps (https://github.com/whitebox-labs/tentacle/tree/master/examples/tentacle-setup/tentacle_setup). That code is probably a bit convoluted due to the auto-detection routines, compatibility with UART devices etc (it's meant to be easy for the "user", not easy to understand code).

Then there is the "continuous reading" example which is kind of basic, but might need some adjustments of i2c addresses etc (https://github.com/whitebox-labs/tentacle/tree/master/examples/continuous/i2c_continuous)

If the above is too much a hassle, I can create some simple test case for you, but I can do this tomorrow earliest.

[PaulStoffregen]

@eriknyquist - It's going in the mail today. You should have it by the weekend. You can find the simple self-contained test program on https://github.com/01org/corelibs-arduino101/issues/166#issuecomment-229852016

[eriknyquist]

Brilliant, thanks @extrapixel and @PaulStoffregen

[mjs513]

I believe @bigdinotech already has a Mpu6050 chip accrording to tje last post on. isssue 166. If you need the libs and sketch let me know and I can post them. If you want the 6050 the me know. I think I have an extra one laying around. Pretty much gave up on getting the i2c issue fixed.

Mike

[PaulStoffregen]

@eriknyquist - Did the EEPROM test shield arrive safely?

[eriknyquist]

yes, literally just this morning! Thanks again.

[mjs513]

@eriknyquist - @PaulStoffregen - @extrapixel - you may want to check the changes that @noelpaz made in a JIRA - discussion seems to still be continuing under issue #166.

[mjs513]

@eriknyquist - @PaulStoffregen - @extrapixel - @noelpaz

I just tested the latest GITHUB changes that incorporated the @noelpaz updates associated with issue #166. Again using the MPU6050 there was significant improvements to the timings: 1.51ms with setClock(100000L) and 0,5 ms using setClock(400000L). This is compared to my original results:

Mega 0.426 0.178 milliseconds 101 4.28 3.29 milliseconds.

haven't done any timing tests with the Fram but maybe Paul or you all can post for reference.

[PaulStoffregen]

I sent my eeprom shield to Erik. The complete code and benchmarks on #166. I'm working on a new Teensy board for release in October, so can't do much until after that's completed.

[mjs513]

Can't wait for the new teensy. Can you share any specifics or is it on the forum

On Aug 24, 2016 10:19 AM, "Paul Stoffregen" notifications@github.com wrote:

I send my eeprom shield to Erik. Working on a new Teensy board for release in October, so can't do much until after that's completed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/01org/corelibs-arduino101/issues/238#issuecomment-242081539, or mute the thread https://github.com/notifications/unsubscribe-auth/AFHhxdO08gkTSwJRbsshlGk9_ETajZeJks5qjFLngaJpZM4JMpg6 .

[eriknyquist]

Yes, I2C performance has improved dramatically. From using Paul's EEPROM shield, and just looking at the signals on a scope, things look much tighter now. Would appreciate if @extrapixel specifically could confirm that his I2C situation has improved (note this fix is not with Arduino yet-- you will have to grab the latest from here).

[extrapixel]

@eriknyquist alright, will test some time next week. Did you have a chance to hook up the hardware I sent you?

[extrapixel]

@eriknyquist Situation has improved a lot! One can now stably talk to the i2c devices. However:

~I have multiple I2C devices here which work on all official Arduinos, except the 101.~

The devices work the same on all Arduinos, except the 101.

There are still problems when scanning the bus - seems asking for an address that isn't there breaks things. I have only tried with the devices listed in OP. While it might be a problem of the devices, it only shows on the 101 (and Edison) - works on all other official R3 boards.

If I talk to the devices directly knowing the address, all works well. But whenever a transmission is started/ended with an address that does not exist, things get weird. Can be somewhat replicated using the bus-scanner from the IDE examples. I have a device at address 54 and 55:

Begin Test

Start I2C Bus Scan from 53 to 56.....
53: 2
54: B
55: B
56: C
done

Start I2C Bus Scan from 53 to 56.....
53: 2
54: 0
55: 0
56: 2
done

Start I2C Bus Scan from 53 to 56.....
53: C
54: 2
55: 0
56: 2
done

Start I2C Bus Scan from 53 to 56.....
53: C
54: 2
55: 0
56: 2
done

(continues repeating the last one) Notice how it finds them in the first test, but fails after.

If I double-check, scanning each address twice, I get this:

Begin Test

Start I2C Bus Scan from 53 to 56.....
53: 2
53: 12
54: 2
54: 0
55: 0
55: 0
56: 2
56: 12
done

Start I2C Bus Scan from 53 to 56.....
53: 2
53: 2
54: B
54: 11
55: B
55: 11
56: C
56: 2
done

(continues switching between the two) At the moment, this works a as workaround.

Not touching an empty address: all fine:

Start I2C Bus Scan from 54 to 55.....
54: 0
55: 0
done

Start I2C Bus Scan from 54 to 55.....
54: 0
55: 0

[extrapixel]

@kitsunami would you mind re-opening this?

[extrapixel]

@eriknyquist btw the hardware i'm having trouble with is the stuff i sent you - should be able to replicate

[piroozn]

@eriknyquist and @PaulStoffregen I found the issue with I2C performance issue before and reported it. I remember it was a fix delay in a readbyte and writebyte. For test, I removed it and Arduino 101 sketch using I2C display start running much faster than Uno specially by using @bigdinotech setclock implementation. Before change, it was 10 times slower because someone just used a delay after every access. I sent my address to Paul and will try to test his board when I get a change. I am short on time for now and try to fit it in my schedule.

[PaulStoffregen]

Are you expecting anything from me? I sent the test board I made to Erik Nyquist several months ago. I do not have it any more, nor do I need it back. At this time all my dev cycles are going into a USB host library, so there's probably not much I can realistically do to help at this time. Sorry.

[eriknyquist]

This issue was specifically opened for different reasons (library being unreliable); please open a new issue if the bus_scan sketch is broken, since it is distinctly different.

Also, I tried running the bus_scan sketch using the latest from Github, and I didn't see the problem you described, everything was working fine, even with no devices attached.

I'm closing this-- feel free to open a new issue if you still can't make the bus_scan sketch with the latest

[extrapixel]

it's not a bug in bus-scan - but the bus-scan demonstrates the problem most easily. @eriknyquist , i've sent you lots of hardware - have you ever tried it?

(in other words - please re-open. the library is clearly unreliable. at least for the devices i've tried)

[eriknyquist]

@extrapixel Yes, of course I've tried your hardware. Although last Friday, I just grabbed some other I2C shield that happened to be on my desk (we have a lot of I2C test shields and devices lying around...). Let me dig out your tentacle shields and try them.

Have YOU tried the latest code? The reason I ask is because, on the date when you posted your comment about this problem, I went to investigate it and found that the bus-scan sketch was completely (and quite obviously) broken, i.e. goes into an infinite loop printing garbage, due to an unrelated and quite nasty bug in the linker script, which is now fixed https://github.com/01org/corelibs-arduino101/commit/d6c68f26914bf9508e8b76081d383a5cd432c7b6

I'm not saying this is the fix for your problem (but hey, it might be, give it a try), but this fact (and the fact that your bus-scan output is printed in a different format than the sketch included with the IDE), leads me to believe that 1) you did not try the latest code (because, on the day you reported it, the latest code was very broken in ways you didn't mention), or 2) you are using your own modified version of the bus scan sketch which you have not yet shared.

Can you share the bus-scan sketch code you were using?

[eriknyquist]

Using your tentacle mini shield, with the pH and RTD sensors attached, the bus scan sketch gives the following output. Only shows one device, which doesn't seem right, however it looks pretty different from the output you showed...

Start I2C Bus Scan from 1 to 127.....
address: 1          address: 2          address: 3          address: 4        
address: 5          address: 6          address: 7          address: 8        
address: 9          address: 10         address: 11         address: 12       
address: 13         address: 14         address: 15         address: 16       
address: 17         address: 18         address: 19         address: 20         
address: 21         address: 22         address: 23         address: 24       
address: 25         address: 26         address: 27         address: 28       
address: 29         address: 30         address: 31         address: 32       
address: 33         address: 34         address: 35         address: 36       
address: 37         address: 38         address: 39         address: 40       
address: 41         address: 42         address: 43         address: 44       
address: 45         address: 46         address: 47         address: 48       
address: 49         address: 50         address: 51         address: 52       
address: 53         address: 54         address: 55         address: 56       
address: 57         address: 58         address: 59         address: 60       
address: 61         address: 62         address: 63         address: 64       
address: 65         address: 66         address: 67         address: 68       
address: 69         address: 70         address: 71         address: 72       
address: 73         address: 74         address: 75         address: 76       
address: 77         address: 78         address: 79         address: 80       
address: 81         address: 82         address: 83         address: 84       
address: 85         address: 86         address: 87         address: 88       
address: 89         address: 90         address: 91         address: 92       
address: 93         address: 94         address: 95         address: 96       
address: 97         address: 98         address: 99  found  address: 100      
address: 101        address: 102        address: 103        address: 104      
address: 105        address: 106        address: 107        address: 108      
address: 109        address: 110        address: 111        address: 112      
address: 113        address: 114        address: 115        address: 116      
address: 117        address: 118        address: 119        address: 120      
address: 121        address: 122        address: 123        address: 124      
address: 125        address: 126        address: 127        
done

[PaulStoffregen]

I just did another quick test. I can confirm the Wire library is not working reliably on Arduino 101.

I tested with the Teensy Prop Shield, which you can also get from Sparkfun or from Adafruit if you wish to reproduce these results. I connected 4 wires between this hardware and Arduino 101: SDA, SCL, 3.3V, GND.

The Arduino Boards Manager says I have Intel Curie Boards version 1.0.7 installed.

I ran this I2C scanner sketch, which is derived from the scanner example on Aduino's playground, with minor edits to print the names of commonly used chips. Hopefully this is enough info for you to be able to reproduce the results?

When I run this sketch on Arduino Zero, I see correct detection of the 4 addresses in the serial monitor:

Scanning...
Device found at address 0x1E  (LSM303D,HMC5883L,FXOS8700,LIS3DSH)
Device found at address 0x20  (MCP23017,MCP23008,PCF8574,FXAS21002)
Device found at address 0x28  (BNO055,EM7180,CAP1188)
Device found at address 0x60  (MPL3115,MCP4725,MCP4728,TEA5767,Si5351)
done

Scanning...
Device found at address 0x1E  (LSM303D,HMC5883L,FXOS8700,LIS3DSH)
Device found at address 0x20  (MCP23017,MCP23008,PCF8574,FXAS21002)
Device found at address 0x28  (BNO055,EM7180,CAP1188)
Device found at address 0x60  (MPL3115,MCP4725,MCP4728,TEA5767,Si5351)
done

Scanning...
Device found at address 0x1E  (LSM303D,HMC5883L,FXOS8700,LIS3DSH)
Device found at address 0x20  (MCP23017,MCP23008,PCF8574,FXAS21002)
Device found at address 0x28  (BNO055,EM7180,CAP1188)
Device found at address 0x60  (MPL3115,MCP4725,MCP4728,TEA5767,Si5351)
done

When I run it on Arduino 101, this is the result:

I2C Scanner
Scanning...
No I2C devices found

Scanning...
Device found at address 0x1E  (LSM303D,HMC5883L,FXOS8700,LIS3DSH)
done

Scanning...
Device found at address 0x1E  (LSM303D,HMC5883L,FXOS8700,LIS3DSH)
done

Arduino 101 only finds 1 of the 4 responding addresses (one of the 3 chips on this board responds as 2 different addresses, correct well documented behavior for this hardware). Arduino 101 also fails to find any I2C addresses on the first scan.

Just in case any of this info about how I tested is unclear, here's a photo of the Arduino 101 test on my workbench:

If you'd like to reproduce this result, the hardware is readily available, currently in stock at all 3 sources I listed above. Just connect it with 4 wires as I did here and run the same scanner example.

Hopefully this test is documented well enough and can help you to resolve the Wire library issues.

[eriknyquist]

Thanks @PaulStoffregen, I can definitely see this behaviour with the devices I'm using as well (photo attached). Also seems to consistently find nothing on the very first iteration.

Since the bus scanning is implemented entirely in the example sketch, I will start by looking for bugs there (start is the key word here. I'm not trying to deny there is a bug in the library itself, but since everyone is reporting problems with bus scanning, it makes pretty good sense to start there). It sounds like you and @extrapixel might be describing different problems, although I'm still not entirely clear on the details of @extrapixel's problem.

Any, thanks for continuing to provide test hardware & information, guys. We'll get this one yet :)

[PaulStoffregen]

Since the bus scanning is implemented entirely in the example sketch, I will start by looking for bugs there

Maybe worth mentioning again this scanner sketch works on Arduino Zero and every other Arduino board which supports Wire. It's been widely used for at least 5 years. The sketch also doesn't do anything particularly special. It merely calls the Wire library functions.

I'm going to go out on a limb here and suggest this scanner sketch is almost certainly not the cause of the problem!

[eriknyquist]

@PaulStoffregen, agreed, I am mistaken. I've been using the bus-scan sketch included with the IDE, which appears to have been written specially for 101 (not sure why, though). I will use the scanner sketch from the Arduino PLayground from now on....

[extrapixel]

thanks guys for getting back to this. @eriknyquist , I'm on official 1.8.1 everything up-to-date, but no dev versions. My test code was based on the included bus scan example included in the IDE. I changed it to run continuously and to print the return value of endTransmission().

I'm rigging things up now and adapt the Playground scanner to demonstrate the issue. @eriknyquist , you can also explore the issue using the interactive setup sketch for tentacle. As soon as a non-existant address is probed, things go weird.

[PaulStoffregen]

If you make any interesting improvements to the Arduino Playground scanner, I'd be curious to hear.

My version (used in the test above) has a list of the commonly used I2C chips for prettier output, but otherwise it's basically the same as the Playground sketch.

At the very least, if there's any chip you know is being used on any widely used Arduino shield not already on my list, please send me the info so I can add it. :)

[eriknyquist]

@extrapixel are you using the latest snapshot of this repository? There have been an awful lot of fixes since 1.0.7--- this I2C problem you describe DOES still occur on the latest, however there are some other fixes you definitely want (like that fix for proper alignment of the .data section, which can manifest itself in horrible and unexpected ways!)

[extrapixel]

here's some adapted code of @PaulStoffregen - just meant to explore the scanning issue. You can adjust the range to scan via startAddress / stopAddress.

#include <Wire.h>

void setup() {
  Wire.begin();
  Serial.begin(9600);
  while (!Serial);        // Leonardo: wait for serial monitor
  Serial.println(F("\nI2C Scanner"));
}

void loop() {
  byte error, address;
  int nDevices;
  int startAddress = 54;
  int stopAddress = 56;

  Serial.println(F("Scanning..."));

  nDevices = 0;
  for (address = startAddress; address <= stopAddress; address++) {

    Wire.beginTransmission(address);
    error = Wire.endTransmission();

    Serial.print(address);
    Serial.print(":\t");

    if (error == 0) {
      Serial.println("found");
      nDevices++;
    } else  {
      Serial.print("error: ");
      Serial.println(error);
    }
  }

  if (nDevices == 0) {
    Serial.println(F("No I2C devices found\n"));
  } else {
    Serial.print(nDevices);
    Serial.println(F(" found.\n"));
  }
  delay(1000);           // wait 5 seconds for next scan
}

My devices are at 54 and 55. It behaves very differently depending on the scan range. For me, it mostly follows this rule: if the previous address has NO device, the next scan will not detect a device, even if it is there. If the previous address HAS a device (even if it does not get detected by the scan!), the next one will be detected. This seems to be true, even if scanning order is reversed, or if picked randomly. The scan of the previous address influences the scan of the next. I'll try with the snapshot now @eriknyquist

[eriknyquist]

@extrapixel thanks for that detailed information- I can confirm I've seen the exact same behaviour with all the devices I've tried. Also, the return value from endTransmission is oddly inconsistent when scanning an address... I'm seeing a repeating pattern of three return values; 12, 2, 2 .......

I can figure out where those individual return codes are coming from, but why the return value is alternating between them like that is a mystery to me. I'm a bit confused by some of the output you posted (I assume it was produced by your sketch in your last comment)

53: 2
53: 2
54: B
54: 11
55: B
55: 11
56: C
56: 2

Are some return codes being printed in hex (i.e. 0xB, 0xC), and others not? I don't get how there is a "B" and also an "11".

Anyway, I'm wondering if you can also seeing alternating between 12 and 2, or is there some kind of corruption also happening in your situation?

[extrapixel]

I'm sorry, the code that produced the output in my earlier comments was only temporary. I have no saved copy of it. It was not the output of the code I posted.

I can confirm: With the newly posted code (stripped down version of @PaulStoffregen 's Scanner) I also get patterns of 12, 2, 2 and an occasional 11. In case of 11, all of the devices will be 11... just to go back to 12 , 2 in the next iteration.

The return value is not just inconsistent between scans - it's also not consistent with the docs: https://www.arduino.cc/en/Reference/WireEndTransmission

Returns

byte, which indicates the status of the transmission:

0:success 1:data too long to fit in transmit buffer 2:received NACK on transmit of address 3:received NACK on transmit of data 4:other error

[PaulStoffregen]

The Wire library has used these same 5 return codes for many, many years. While most Arduino sketches ignore it, or look for only 0 vs non-zero, returning anything else risks Arduino compatibility.

[PaulStoffregen]

Just curious if this issue was ever investigated?

I've been working on Teensy's Wire library recently. It's remarkably difficult to properly handle all possible cases that can happen with multi-master systems and recovery from all possible cases of electrical corruption on the data signals.

[extrapixel]

@eriknyquist any progress on this? Is there something I can do to help?

[eriknyquist]

Sorry, I'm not able to work on this right now.

[RonaldPhilipsen]

any progress on this? i'm running into problems where data is nacked