search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.17k stars 5.01k forks source link

Issues with i2c-bcm2708 driver #1639

Closed ghost closed 8 years ago

ghost commented 8 years ago

I'm building a power management board for the RPI that controls the power to the RPI via software controls. Power on, power off, reboot, power on at a certain time, etc... The board is connected to the RPI via i2c.

During my tinkering, I've found a few issues with the i2c-bcm2708 driver. I've cleaned it up some and if I decide to publish it I was wondering if it would be better to publish it here or talk to Eric Anholt about sending it upstream. I saw very few differences between the downstream and upstream driver. I have absolutely no interest in trying to distribute a third i2c driver...

Here are the issues.

    1. In 4.8, the clocks are adjusted via cprman. It appears the clocks change a bit over time due to dynamic adjustment. Since the driver only computes the divider at boot time, it is often wrong.

The workaround I did was to recompute the driver on each bus transaction(not in the IRQ handler). This seems to fix things.

  1. If the driver is called through a power off handler very late in the shutdown sequence, the drive will hang do to the fact that the kernel completion mechanism and jiffies counter is shutdown.

I worked around this by switching to a polling mechanism of the i2c registers after the driver shutdown call is made. This seems to work even very late in the shutdown sequence.

  1. The tout register is computed incorrectly. Since it's integer arithmetic, it always computes to 0 which means no timeout.
pelwell commented 8 years ago

As Eric will tell you, getting changes accepted and tested (sometimes in that order) downstream is a lot quicker than upstream. If you put together a Pull Request, explaining carefully what it changes and why (and ideally explaining how it won't break anything), then it will almost certainly be accepted.

As a matter of interest, does the upstream I2C driver (i2c-bcm2835) have the same faults?

ghost commented 8 years ago

Yes, the upstream has the same issues.

pelwell commented 8 years ago

OK - thanks.

Our goal is to use as much upstream code as possible, but we don't want to lose functionality in the process. Getting your changes tested widely downstream will strengthen the case when it comes to upstreaming them.

ghost commented 8 years ago

Ok, cool.

I'm going to close the issue until I get a pull request together. I also want to do more testing on my end.

bradanlane commented 8 years ago

(I realize this is issue closed and primarily looking for an opinion)

Could this issue result in various boot issues with an RPi3 - specifically, random crashes during boot or ethernet odd behavior?

I have been struggling to nail down a problem when I have I2C enabled. If I have the board (an adafruit PWM controller) connected via I2C, and boot, I successfully boot about 30% of the time. With the board not connected, the RPi3 boots perfectly 100% of the time. I've tried with several permutations. If I plug the board in after boot, I can reliably use the system.

The problem is more or less reproducible with Ubuntu Xenial 16.0.4 for the RPI3 but I have not been able to product the issue with Debian Jessie.

pelwell commented 8 years ago

What does vcgencmd get_throttled return, with and without the PWM controller attached? I'm suspicious that you might be hitting the limits of your power supply.

bradanlane commented 8 years ago

The results of vcgencmd get_throttled are:

with PWM controller: 0x50000 without PWM controller: 0x0

pelwell commented 8 years ago

0x50000 means that a throttling of the CPU clocks occurred, and that it was due to undervoltage. If it was very bad you would see 0x50005, meaning that the throttling was ongoing.

The Pi3 has a high power demand at startup as multiple subsystems are initialised in parallel; it looks like your external board is tipping it over the edge. Make sure you have at least a 2.5A power supply, but with external devices drawing power you may need more.

bradanlane commented 8 years ago

One Note: the PWM board is powered off the 3.3v pin of the GPIO.

The current draw is very little - less that 0.5A throughout the boot process and login. In testing other projects, I agree the voltage drop is to be avoided.

I have some in-line voltage/current "doctors". These "doctors" incur a penalty (aprox 0.05v) which I can use to test low voltage scenarios just by in-lining multiple testers.

I was able to drop the voltage enough (low point during boot of 4.65v) to get 0x50005 without the PWM board attached. It did not crash over 5 different tests. With the PWM board attached and a single monitor on the power, the low point was 4.82v

So, while data suggest it is not a power issue, you may have identified a weak link. My primary power source is nearly exactly 5v I also have sources that are 5.15v and 5.25v I will keep testing with different power sources.

ghost commented 8 years ago

@bradalane: with PWM the voltage drop can be very erratic depending on your PWM frequency. If you measuring with a standard multimeter you will only see the AVERAGE voltage. You need something like an oscilloscope to measure the low and high points.

bradanlane commented 8 years ago

Good point of the duration of power fluctuation. my cheap monitors are not very fast.

I amended my prior post to note the PWM board uses 3.3V for its control. It has its own source input for driving the PWM "devices" - eg LED strips, servers. etc.

ghost commented 8 years ago

BTW, I've personally found the biggest issue of voltage drop to be cable length. Standard USB cables can have a very high resistance sometimes as high as 1Ohm. If you get a transient current of 1A for only a few microseconds you can end up with a voltage as low as 4V on the RPI end.

bradanlane commented 8 years ago

Agreed on USB cables. I have thrown out the 24 and 28 awg cables. I use short cables of 22awg or even 20awg when possible.

ghost commented 8 years ago

Cool.

The issue is somewhat different then what I was seeing through. In my case, I was seeing random bus errors because of the CPU and other clocks switching frequency. The clock divider of the I2C was not being recalculated on each clock switch.

ghost commented 8 years ago

BTW, how long are your 22 and 20awg length cables. because as you know, the resistance goes up with cable length.

bradanlane commented 8 years ago

The two USB cables I have been working with today are 1m (3.3ft). My really short ones are "lower quality".

ghost commented 8 years ago

Also how is the 3.3V for the devices created. Is it off the 5V rail that is used for the RPI, or is it say from a completely different power supply.

Because if it's derived from the 5V rail, you can still get ripples in the RPI line.

bradanlane commented 8 years ago

I am currently using GPIO pin 1 for the 3.3v.

Given this discussion, I will try to find an alternate source for the 3.3v and test.

ghost commented 8 years ago

I think that's best to try an additional source to make sure ripples aren't an issue.

Another thing to try is to put a big capacitor across the 5V rail. You can easily go as big aa 440uF. That will remove some of the ripples. But of course you need a scope to really see what is happening.

bradanlane commented 8 years ago

I appreciate the quick replies and diagnostics. I'll work through the voltage related suggestions and see if things stabilize.

Given my other projects, I suspect this is indeed the issue.

ghost commented 8 years ago

Cool, let me know if you still see issues later.

bradanlane commented 8 years ago

an FYI for posterity ...

I tried several different power sources for both the Raspberry Pi and the PWM board. While I could make the system run reliably, I would nearly always see the red LED blink frequently.

As reported before, The results of vcgencmd get_throttled was often: 0x50000 and occasionally 0x50005

The only reliable solution has been to underclock to 900MHZ. I did not try other values as this has been very stable.

I do not known the complete problem. the PWM board draws very little current (probably less that 15mA) as the board has its own power input for driving the PWM circuit and servos.

Anyway, I have a stable system and the loss of performance is not significant to my needs.

Thanks to @Electron752 this thread for providing the clues to address my issue.