i2c not working?

[MrRSquared]

I am having some difficulty implementing the i2c protocol. I have attempted to use three different sensors, and the libraries for each of them fail. They each work with an Arduino so I know the sensors are working.

The sensor that should be the most compatible with the VMX PI is the Rev Color Sensor so that is the one I will focus on.

If I start from a fresh FRC Installation using the image on your website; I can log into the Raspberry Pi (with the Pi user), and run i2cdetect -y 1 from the terminal, I get the expected readout instantly. With the color sensor showing up at 0x52 (and the other sensors showing appropriate addresses).

If I try to update the fresh install to run WPilib according to the documentation, the port scan continues to work.

However, if I push robot code (using wpilib 2019 or 2020 libraries) even if it is not using the dataio ports...

I have two issues.
1) If that code includes the color sensor (2020 only), that code cannot find the color sensor. I get the following error.

********** Robot program starting **********
Could not find REV color sensor
Error at com.revrobotics.ColorSensorV3.checkDeviceID(ColorSensorV3.java:400): Could not find REV color sensor

2) If I do another i2c scan, it detects nothing and takes ~5 minutes to finish.

This is true even if I run frcKillRobot.sh Even if I run it and then reboot, the pi cannot see any i2c device unless I burn a new image.

Full disclosure, we have not used i2c on the Rio much (we do have this exact color sensor working on our current robot with Python).

The other sensors are a RevDistance sensor and a TOF 10120(which has no library for FRC, but I tried using the getaddress method).

I am using a raspberry pi 4b.

It is entirely possible I am doing something wrong, but I think there may be something else as well.

[kauailabs]

Thanks for reporting this - it’s apparently a software problem and it’s not clear yet what the root cause is - though you’ve provided some clues. My intuition tells me it may take a little while to solve this puzzle, and some change in the Linux kernel may have triggered it. We will update this issue once we learn more, but heads up that this is a challenging - and important - issue.

[MrRSquared]

Thank you for looking into it. I understand this is an odd and potentially elusive one. I would be happy to perform any other tests you would like to help root out the cause and/or test potential solutions.

[kauailabs]

After a bit of debugging, the root cause of this issue has been found.

For the VMX-pi I2C interface, some recent code was add in the VMX-pi platform library to configure the I2C pins as "Input" upon initialization; while that works when using these "CommDIO" pins (which are Raspberry Pi GPIO pins 2 and 3) as GPIOs, when these pins are later allocated to the I2C function, the GPIOs are not being configured as outputs in order to function correctly w/the Raspberry Pi 4 I2C circuitry.

Additionally, the original state of these Raspberry GPIO pins are not being saved during initialization, so that they can be restored upon exit. This should be resolved in order to ensure that if i2cdetect works before running an application that uses the VMX-pi platform library, that it will still work once the application is exited.

The actual work to resolve this issue is not yet completed, and will require some rework to the VMXIO module within the VMX-pi platform library to save initial GPIO state on it, to restore it on exit, and to set the correct state when configuring the GPIOs to the I2C function.

[MrRSquared]

Thank you. I guess it is good to know that it was not just me.

[kauailabs]

The fix for this is now available in VMX-pi platform library version 1.1.224.

Note also that coincident with this change, the GradleRIO plugin configuration has been simplified. Rather than needing to update the custom GradleRIO plugin from Github, build it and deploy it, the kauailabs GradleRIO plugin can be updated now simply by changing the "plugin" specifier in the build.gradle file.

So to get this update, and to ensure you're in synch with the latest WPI Library software:

• On your raspberry pi, update your vmxpi-hal platform library to v. 1.1.224:

sudo apt-get update sudo apt-get remove vmxpi-hal sudo apt-get install vmxpi-hal

Verify that the last step indicates v. 1.1.224 is installed.

• On your development PC, update your build.gradle file as described in the updated instructions on this page:

https://pdocs.kauailabs.com/vmx-pi/advanced/vmx-pi-for-frc-2020-robot-programming/

[MrRSquared]

Hello, I am not sure if you will all get this, but I am still having issues with I2C. Here is what I did. I tried running through the steps above, and after the update it did not recognize my device. Then, I tried running the Rev Color Sensor sample program. It did not find the sensor.
I ran i2cdetect and it did not find anything. However, this time it did not take forever to run through the scan. I tried to uninstall Hal and the frc libraries, and still i2c did not detect anything.

So, I downloaded a new image from the Kauai cloud site (the 2020_1_0.zip) I tried to find the device (I used a few) with i2cdetect and it found it (all of them actually). Then, I updated everything, and they were still found. I installed the application. It had the following error.

(VMXIO.cpp [1259])
*******VMX HAL SPIClient Performance*******
- Writes:
Total #: 27
Avg Us: 249
Min Us: 55
Max Us: 455
# long: 0
# lng2: 0
# lng3: 0
MinWait: 33
MaxWait: 77
Write CRC Errors: 1
Write Retries: 0
Write Failures: 0
- Reads:
Total #: 294553
Avg Us: 199
Min Us: 49
Max Us: 532
# long: 0
# lng2: 0
# lng3: 0
MinWait: 12
MaxWait: 23
Read CRC Errors: 1
Read Retries: 1
Read Failures: 1
*******************************************
2020-05-13 16:49:42 spiClose: pigpio uninitialised, call gpioInitialise()
VMX HAL: Error closing SPI AUX Channel 2.
Entering VMX HAL Internal Kill Handler.
Entering VMX HAL Internal Kill Handler.
MAU HAL: Error expiring IO Watchdog.
VMX HAL: pigpio library version 69 opened.
VMX HAL: SPI Aux Channel 2 opened with baudrate of 4000000.
VMX HAL: Established communication with VMX-pi model 0x32, hardware rev 60, firmware version 3.0.411
VMX HAL: Acquired navX-Sensor configuration.
VMX HAL: Library version 1.1.224
Set VMX CAN Mode to NORMAL.
Server Running...
********** Robot program starting **********
Could not find REV color sensor
   from: com.revrobotics.ColorSensorV3.checkDeviceID(ColorSensorV3.java:400)

Error at com.revrobotics.ColorSensorV3.checkDeviceID(ColorSensorV3.java:400): Could not find REV color sensor
Default robotInit() method... Override me!
Default simulationInit() method... Override me!
Default disabledInit() method... Override me!
Default disabledPeriodic() method... Override me!
Default simulationPeriodic() method... Override me!
NT: server: client CONNECTED: 10.67.62.14 port 55895

While the program was running, I tried i2cdetect and it failed with the same slow failure as before. However, when I killed the robot process using frcKillRobot.sh The i2cdetect runs fine, it just cannot find any device. This is true even after reboot.

So, either I am doing something wrong, or there is still something awry.

[kauailabs]

Thanks for letting us know; it does sound like something is still not right; so the issue is reopened.

The initial hypothesis is that after the first time the robot app runs (which happens automatically on startup), some i2c-related resource is not being released; when the robot app runs the second time that resource can no longer be acquired; and after the robot app quits that first time, the operating system can't access the resource either (which is why i2cdetect doesn't work either at that point). Internally, there's a linux file descriptor that's being opened to access the i2c hardware; it may be that this is not being closed on robot-app exist; and the normal work the OS performs to close open file handles isn't occurring in this case (which if true is indeed surprising).

The only symptom that hypothesis doesn't address is why the first time after reboot the i2c communication with the REV color sensor doesn't work. If you have any thoughts on that, please let us know.

Overall, it appears we may be peeling an onion; with error cases occurring when multiple components interact; the implication is we'll have to expand our testing to cover more test scenarios.

[MrRSquared]

Thank you for your work with this. I think the onion is an apt and potentially unfortunate analogy. The only guess I can make involving the first instance failing is that perhaps there is either a permissions issue somewhere or a handshake issue. I do not know a lot about I2C and am not sure what the pi needs for the initial connection with the device. But, being a communication protocol, I assume there is a handshake going on somewhere along the line.

[kauailabs]

Good news, we have a breakthrough and know the root cause now.

Root cause: The "IO Watchdog" code in the VMX-pi firmware exists to disable the outputs of the various IOs on the VMX-pi board if those watchdog is not periodically "fed" (which only happens in the robot application when the robot is enabled), This functionality exists to shut down externally controlled (e.g., PWM) devices if the robot application "dies". While this is the expected behavior for GPIOs (the High Current DIOs and the FlexDIOs), when the communication IO pins (I2C, SPI, UART) are configured for non-GPIO usage, the IO Watchdog was still doing it's job - disabling the outputs (which included the I2C SDA and SCL lines) unless the robot was enabled.

So during robot initialization and in the teleop/autonomous disabled states, the I2C IOs were being disabled. When the robot was enabled, the Watchdog was fed, and the outputs would be re-enabled again.

The root cause has been validated by disabling the watchdog protection on the "CommDIOs" and verifying that the robot application is correctly accessing the I2C interface.

The symptom that "i2cdetect" would not work after running the robot program was only the case after the CommDIOs got placed into the "watchdog disabled" state; the reason no response from the device was received during this time was that the device had received the initial "read request" and was in the state of trying to respond but the bus lines (esp the SCL clock line) were not being manipulated and this the device was "stuck" transmitting the data that had been requested.

The resolution for this issue will be to update the VMX-pi platform library to not allow the watchdog to be enabled for the COMMDIO Outputs. This is not exactly the ideal solution, since it might mean that if (for instance) I2C is used for digital communications, but SPI or UART are used for outputting PWM to a motor, that the PWM outputs would not be auto-disabled on timeout. While the chances of that happening are slim, it's possible. Since we have a pending release, we'll create a separate issue to handle this edge case.

[kauailabs]

This issue is now resolved. As indicated earlier, the root cause was the IO Watchdog incorrectly disabling CommDIO Channels that were configured for Digital Communication (I2C, SPI, UART).

This is resolved in the VMX-pi Platform Library v. 1.1.225.

To retrieve this library, update the GradleRIO plugin version to "2020.3.2-beta-3".

You can verify that the 1.1.225 library is used by viewing the console log available at the DriverStation when the robot application starts.

[MrRSquared]

Thank you. That totally worked. I updated the Gradle (and it downloaded and built on its own when I just did build connected to the Internet). I ran the color sensor test code and it worked. Proximity, color and IR are all working. I concur that this issue seems fixed. Thank you again. This is excellent.