Issues with firmware v0.0.13

[dudleyjosh]

@johnnyman727 @rwaldron I was looking into this issue some more and I discovered that I now have issues using the accelerometer module with firmware v0.0.13 too. It seems to lock up after a random period of time with v0.0.13 but not with v0.0.12, including one time where it was returning accelerometer data but only the accelerometer data from the position it was in when it started up (same xyz data was emitted repeatedly). Could it be something related to the I2C changes that were part of firmware v0.0.13?

I notice that the data is being emitted at a much faster rate with firmware v0.0.13 too (data is emitted ~15 ms vs ~80 ms)?

I was testing both firmware versions using by running the same example code t2 run accel.js over and over again.

var tessel = require('tessel');
var accel = require('accel-mma84').use(tessel.port['A']);

// Initialize the accelerometer.
accel.on('ready', function () {
// Stream accelerometer data
    accel.on('data', function (xyz) {
        console.log(
            'x:', xyz[0].toFixed(2),
            'y:', xyz[1].toFixed(2),
            'z:', xyz[2].toFixed(2)
        );
    });

});

accel.on('error', function(err){
    console.log('Error:', err);
});

[johnnyman727]

@dudleyjosh I am able to reproduce on 0.0.13. It doesn't appear to happen everytime for me but maybe every fifth time, output stops after ~3 seconds. I had a logic analyzer on the I2C lines and the interrupt lines and can confirm that the accelerometer is pulling the interrupt line (bottom trace in image below) low to indicate it has a new reading ready but the Tessel does not respond to the interrupt by reading the data register:

Also, strangely, the I2C lines are brought low which indicates to me that there might have been some sort of crash involved. The I2C lines should be held high unless communication has commenced.

At this point, my guesses as to the cause are:

1. The low interrupt was somehow not configured again after the previous interrupt (either in the tessel-export.js driver or on the SAMD21 firmware).
2. The interrupt was configured but was missed on the SAMD21. This is pretty unlikely.
3. The interrupt notification never made it to the tessel-export.js driver from the SAMD21 for whatever reason
4. The interrupt notification never made it into the accelerometer driver.
5. As mentioned above, something crashed the I2C bus

Update: The issue does seem to be something hanging with I2C. I added console.logs to the beginning and end of the of accelerometer data reading and it looks like it eventually gets to the point where it reads the data over I2C but does not hit the callback.

Update: I can get this issue to reproduce on 0.0.12 as well so I think it's something that's been lurking for a while.

[dudleyjosh]

@johnnyman727 any more on this? I'm kind of curious what the actual root issue is plus I have a machine that depends on the accel-mma84 module that continues to have intermittent issues, even with 0.0.12 (it is unusable with 0.0.13).

Also, process.on('uncaughtException', function(err){}) doesn't catch anything... I've been trying to get to the bottom of this intermittent issue for awhile but I haven't had any luck and I am thinking it has to be related to this.

[johnnyman727]

@dudleyjosh I hadn't had a chance to investigate since my comments two weeks ago and I'm not sure when I will because I'm traveling. The root cause is almost definitely in the SAMD21 firmware and I'm pretty sure something is crashing the I2C bus (as described above). I believe it occurs somewhere between where the SAMD21 gets the command to start an I2C transmission (which should read the data register) and starts sending the I2C data.

I'm honestly at a loss for how to advise on investigation. @kevinmehall may be able to help advise if he has time.

[rwaldron]

This is basically the same issue I observed and DM'ed to @johnnyman727 and @tcr, except in my case it was analog read. I believe the issue may be related to overwhelming stdout/stderr, but that's just "spit balling"

[johnnyman727]

@rwaldron a good way to test that spitball would be to try running the script over both USB and LAN. It's more likely to fail over USB and if it only fails over that connection then it is very likely the data rate that's choking the USB communication daemon I hand rolled.

[rwaldron]

Presently using the following rig to test: https://www.youtube.com/watch?v=1HYuTg2Puxo

Test:

1. Start external servo rig
   1. Note: This was never stopped during the testing.

2. t2 run accelerometer-test.js

   var tessel = require("tessel");
   var accel = require("accel-mma84").use(tessel.port.A);
   
   accel.on("ready", function() {
    accel.on("data", function(xyz) {
      console.log(Date.now());
      console.log(
        "x:", xyz[0].toFixed(2),
        "y:", xyz[1].toFixed(2),
        "z:", xyz[2].toFixed(2)
      );
    });
   });
   
   accel.on("error", function(error) {
    console.log("Error:", error);
   });

3. Once INFO Running accelerometer-test.js... is displayed, start external timer (preset to 5 minutes)\
4. Record first timestamp (Ftms)
5. Observe possible outcomes:
   1. Program runs for entire 5 minutes
   2. Program fails before 5 minutes
      1. Record last timestamp (Ltms)
      2. Calculate exact uptime: (Ltms - Ftms) / 1000

Observations:

Connection Type	Run	Result	Up Time (mm:ss.ms)
LAN	1	Complete	> 05:00.00
LAN	2	Complete	> 05:00.00
LAN	3	Complete	> 05:00.00
LAN	4	Complete	> 05:00.00
LAN	5	Complete	> 05:00.00
USB	1	Incomplete	00:07.717
USB	2	Incomplete	00:00.078
USB	3	Incomplete	00:02.856
USB	4	Incomplete	00:01.207
USB	5	Incomplete	00:16.367

[dudleyjosh]

@rwaldron I wish I could setup and run some tests myself right now but I will have to wait on my next batch of boards to arrive... I just shipped my last 3 to the UK for some stuff I did at work (more of the talking speakers).

I can tell you that I have definitely experienced (and continue to experience) the failure via LAN too on the board I have setup to run a little machine here at work... but I haven't been able to successfully capture the error via code (I tried using the process.on('uncaughtException', function(){}) to catch it, but no luck).