Distributed hardware unit test infrastructure proposal

[nebrius]

A Proposal for a distributed on-hardware unit test infrastructure

First, a big thank you to @noopkat for letting me bounce ideas off of her and for encouraging me to create this issue. This idea has been bouncing around in my head for a while, but I wasn't doing anything more than that with it until now.

Motivation

Much of the NodeBots ecosystem can be unit tested using traditional shimming techniques, but there are some things that can only be tested on hardware.

As an example, consider raspi-pwm, a part of raspi-io, an IO plugin for Johnny-Five that adds Raspberry Pi support. The Raspi PWM module exposes the PWM hardware to Node.js. Part of the initialization mechanism is to set clocking values in the PWM registers: https://github.com/nebrius/raspi-pwm/blob/master/src/init.cc#L38-L39.

This code cannot be tested from software. When I first implemented this code, I used an oscilloscope to verify that the values were correct. This process is completely manual and cannot be realistically automated (oscilloscopes are expensive).

A new version of the Raspberry Pi was released recently that increases the clock speed of the main processor. I realized very recently that this may mean that the PWM frequency being set is not correct for the Raspberry Pi 2, and that servo support is broken (I'm in Berlin right now, and don't have access to my hardware to verify at the moment).

If we had some sort of hardware unit testing mechanism, this would have been caught or verified to work as soon as the Raspberry Pi 2 was released, not several months later.

Methodology

There are a number of different pieces in play here, as seen in the diagram above. There are four different devices, from a software perspective, and each device has one or more software components running on them.

CI Server

The CI server is basically Travis (or Jenkins, or whatever). There are two key pieces of software running here: the CI Translation Layer, and the Unit Test Bridge.

The Unit Test Bridge takes in a test specification (url of a repo w/ tests?) and sends it to the Device Router. Once testing is complete (or times out), the bridge signals back to the caller with the results of all tests.

The CI Translation Layer converts between anything that may be specific to the CI system being used into a generic format that the Unit Test Bridge expects. This layer may or may not be necessary.

Devices Router

The Devices Router is a router that is connected to all of the Devices Under Test. The router and all of the devices under test are expected to be on the same LAN, although this restriction may not be necessary.

The Devices Router takes in a test specification from the CI server and routes it to all of the appropriate devices registered with the router. Different devices can be used to specify Devices Under Test with, say, different versions of hardware and/or different versions of Node.

Device Under Test

The Device Under Test (DUT) will be or two devices. If the board being tested is self-hosted (i.e. runs a full-blown operating system on device), then this will be one device, but if firmata is used, then there will be two devices in play.

The DUT is composted of two key pieces of software: the Test Server and the Unit Test Runner.

The test server is responsible for:

• listening for requests to run a test
• Initializing the test infrastructure (cloning repos, npm installing things, etc)
• Acting as the communication bridge between the Device Under Test Evaluator and the actual unit tests

The Unit Test Runner is a unit test runner (probably just Nodeunit with some extra interfacing work, custom reporter?). The Unit Test Runner is responsible for starting up the unit tests, collecting the resuts, and sending them back to the test server to be relayed onward. This may be split into multiple modules.

Device Under Test Evaluator

The Device Under Test Evaluator is a second microcontroller with custom software/firmware that is responsible for measuring the output of the DUT, or sending signals to the DUT. This is where the output of the DUT is measured.

This would probably be a custom version of firmata with a few extensions to, for example, measure PWM duty cycle. The DUT will be communicating over this serial protocol back to the DUT test server.

[ajfisher]

I like where this is headed and I think we can get some efficiency in running tests for those that can supply a batch of hardware that we link through a switching board.

Eg say I have a setup that we can use as a node in the system and I declare I have an RPi, RPi2, Arduino and BeagleBone (the 4 boards closest to hand). If we built a board that had a bunch of standard test hardware attached (eg servo, I2C device etc) and then provided the mechanism for switching between connected boards to provide the routing then you could run a battery of tests against fixed hardware very quickly and efficiently.

[nebrius]

My thoughts exactly. I have an Arduino Mega at home I was thinking about using as the initial DUT Evaluator since it has lots of ports, including multiple UARTs. I was kinda thinking that orchestration between DUT and DUT evaluator would be, e.g. for testing digital outputs would be:

• DUT unit test sends "set pin x high" to the IO plugin (or whatever is being tested)
• DUT unit test requests that the DUT server send a "read pin y value" command to the DUT evaluator, where pin y is the pin on the DUT evaluator connected to pin x on the DUT
• DUT server sends firmata command (using J5?) to the DUT evaluator to get the value of pin y
• DUT evaluator sends value back to DUT server
• DUT server sends value to DUT unit test
• DUT unit test does nodeunit-y expect-y things

I was also thinking about adding a little circuitry in the middle (basically a bunch of OR gates and level shifters), so that we could test all the GPIOs on the DUT while only needing to use a few pins on the DUT evaluator (one for the output of the ORs, the others to do the channel selection). If we wanted to get fancy, we could use an FPGA in the middle and really crank up the possibilities, such as using one DUT evaluator for multiple DUTs.

Either way, the idea is we can do hardware unit testing w/o having to create hardware that's specific to the tests being run.

[nebrius]

Oh, and the reason I was thinking custom firmata was so that we could add new functionality that measures PWM signals (setting rising and falling edge interrupts and measuring the time between the two). We could also have the DUT evaluator act as an I2C peripheral.

This way, the only thing that needs to be connected is the DUT evaluator and any interface circuitry needed (level shifters, etc).

[ajfisher]

That's a really good, detailed outline of how you thought it could work and made me think of another option.

What about using a software based logic analyser (eg salae) which has probes on the DUT and then it can act as the evaluator. Basically what you're doing then is this

Test runner sends required test to DUT, eg pwm pin 3 at x freq.

Logic analyser then records the data and emits a stream over a serial port.

We write a stream analyser and interpret results.

Salae logic analyser is about $10 with 8 ports... And you can get bigger ones.

This would mean we remove specific component requirement and simply test the boards, IO plugin layer and data that is pumping out.

Would that do what you're attempting? It's like a poor but automatable oscilloscope.

On Thu, Oct 1, 2015, 11:41 Bryan Hughes notifications@github.com wrote:

Oh, and the reason I was thinking custom firmata was so that we could add new functionality that measures PWM signals (setting rising and falling edge interrupts and measuring the time between the two). We could also have the DUT evaluator act as an I2C peripheral.

This way, the only thing that needs to be connected is the DUT evaluator and any interface circuitry needed (level shifters, etc).

— Reply to this email directly or view it on GitHub https://github.com/nodejs/hardware/issues/17#issuecomment-144591782.

[nebrius]

That's a great idea! less software to write is always less software to write. We could probably get more accurate timing info our of it too. We will still need something so that we can set pins high and low (for reading on the DUT), but this makes pretty much all the other tests a lot easier

[ajfisher]

Definitely. So I have one of these analysers mostly just sitting around for the occasional I2C debugging I end up doing. I'll take a look over the weekend (as no doubt I'll need it for some of the Interchange stuff I'm working on anyway) and see what sort of interfaces we can get etc and quantify work to write the analysis side of this.

On Thu, Oct 1, 2015, 12:15 Bryan Hughes notifications@github.com wrote:

That's a great idea! less software to write is always less software to write. We could probably get more accurate timing info our of it too. We will still need something so that we can set pins high and low (for reading on the DUT), but this makes pretty much all the other tests a lot easier

— Reply to this email directly or view it on GitHub https://github.com/nodejs/hardware/issues/17#issuecomment-144596017.

[nebrius]

nice, thanks!

[noopkat]

Awesome, Bryan! I was worried I may have been a bit too enthusiastic when we chatted haha.

Thank you for the diagram, it's a huge help to understand the idea.

WRT the Salae logic analyser - Eli has one, I thought they started at $100, with 8 ports being $200? Maybe I'm looking at the wrong product. If I'm not barking up the wrong tree, a recording from those things can be gigs of data for just a few seconds. They are brilliant little things though! Top idea to use one, @ajfisher :+1:

If this works well with the RPi's, it'd be great to roll it out for other hardware too :tada:

[ajfisher]

Well "official" ones are quite pricey, however they are easily found that do the same protocol from about $20 though resolution probably won't be quite as good.

On Thu, Oct 1, 2015, 14:11 Suz Hinton notifications@github.com wrote:

Awesome, Bryan! I was worried I may have been a bit too enthusiastic when we chatted haha.

Thank you for the diagram, it's a huge help to understand the idea.

WRT the Salae logic analyser - Eli has one, I thought they started at $100, with 8 ports being $200? Maybe I'm looking at the wrong product. If I'm not barking up the wrong tree, a recording from those things can be gigs of data for just a few seconds. They are brilliant little things though! Top idea to use one, @ajfisher https://github.com/ajfisher [image: :+1:]

If this works well with the RPi's, it'd be great to roll it out for other hardware too [image: :tada:]

— Reply to this email directly or view it on GitHub https://github.com/nodejs/hardware/issues/17#issuecomment-144614765.

[noopkat]

@ajfisher wooooah nice! I had no idea you could come by cheaper versions :+1:

[reconbot]

This is great! :grinning:

Have you given thought to a way of encapsulating the software and tests through the bridge and device router? A way to package up some code and say run it against this matrix. The package being what runs on the test server(?)

We also have a few things around a test package. Some devices run node and have ssh access, some don't and need flashing. Also some test would need to be run locally and others would need the sensor rig ("Device Under Test Evaluator"). This means output from both devices.

And while I'm listing things I don't have answers to. In a server environment an ILO is amazing. (integrated lights out management, these things have many names.) It gives you the ability to remote power cycle, see serial out, and get a console on each server. You couldn't debug colo'd machines without it. Our version would probably want to control power, monitor status lights and serial outputs.

[nebrius]

Have you given thought to a way of encapsulating the software and tests through the bridge and device router? A way to package up some code and say run it against this matrix. The package being what runs on the test server(?)

I've thought about it, sure... ;-)

In my head, I can foresee either packing up everything and serializing it over to the other devices (zipping up a directory, eval'ing code, etc), or passing a git url over that then get's cloned along with a serialized (BASH?) script that gets run afterwards to perform initialization, or even just running jenkins on each node so we can just pass a jenkins config over...I dunno, pros and cons to each. I'd love to hear everyone's thoughts.

We also have a few things around a test package. Some devices run node and have ssh access, some don't and need flashing. Also some test would need to be run locally and others would need the sensor rig ("Device Under Test Evaluator"). This means output from both devices.

Correct me if I'm wrong, but I'm pretty certain we can split everything into "firmware" and "software", with probably 90% of what we're doing being software. The difference being that firmware has to be flashed to the device using, e.g. the Arduino CLI tools, whereas software runs in node (either on device or on a connected computer). I haven't really put any thought into how we test firmware yet, so I'm all ears for suggestions!

And while I'm listing things I don't have answers to. In a server environment an ILO is amazing. (integrated lights out management, these things have many names.) It gives you the ability to remote power cycle, see serial out, and get a console on each server. You couldn't debug colo'd machines without it. Our version would probably want to control power, monitor status lights and serial outputs.

That might even be necessary for some of the tests, e.g. the Raspberry Pi where once you set the pin mode on an I2C pin to something other than I2C, it can't be changed back to I2C mode until the device is rebooted. No idea on this one either, but definitely something to think about. Good suggestions!

[noopkat]

Good thoughts! My gut feel is that some of this won't be immediately obvious until a POC is built. However - would it help if those interested contributed their own set of requirements so we know what would ideally be supported on this system? Hardware + firmware etc, what needs to be verified exactly etc.

[johnnyman727]

Great proposal, @nebrius!

Your architecture is very similar to the hardware integration testing rig we had at Tessel with the exception of the secondary device to actually check logic levels. We had this job server (essentially your CI server) present a webpage where you could run any branch of Tessel firmware on Github through the test suite. One computer (basically your Test Controller) was connected to four Tessels (Device Under Test Controller) each of which had 4 or 5 Tessel modules connected to them. That computer ran unit tests through each Tessel depending on which modules were connected to it.

I think the addition of Device Under Test Verifier is a great idea and probably would have caught some lurking issues we had much earlier. :+1:

We've also used Saleae knock offs to do some programmatic testing and validation of different components on the board (specifically to debug our CC3000 wifi chip).

[reconbot]

Where's a good place to find these Saleae knockoffs?

[ajfisher]

Try Ali - or ping me and I'll link you to where I got one.

On Mon, Oct 5, 2015 at 9:10 AM Francis Gulotta notifications@github.com wrote:

Where's a good place to find these Saleae knockoffs?

— Reply to this email directly or view it on GitHub https://github.com/nodejs/hardware/issues/17#issuecomment-145392581.

[nebrius]

@johnnyman727 That's really cool! It's good to know the idea isn't totally off the wall :)

By any chance, is any of that software open source, or able to be open sourced?

[johnnyman727]

@nebrius the UI to the test bed (the one I linked to) is open source. I can't seem to find the code that actually ran on the testing machine :/

[nebrius]

I'm working on a Proof of Concept over at https://github.com/nebrius/hwt-dut

[nebrius]

I chatted with the CEO of a startup called krtkl (pronounced critical) last night that's working on a new board called the Snickerdoodle. This board has a lot of potential for the DUT Evaluator IMO.

The processor is a Xilinx part that combines a dual core ARM Cortex A9 with an FPGA in a single package plus WIFI and a few other things on board. This could be huge for us because it gives us either pins that can be whatever we want them to be, GPIO, I2C, SPI, UART, or something custom. The custom bit is especially interesting for verifying PWM, as the timing will be much more accurate when done in hardware(ish), not software. This also means we can run an OS on the ARM cores w/o having to worry about it not being real time.

This board is also interesting because we get either 150 or 179 of these pins, which means we should be able to use a single of these boards to test 3-5 other devices, instead of one DUT Evaluator per DUT, saving us some money and simplifying the architecture (the router can run on the ARM core too, probably).

These boards are going for $55.

The only downside: they're not out yet :(. This may not be a big deal, since there's a lot of other work to do as well. They're currently being crowd funded over at https://www.crowdsupply.com/krtkl/snickerdoodle