Microsoft "sponsored" hardware test machine for node-serialport

[rwaldron]

We'd like to get some help from MS on this, wherein we'd like:

• An actual dedicated machine
• Runs real tests that do real things to really connected usb devices
• Runs these tests for every node, iojs, node-serialport and windows release

I started by talking to @reybango, who directed me to @mattpodwysocki who can fill us in on progress so far.

[mikeal]

Pinging @nodejs/build since they will end up integrating this hardware once donated.

[rwaldron]

Maybe there is a misunderstanding, or maybe I'm misunderstanding something? I had assumed someone at Microsoft is going to maintain this?

[mikeal]

If it is integrated in to the build process then the build team will be maintaining it in some way.

[rwaldron]

That's not what we were asking for, and not what's being given to us. We asked for a dedicated resource at Microsoft to take responsibility for maintaining this, because:

1. Windows is the weakest link
2. It's an actual, physical machine with things plugged into it.

[mikeal]

Runs these tests for every node, iojs, node-serialport and windows release

That requires integration in to the iojs/node build system, regardless of who maintains the actual hardware.

[voodootikigod]

nope. disagree. It uses the output of those build system, not the processes of them. This is not for the long vision of 'deploying always built images of node-serialport' but for testing node-serial port versions against windows and windows devices.

Chris Williams

@voodootikigod http://twitter.com/voodootikigod | GitHub http://github.com/voodootikigod

The things I make that you should check out: SaferAging http://www.saferaging.com/ | JSConf http://jsconf.com/ | RobotsConf http://robotsconf.com/ | RobotsWeekly http://robotsweekly.com/

Help me end the negativity on the internet, share this http://jsconf.eu/2011/an_end_to_negativity.html.

On Wed, Jun 3, 2015 at 4:13 PM, Mikeal Rogers notifications@github.com wrote:

Runs these tests for every node, iojs, node-serialport and windows release

That requires integration in to the iojs/node build system, regardless of who maintains the actual hardware.

— Reply to this email directly or view it on GitHub https://github.com/nodejs/hardware/issues/13#issuecomment-108601112.

[mikeal]

Ahhhhh, I misunderstood, I thought we would add these test runs to the regular build process so we can validate that we don't break hardware use cases in new versions of node.

[orangemocha]

What specific devices would this machine need to be connected to? Can it be run in a VM that has access to those devices on the host?

[orangemocha]

/cc @munyirik

[rwaldron]

@orangemocha are you working with @mattpodwysocki on this?

[orangemocha]

Now I am :)

[rwaldron]

@orangemocha ideally we'd like some physical device attached to the machine; that could be an Arduino that just echos back whatever comes over the wire. The primary environment shouldn't be a VM, because that's not the most common user case.

[nebrius]

Related to this conversation: https://github.com/nodejs/build/issues/151. It's probably too early to try and collaborate, but I could see these two initiatives dovetailing nicely in the future.

[orangemocha]

The primary environment shouldn't be a VM, because that's not the most common user case.

@rwaldron : testing this on multiple versions of Windows, node, and node-serialport would be a lot more manageable using VMs. Can't we assume that if it works in a VM, then it would work on physical hardware?

[rwaldron]

The point is to test with actual serial devices plugged into an actual USB port, if the VMs can do that then fine. Incidentally we only care about one version of node-serialport: latest. We only care about ensuring that, when a Windows user installs node-serialport and writes programs that depend on it, it "just works".

[munyirik]

Hi @rwaldron, Are there tests that already exist for physical devices? I was looking at the tests in https://github.com/voodootikigod/node-serialport/blob/master/test and see the use of MockedSerialPort (I'm assuming that's a virtual serial port..haven't taken a close look). Should we modify these tests to optionally use physical hardware when specified?

[reconbot]

It's a mock for the javascript bindings that communicate with the C layer that talks to the OS to use the serialport. It's serialportish enough, but not an actual virtual serial port. So most of our tests aren't covering talking to the OS.

[rwaldron]

Are there tests that already exist for physical devices?

Yes https://github.com/voodootikigod/node-serialport/tree/master/arduinoTest

[joshperry]

I'm not entirely convinced that a library like node-serialport needs a physical test target. I think the thing that is most important when testing node-serialport, is that it is properly interacting with the host OS serial APIs. This is to ensure it calls the proper APIs in the proper order to configure and interact with the serial port in a way the user expects.

Since the interface to the actual physical hardware is managed completely by the host OS a physical test target would merely be an integration test that validates the OS's UART code hasn't broken. This seems to be outside the scope of our responsibility; I would hope that Microsoft, et. al. have procedures to test this part of their stack before each OS release.

I know coding for serial ports on Windows is a little wonky (I maintain a stream-based fork of node-serialport), but I don't think it is nondeterministic. IMO, these tests could be easily conducted using a software null-modem solution like com0com.

[munyirik]

To kick this effort off I've set up a Windows 10 (physical) machine connected to an Arduino Leonardo. I got help from @orangemocha to create a job in Jenkins (https://ci.nodejs.org/job/serialport-test-commit-windows/) to run tests in 'test' and 'arduinoTest' directories. @rwaldron can you access it?

[munyirik]

Ping. As mentioned earlier, I set up one machine that is ready to start running tests. In future I can expand to test more versions of windows, add more devices, etc. I'll also be maintaining the hardware in case there are any issues. Please let me know what the next steps are to hook this up to the serialport CI.

[joaocgreis]

The machine is now connected to CI and the job is run every day: https://ci.nodejs.org/view/All/job/serialport-daily-master/

[reconbot]

Hi @joaocgreis thanks for pushing ahead on this! I've responded about node-serialport on the issue you opened there. https://github.com/EmergingTechnologyAdvisors/node-serialport/issues/879 Appveyor hasn't been great but it's been something where we had nothing and a real machine with an arduino we're allowed to modify would be wonderful.

I need more info in the meantime.

Thanks!