Tooling: Run full test suite with loopback serial device

[cinderblock]

💥 Proposal

The CI testing should try to test against a "plugged in" serial port.

Motivation

Some tests require talking to some "real" hardware. As discussed briefly in #2314, it would be nice if this was handled by the CI.

This issue is intended to track the discussion about certain options before committing to a particular path or starting a PR.

Pitch

Option 1: Virtual Serial Port

There are a number of virtual serial port emulation softwares available for install. Notably:

• Virtual Serial Port Driver (Windows)
• tty0tty (Linux) - See also
• macOS?

I personally think these options would be prefered since they could run directly on GH Action Runners

Option 2: Custom Runner Server

Maintain some dedicated hardware running a self-hosted runner and link it to this GH project. Someone would need to keep that hardware powered, running, and up to date. Doing this for 3 platforms seems possibly prohibitive.

Arduino vs Wire

If running on self-hosted hardware, is there some reason to not just run a wire from Rx to Tx? Why is there even an Arduino in the loop?

What about RTS/CTS/DTR/DTS signals?

---

Work in Progress

• [ ] Windows
  • [ ] Create virtual COM port - com0com or something else?
  • [ ] Manual loopback test works
  • [ ] node-serialport tests pass
    • [ ] .open(), .write(), .read() etc.
    • [ ] .set() & .get()
    • [ ] .list()
• [ ] Linux
  • [x] Create virtual COM port - sudo socat pty,link=/dev/loopback,raw,user=runner system:cat &
  • [x] Manual loopback test works
  • [ ] node-serialport tests pass
    • [x] .open(), .write(), .read() etc.
    • [x] .set() & .get() - disabled via test-config.json socat pty does not support all ioctl commands:

      sets flags on the port:
      Error: Inappropriate ioctl for device, cannot set

    • [x] .list() - /dev/loopback is a name I made up... should we use something different?
• [ ] macOS
  • [x] Create virtual COM port - sudo socat pty,link=/tmp/loopback,raw,user=runner system:cat &
  • [x] Manual loopback test works - Working on getting access to a mac to try
  • [ ] node-serialport tests pass
    • [x] .open(), .write(), .read() etc.

      instance method
      #open
       arbitrary baud rates
         25000 baud
           opens at 25000 baud:
      Error: Error: Inappropriate ioctl for device calling ioctl(.., IOSSIOSPEED, 25000 )

    • [x] .set() & .get()
    • [x] .list()

[reconbot]

A loop back is fine, once upon a time we had a more sophisticated arduino program that would change baud rates and mess with signals.

However I've never seen a virtual serialport without quirks. I tried pretty hard (pre GitHub actions) to get a reliable test env with a virtual port and failed. I'm 💯 in favor of us trying again.

Custom runners for open source projects are an ops challenge as the runner doesn't provide any sandboxing.

[cinderblock]

I've been working on this a little for the holidays.

I've got loopback devices for Linux and macOS partially working, using socat. When I test them manually, they seem to work. However there are some problems with setting ioctl options early in the node-serialport tests that prevent any subsequent tests from working.

Windows tools for virtual serial ports are a little lacking. com0com seems like it might be what we want but it doesn't directly support a loopback device. I'm investigating if com2tcp would be a workable alternative if used similar to how socat is used (using cat to echo back).

If anyone would like to follow along, I'm working over in my ci-loopback-serialport branch, but the good stuff is in the Actions.

[reconbot]

Very cool

[cinderblock]

@reconbot I have some questions you can probably shed some light on

What generates the bytes that the #read tests use?

I see that the Arduino test code echoes "READY", which the first read test needs, but what about the others?

I'm not seeing where in the tests (yet?) you expect the loopback functionality of the Arduino code to be used. It looks to me like every time the port is .open(), the Arduino resets, and re-runs it's setup() within a second and the tests pass from that. This might be an easier thing to test than a full loopback device...

The loopback devices I'm setting up now do not echo "READY" at startup. I'm currently hacking around this by making the tests .write('READY') after every open().

One crazy idea I'm considering is a simple C/C++ (or Go? That'd be fun to learn) program that creates a loopback serialport as I'm not particularly happy with the current socat solution. This would allow adding a "READY" echo, possibly on every open(). I think doing this on Windows wouldn't be that terrible either because com0com is fully open source. Would just be stripping a bunch of unnecessary features and adding a couple loopbacks.

---

End of weekend update

Tests are almost fully passing on Linux. MacOS is giving me some trouble. Not much progress on getting Windows's loopback COM port working.

[cinderblock]

Many tests are passing on Linux and macOS. Most issues are related to sending "READY" after DTR (open), which the Arduino does, but doesn't work in CI yet.

I can get some tests to "pass" by disabling certain tests with test-config.json, but these should be investigated more.

Windows is giving more trouble. Chocolatey's version of com0com isn't signed and doesn't work in the action runner. While you can download a signed version from sourceforge, I'm getting CERT_E_WRONG_USAGE errors when trying to load those, on my machine or CI.

I've also given Virtual Serial Port Driver a try, to no avail.

---

Helpful links relating to driver signing:

• Lookup signed drivers
• Official Windows Driver Signing Tutorial
• How to Test-Sign a Driver Package
• Someone's copy of the script