Robust SUT control I/O

[richiejp]

We control the SUT with a serial port (SSH or plain serial) connected to a TTY/PTY. Logging in like an interactive user and typing in commands. This is flexible and well supported (most distros allow the user to login to a serial port). However we are resolving a lot of issues with doing this.

Alternatives:

1. Use Expect
2. Use SSH to run commands directly (without simulating the user typing)
3. Run another daemon on the SUT to execute tests. (Executor)

If we do 2. or 3., we may still need 1. to monitor the kernel log.

[richiejp]

18 #16

[shunghsiyu]

I believe alternative 1 should be able to help with #16 and #18, but retrieving exit status will still be a issue, so #20 cannot be helped. Alternative 2 and 3 looks quite similar to a degree, both requires some process running inside SUT so we can more reliably retrieve exit status and stdout/stderr, and thus solving #16, #18 and #20.

Personally I prefer 2 (syz-manager seems to have chosen this path as well), since it can retrieve both exit status and stdout/stderr reliably, and sshd is ubiquitous; however choosing a suitable SSH library might to require a bit of work and I think implementation requires much more work is required compared to alternative 1.

In theory monitoring of the kernel log output could still be done through reading the serial console as we do today?

[richiejp]

OK, this may sound crazy, but it is possible to compile the kernel without TTY or network. In such a case someone could still communicate with the SUT using some method we haven't thought of. However we can write an executor which takes input over stdin and writes to stdout. Then it is possible to pipe I/O from SSH, serial, virtio, bluetooth, USB etc. It is not tightly coupled with any transport mechanism.

In runltp-ng we could use SSH by default, but then you need the network to be configured correctly and some tests interfere with the network interfaces. I think Syzkaller uses a lot of sandboxing and namespaces to avoid that breaking the connection, but we may want to run the LTP tests on the real hardware interfaces.

So I propose:

1. Write a simple generic executor in C which runs on the SUT. Takes I/O from stdin/stdout
2. Use SSH to start and feed executor and a serial port to monitor kernel log
3. Use 2 serial ports to login, monitor the kernel log and start/feed executor

I guess 2. or 3. could be dropped to begin with. As for SSH libraries; I think it is acceptable to call the SSH client executable directly in a Perl script and not use any library. We have to do that for QEMU anyway.

@metan-ucw wdyt?

[metan-ucw]

@richiejp actually yes. The current code that uses SSH and shell to execute testcases is a bit of hack. What we really need is a minimal daemon locked in memory with a realtime priority that executes these testcases and reports the results.

[metan-ucw]

Also there are other considerations here, in a future I would like to be able to run testcases in parallel, so the daemon will gain support to juggle more than one job to keep the machine bussy, but for that we will have to feed it constraints to avoid false negatives...