Test scripts

[networkimprov]

Run scripts found in a test/ directory after reboot, before client-close. First script checks log files for update-related errors.

[thomasdziedzic]

Run scripts found in a test/ directory after reboot, before client-close.

do you have such a script?

First script checks log files for update-related errors.

how does it check for update related errors? I thought this was only to make sure updates don't break the boot process and to make sure some tests we run afterwards don't break. How do we notify the server that the reboot was successfull but the tests failed? should never close it?

[networkimprov]

I was thinking check the system logs for obvious errors.

When a test script fails, the system should probably wait there for admin intervention, so not call /close. Eventually we could let /close post an error log to forward to admins...

[thomasdziedzic]

I thought we wanted to run actual unit tests when we talked about running system check tests after a reboot.

Parsing log files for errors is a high effort, small returns project in my opinion. Unless you have anything specific in mind?

[networkimprov]

Well, can find out if any systemd services failed to start? Or if systemd logged any internal errors?

For unit tests, we'll want to test any essential apps, esp our own.

[thomasdziedzic]

Ok, I think Tony mentioned that he would like to do some hardware checks after an update, maybe we can write those initially?

We could aggregate a list of systemd failed units also if you would like.

[networkimprov]

For now a failed test script can cause pacman-watch to exit, so the server notices the client has gone quiet.

@tmlind, suggestions for ways to test system health post-update?

[tmlind]

Here are some tests that could be scripted to be run automatically:

• USB gadgets, also check speed and USB charging
• MMC speed test
• WLAN connectivity and speed test
• Measure idle power consumption from battery connector without USB cable
• Check timers in /proc/interrupts don't generate lots of interrupts during idle

[networkimprov]

Timers test -- the most likely to reveal sw problems; discussed here https://github.com/networkimprov/arch-packages/issues/26 WLAN test -- via P2P link with host PC; suggestions for benchmark util? USB test -- exercise usb-ethernet with benchmark; other gadgets require host-side script? Disk test -- suggestions for benchmark util? USB charging -- do we need EVT1's battery gauge for this? Battery test -- self-updating units will be sitting around plugged in, so can't do these here.

[networkimprov]

@tmlind or @gostrc, any comments on above Q's?

[thomasdziedzic]

p2p link test

We would have to set up a configuration where it runs a script remotely to setup the anvl for p2p mode, and then setup the computer host to connect to the anvl and I guess run a ping once it's done.

usb test

when we boot the anvl, we would have to detect the usb0 interface coming up and configure the host usb network with the correct settings. Then we can ping the anvl ip

disk test

possibly hdparm or some other utility which exposes an api?

[self@anvl ~]$ sudo hdparm -t /dev/mmcblk0 

/dev/mmcblk0:
 Timing buffered disk reads: 108 MB in  3.02 seconds =  35.81 MB/sec

usb charging

I'm not sure how we would test this, since we would have to discharge the battery and then somehow enable charging without unplugging the cable, we would also have to do this pretty quickly since we want the test to run relatively quickly :)

[networkimprov]

Suggestions for an ethernet benchmark util?

[thomasdziedzic]

ethernet benchmark

possibly iperf? I guess it depends what specific datapoints we want to be collecting. https://github.com/esnet/iperf is a possibility but again, we might want to find a tool with an api rather than parsing the output.

[networkimprov]

@tmlind, your thoughts on the above?

[tmlind]

sounds good to me, the battery test we want to wait with for evt1 as right now we don't really get much information