there's been interest in running tests on a phsyical DUT, but no access to autokit hardware
this limits some functionality - mainly concerned with flashing, but also hdmi, bluetooth, and some networking manipulation, and GPIO
however many of the OS tests actually are fine - so this PR will have a collection of changes to allow for running tests on a physical DUT, from your laptop
Test suite specific caveats:
OS suite: Must put device in local mode first (via dashboard). Also manually enable persistent logging in the dashboard too. If your device has wifi, provision it to use your wifi (same network as laptop).
Cloud suite: Must not be in local mode, "SSH authentication test" must be set as unstable
Hup suite: Annoying in general - make sure you provision your DUT into the OLD OS version you want to hup from. Then in the image property put the path to the <hostapp>.docker hostapp image, instead of false
You must provision your DUT manually to a balena cloud fleet, with no release, and your organization must have access to it. The DUT must be on the same network as your laptop. Ensure the DUT is in development mode first. Then run:
make local-test
If there are ssh authentication errors, then the organization you put in config.js doesn't have access to the DUT you provisioned.
should be added in the above config.js, right after manual:true, only if testing in staging.
The device should have both local mode enabled in the dashboard, and developmentMode:true in it's config.json, for the OS suite.
If the OS test suite need to be re-run, /mnt/boot/system-proxy should be deleted, the device should then be re-booted and local-mode should be toggled to off and back on prior to restarting the test suite. Depending on the leftovers from a failed or interrupted test suite, re-flashing may be necessary.
On particular devices that also have a p2p-dev-wlP1p1s0 interface for example, the connectivity test should be modified to
let connection = adaptor === 'wireless' ? 'wifi' : 'Wired';
return this.worker.executeCommandInHostOS(
`nmcli d | grep '${connection}' | grep connected | awk '{print $1}' | head -n1`,
How to use: set your
workspace/config.js
to:Test suite specific caveats: OS suite: Must put device in local mode first (via dashboard). Also manually enable persistent logging in the dashboard too. If your device has wifi, provision it to use your wifi (same network as laptop).
Cloud suite: Must not be in local mode, "SSH authentication test" must be set as unstable Hup suite: Annoying in general - make sure you provision your DUT into the OLD OS version you want to hup from. Then in the
image
property put the path to the<hostapp>.docker
hostapp image, instead offalse
You must run this branch of the tests from meta balena: https://github.com/balena-os/meta-balena/tree/ryan/manual_tests
You must provision your DUT manually to a balena cloud fleet, with no release, and your
organization
must have access to it. The DUT must be on the same network as your laptop. Ensure the DUT is in development mode first. Then run:If there are ssh authentication errors, then the
organization
you put inconfig.js
doesn't have access to the DUT you provisioned.Other considerations:
should be added in the above config.js, right after
manual:true,
only if testing in staging.The device should have both local mode enabled in the dashboard, and developmentMode:true in it's config.json, for the OS suite.
If the OS test suite need to be re-run, /mnt/boot/system-proxy should be deleted, the device should then be re-booted and local-mode should be toggled to off and back on prior to restarting the test suite. Depending on the leftovers from a failed or interrupted test suite, re-flashing may be necessary.
On particular devices that also have a p2p-dev-wlP1p1s0 interface for example, the connectivity test should be modified to
For the cloud suite, the following line should be changed to use "that" instead of "this": https://github.com/balena-os/meta-balena/blob/ryan/manual_tests/tests/suites/cloud/suite.js#L248