balena-os / leviathan

A distributed hardware testing framework
https://balena-os.github.io/leviathan
Apache License 2.0
22 stars 6 forks source link

enable testing without autokit on physical DUTs #1060

Open rcooke-warwick opened 9 months ago

rcooke-warwick commented 9 months ago

How to use: set your workspace/config.js to:

module.exports = [{
    deviceType: '<DEVICE_TYPE_SLUG',
    suite: `${__dirname}/../suites/os`,
    config: {
        networkWired: false,
        networkWireless: false,
        downloadVersion: 'latest',
        balenaApiKey: '<API_KEY>',
        balenaApiUrl: 'balena-cloud.com',
        organization: '<YOUR_BALENA_CLOUD_USERNAME',
        dutIp: '<LOCAL_IP_OF_YOUR_DUT',
        dutUuid: '<UUID_OF_YOUR_DUT',
    manual: true
    },
    debug: {
        unstable: ["SSH authentication test"],
    },
    image: false, 
    workers: ['http://worker'],
}]

Test suite specific caveats: OS suite: Must put device in local mode first (via dashboard). Also manually enable persistent logging in the dashboard too. If your device has wifi, provision it to use your wifi (same network as laptop).
Cloud suite: Must not be in local mode, "SSH authentication test" must be set as unstable Hup suite: Annoying in general - make sure you provision your DUT into the OLD OS version you want to hup from. Then in the image property put the path to the <hostapp>.docker hostapp image, instead of false

You must run this branch of the tests from meta balena: https://github.com/balena-os/meta-balena/tree/ryan/manual_tests

You must provision your DUT manually to a balena cloud fleet, with no release, and your organization must have access to it. The DUT must be on the same network as your laptop. Ensure the DUT is in development mode first. Then run:

make local-test

If there are ssh authentication errors, then the organization you put in config.js doesn't have access to the DUT you provisioned.

Other considerations:

 sshConfig: {
                host: 'ssh.balena-staging-devices.com',
                port: '22',
    }

should be added in the above config.js, right after manual:true, only if testing in staging.

The device should have both local mode enabled in the dashboard, and developmentMode:true in it's config.json, for the OS suite.

If the OS test suite need to be re-run, /mnt/boot/system-proxy should be deleted, the device should then be re-booted and local-mode should be toggled to off and back on prior to restarting the test suite. Depending on the leftovers from a failed or interrupted test suite, re-flashing may be necessary.

On particular devices that also have a p2p-dev-wlP1p1s0 interface for example, the connectivity test should be modified to

 let connection = adaptor === 'wireless' ? 'wifi' : 'Wired';
                                                return this.worker.executeCommandInHostOS(
                                                        `nmcli d  | grep '${connection}' | grep connected | awk '{print $1}' | head -n1`,

For the cloud suite, the following line should be changed to use "that" instead of "this": https://github.com/balena-os/meta-balena/blob/ryan/manual_tests/tests/suites/cloud/suite.js#L248