Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
This has been blocking regression testing for a while because I'm unable to run network tests using testflinger, which depends on checkbox-remote to run the tests on the SUT from the agent container.
On a bare metal test, we have the file /etc/xdg/canonical-certification.conf that provides a list of IPs for iperf targets to the network test.
This has been failing when using checkbox remote because, apparently, the client side of that connection has to own that data and then pass it to the individual tests on the remote side to be executed (if I'm understanding this correctly).
So the suggestion was to set up the environment variable instead (TEST_TARGET_IPERF) on the agent container so that when checkbox-cli is run, they will be passed on to the SUT when the tests are executed.
This does not work either. To try this, I set up the environment variable on my local machine, and ran network tests and, as with other attempts, they failed because the environment where network.py is being executed does not have the environment var, nor does it have access to the config file.
----------------------------[ Running job 45 / 45 ]-----------------------------
------------[ Multi-NIC Iperf3 stress testing for NIC enp33s0f0np0 ]------------
ID: com.canonical.certification::ethernet/multi_iperf3_nic_device1_enp33s0f0np0
Category: Ethernet Device tests
--------------------------------------------------------------------------------
ERROR:root:Valid target server has not been supplied.
ERROR:root:Configuration settings can be configured 3 different ways:
ERROR:root:1- If calling the script directly, pass the --target option
ERROR:root:2- Define the TEST_TARGET_IPERF environment variable
ERROR:root:3- If running the test via checkbox/plainbox, define the
ERROR:root:target in /etc/xdg/canonical-certification.conf
ERROR:root:Please run this script with -h to see more details on how to configure
--------------------------------------------------------------------------------
Outcome: job failed
Then I tried setting up the ENV var on teh SUT
ubuntu@birdo:~$ export TEST_TARGET_IPERF=10.1.11.230,10.1.11.235,10.1.11.231,10.1.11.236,10.245.128.3
ubuntu@birdo:~$ echo $TEST_TARGET_IPERF
10.1.11.230,10.1.11.235,10.1.11.231,10.1.11.236,10.245.128.3
----------------------------[ Running job 45 / 45 ]-----------------------------
------------[ Multi-NIC Iperf3 stress testing for NIC enp33s0f0np0 ]------------
ID: com.canonical.certification::ethernet/multi_iperf3_nic_device1_enp33s0f0np0
Category: Ethernet Device tests
--------------------------------------------------------------------------------
ERROR:root:Valid target server has not been supplied.
ERROR:root:Configuration settings can be configured 3 different ways:
ERROR:root:1- If calling the script directly, pass the --target option
ERROR:root:2- Define the TEST_TARGET_IPERF environment variable
ERROR:root:3- If running the test via checkbox/plainbox, define the
ERROR:root:target in /etc/xdg/canonical-certification.conf
ERROR:root:Please run this script with -h to see more details on how to configure
--------------------------------------------------------------------------------
Outcome: job failed
I then put the config file on my local machine and tried again, and this also failed just like every other attempt with checkbox-remote.
So to summarise:
Checkbox run on SUT directly using config file: PASS
Checkbox run remotely on SUT with config file on SUT: FAIL
Checkbox run remotely on SUT with ENV Var set on STU: FAIL
Checkbox run remotely on SUT with ENV Var exported to local system: FAIL
Checkbox run remotely on SUT with ENV Var passed on cli when calling checkbox remote: FAIL
Checkbox run remotely on SUT with config file present on local system
This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1974245
Summary
Description
This has been blocking regression testing for a while because I'm unable to run network tests using testflinger, which depends on checkbox-remote to run the tests on the SUT from the agent container.
On a bare metal test, we have the file /etc/xdg/canonical-certification.conf that provides a list of IPs for iperf targets to the network test.
This has been failing when using checkbox remote because, apparently, the client side of that connection has to own that data and then pass it to the individual tests on the remote side to be executed (if I'm understanding this correctly).
So the suggestion was to set up the environment variable instead (TEST_TARGET_IPERF) on the agent container so that when checkbox-cli is run, they will be passed on to the SUT when the tests are executed.
This does not work either. To try this, I set up the environment variable on my local machine, and ran network tests and, as with other attempts, they failed because the environment where network.py is being executed does not have the environment var, nor does it have access to the config file.
First, I exported it locally:
and ran
and that failed as described above.
Next, I tried it this way:
And that too failed:
Then I tried setting up the ENV var on teh SUT
I then put the config file on my local machine and tried again, and this also failed just like every other attempt with checkbox-remote.
So to summarise:
Tags: ['servcert-364']