Introduction :wave:
Testrun automates specific test cases to verify network and security functionality in IoT devices. It's an open-source tool that manufacturers use to test their IP-capable devices for the purpose of device qualification within Google's Building Operating System (BOS) program.
Motivation :bulb:
Test labs and engineers often need to maintain a large and complex network coupled with dynamic configuration files and constant software updates. Testrun helps address major issues like:
- The complexity of managing a testing network
- The time required to perform testing of network functionality
- The accuracy and consistency of testing network functionality
How it works :triangular_ruler:
Testrun creates an isolated and controlled network environment on a Linux machine. This removes the necessity for complex hardware, advanced knowledge, and networking experience while enabling test engineers to validate device behavior against Google's BOS requirements.
Testrun supports two modes: automated testing and lab network.
Automated testing
Automated testing of the device under test (DUT) begins once the device is operational (steady state). Containerized test modules execute against the device one module at a time. Testrun produces a report with the results after all modules are executed.
Lab network
Testrun provides the network and assistive tools for engineers when manual testing or configuration changes are required, reducing the need to maintain a separate but identical lab network. Testrun handles packet captures and logs for each network service for further debugging.
Minimum requirements :computer:
Hardware
- PC running Ubuntu LTS 20.04, 22.04, or 24.04 (laptop or desktop)
- 2x USB Ethernet adapter (one may be built-in Ethernet)
- Internet connection
Software
Testrun requires Docker. Refer to the installation guide for more information.
Device under test (DUT)
The DUT must be able to obtain an IP address via DHCP.
Get started :arrow_forward:
Once you meet the hardware and software requirements, follow the Testrun Get started guide. Additional guidance is available in the docs directory.
Roadmap :chart_with_upwards_trend:
Testrun continually evolves to further support end users by automating device network behavior against industry standards. For information on upcoming features, check out the Roadmap.
Accessibility :busts_in_silhouette:
We're proud of our tool and strive to provide an enjoyable experience for everyone. Testrun goes through rigorous accessibility testing at each release. Download the Testrun: Accessible features video to learn more.You're welcome to submit a new issue and provide feedback on our implementations. To learn more about Google's Belonging initiative and their approach to accessibility, visit their site.
Issue reporting :triangular_flag_on_post:
If you encounter a problem during setup or use, raise an issue under the Issues tab. Issue templates exist for both bug reports and feature requests. If neither of these apply, raise a blank issue instead.
Contributing :keyboard:
We strongly encourage contributions from the community. Review the requirements on the "How to Contribute" page, then follow the developer guidelines.
FAQ :raising_hand:
1. What should I do if I have an issue while installing or upgrading Testrun?
You can resolve most issues by reinstalling Testrun using these commands:
sudo docker system prune -asudo apt install ./testrun-*.deb
If this doesn't resolve the problem, raise an issue.
2. What device networking functionality does Testrun validate?
Best practices and requirements for IoT devices change often due to technological advances and discovery of vulnerabilities. You can find the current expectations for IoT devices on Google deployments in the Application Security Requirements for IoT Devices. Testrun aims to automate as much of the Application Security Requirements as possible.
3. What services are provided on the virtual network?
The following network services are containerized and accessible to the DUT:
- DHCP in failover configuration with internet connectivity
- IPv6 SLAAC
- DNS
- NTPv4
Note that this list is likely to change over time.
4. Can I run Testrun on a virtual machine?
Testrun can be virtualized if the 2x Ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. Visit the virtual machine guide for additional details.