search

qca / boardfarm

Main repo moved to:
https://github.com/mattsm/boardfarm/
BSD 3-Clause Clear License
70 stars 54 forks source link

where to find the test purpose documentation? #83

Closed yi-liang closed 7 years ago

yi-liang commented 7 years ago

we're adding the boardfarm tests as part of the CI and trying to understand the purpose of the existing test cases. where can we find a documentation about it please? as an example, test case "sleep 2 min", is that because ssh connect default timeout is 2 min? but if we connect to DUT via serial, is the test still valid? How about the purpose of the test case "sleep 5 min"? It would be very helpful if we can have a document to understand why the test cases were introduced and what they're testing. Thanks.

mbanders commented 7 years ago

Test Cases such as "sleep 2 minutes" only have meaning within a Test Suite. A Test Suite is an ordered series of test cases. A user can put test cases in any order and even repeat test cases.

See examples of test suites here: https://github.com/qca/boardfarm/blob/master/testsuites.cfg

A test case like "sleep 2 minutes" was written so that if someone needs a sleep in in their test suite - perhaps after enabling wifi, enabling a kernel module, rebooting, or whatever - they can easily put that into their suite.

Some people created suites that run a lot of network traffic, sleep a bit, and repeat. An example test suite you could create:

[example]
RootFSBootTest
InterfacesShow
LanDevPingRouter
RouterPingWanDev
RestartNetwork
Sleep2Minutes
LanDevPingRouter
RouterPingWanDev
RestartNetwork
Sleep2Minutes
LanDevPingRouter
RouterPingWanDev
RestartNetwork
Sleep2Minutes
LanDevPingRouter
RouterPingWanDev
RestartNetwork
Sleep2Minutes
LanDevPingRouter
RouterPingWanDev
 ...

Basically "Sleep2Minutes" and some other tests might not actually test anything - they are just there if you need it in your suite.

mbanders commented 7 years ago

Other tests in https://github.com/qca/boardfarm/blob/master/tests/status.py also don't really test anything by themselves. "Logread" simply displays the contents of the system log. "DiskUse" simply displays info about how the filesystem is being used.

You could modify those tests to make them more useful to you. For example you could make "Logread" fail if the word "error" appeared in the system log. You could make "MemoryUse" fail if there isn't enough free memory.