frozenstein
commented
7 years ago All the show-* tools should be covered by the tests. All of them have some filtering options, so all of the options should be tested. Having that, some combinations of hte filtering options would be great to have too, but let's start with covering the options separately. As an example, let's take rcl-show-environments:
rcl-show-environments
rcl-show-environments --arch=ppc64
rcl-show-environments --microarch=IVB
rcl-show-environments --family=6 --model=45
rcl-show-environments --family=6 --model=45 --stepping=7
rcl-show-environments --virt=kvm
rcl-show-environments --kernel=3.10.0-693.el7
rcl-show-environments --vendor=GenuineIntel
rcl-show-environments --microarch=IVB --csv
The family/model/stepping options make sense to test as above. Also, currently, virt, kernel and vendor options don't work, which is another bug.
Try to develop a configurable testsuite containing two parts: one that runs a specified tool with a specified list of options and saves logs another that compares two sets of logs and reports a fail for each pair of logs that differst one from each other.
The set of tools and options should be easily configurable, so the core of the first part should be the same for all the tools and options. It can either have some lists/tuples defining the coverage at the beginning or taking it from a file.
How it could be called?
./run_tests.py rcl-show-environments
The above should run all the tests for the tool specified on the command line and save all the logs, let's say into logs/rcl-show-environments/.
Please, create a dir tests in the project and place everything there.
mpavlase
develop branch is pretty 'live', we need to ensure that huge refactoring wouldn't break existing results. Most of scripts (probably all of them) produce output to stdout that can be compared via diff.
If if does, it can be still valid scenario, due bugs in old code those are already fixed in develop branch.