autotest / virt-test

Linux Virtualization Tests
Other
97 stars 140 forks source link

./run: --list-tests: list disabled tests as well #177

Open ehabkost opened 11 years ago

ehabkost commented 11 years ago

Today I got confused trying to get the "cpuflags" test case to run and not seeing it in the --list-tests output. After asking for help, I learned that the --list-tests output depend on the guest-os choice, and "cpuflags" has "no JeOS" in its config file.

What I request is that --list-tests lists the disabled test cases as well (clearly indicating that they are disabled), so we know the test case exists.

lmr commented 11 years ago

I need to think about this one... some internal reorganization will be required for this.

lmr commented 11 years ago

This is the only one that I could not implement in a straightforward way. The way we generate the list of tests is to parse the base config files without any test filtering, but with the guest OS filter already applied. Now, we could generate 2 sets of tests, one with the guest OS filter not applied, and other with the guest OS filter applied, and mark the difference as "Tests disabled on guest os type [guest-os-type]"... what do you say @ehabkost ?

ehabkost commented 11 years ago

I guess it would work. I don't care too much about the actual implementation, I am just looking for a "user-visible model" that doesn't require understanding the cartesian config file. I like a lot the model exposed by ./run today, because it rarely requires understanding of the config file and cartesian products.

Maybe a solution is to make the config parsing code allow more "introspection". Instead of asking "give me the test dictionaries generated from subtests.cfg", we could ask it "give me the list of variants parsed on subtests.cfg (just the parser syntax tree, without any filtering or cartesian product calculation)".