search

linux-test-project / kirk

Kirk application aims to merge multiple Linux testing frameworks in one tool.
GNU General Public License v2.0
5 stars 5 forks source link

Phase out runtest files #13

Open metan-ucw opened 7 months ago

metan-ucw commented 7 months ago

The runtest files are not flexible enough and maintaining them by hand is error prone. My idea for the future would be to phase out the runtest files eventually and provide more flexible filtering capabilities in the test executing framework based on the test metadata.

The runtest files actually provide two different purposes. Firstly tests are split into different categories, some categories are overlapping and some categories are subset of others. Secondly the runtest files allow us to add test variants, e.g. run test with different sets of parameters.

The idea to replace the test categories is actually quite simple, instead of maintaining list of tests the tests would be filtered by metadata. We would start with the full set of tests and filter out user requested sets. Everything in the syscalls/ directory would be collected as a syscall test. Then we would add lists of syscalls per special purpose, e.g. sysvipc file would contain names of all sysvipc syscalls that could then be again filtered based on the metadata. If we wanted to run CVE tests we would select tests with a CVE tag. All of this should be actually present in the test metadata already, for new library testcases only though. We can easily start to prototype this filtering capability soon enough and it will be fully usable once we are done with test conversions into the new test library.

Second part would be a bit more complicated, we would have to move the test variants somewhere else. I would vote to move them into the tests themselves and into the tst_test structure. That way the test binary would include list of all variants and we could also add shortcuts to running the different variants, e.g. -t 2 would mean run test variant 2 and -t would list all pre-defined variants. This would also be parsed into the metadata and the test runner framework could use that information to schedule tests as well.

metan-ucw commented 7 months ago

@tbird20d FYI

acerv commented 7 months ago

This ticket would make sense in the LTP issues list as well.

metan-ucw commented 7 months ago

@acerv I guess that the part about moving test variants into the test could be tracked as a LTP ticket as well, will you create that one or should I do so?

acerv commented 7 months ago

@metan-ucw you have a better vision on the specific implementation. Thanks!