kernelkit / infix

Linux :yellow_heart: NETCONF = Infix
https://kernelkit.github.io/
GNU General Public License v2.0
45 stars 11 forks source link

Add framework for generating test spec + report #439

Open troglobit opened 1 month ago

troglobit commented 1 month ago

Nothing fancy, could be a README.md (markdown) in each test directory, or pydoc to extract the test's docstring in reStructured Text format.

Either way, how tests are documented should be standardized: description, topology, pass criteria, allowed deviations, i.e., reasons for SKIP, e.g., tests running on virtual hardware.

The goal is to be able to extract the documentation for each test and generate a test specification based on the test suite. Each test is numbered (e.g., section headings) and the test results should match those numbers in the test report. The spec and report can be different documents. Possibly everything is extracted to a test-result/ directory with sub-directories for each test so any type of format can be generated.

troglobit commented 1 month ago

Decision: