search

kernelci / kernelci-core

Core KernelCI tools
https://kernelci.org
GNU Lesser General Public License v2.1
104 stars 97 forks source link

KernelCI-specific test paths are sent to KCIDB #1098

Open spbnick opened 2 years ago

spbnick commented 2 years ago

KCIDB expects tests to be identified by "paths", which consist of a commonly-known test (suite) name followed by names of subtests, separated by dots, pointing to the test (suite) or its subtest being executed.

For example, the first path below points to a run of LTP test suite as a whole, and the rest to various LTP subtests:

ltp
ltp.access01
ltp.alarm02
ltp.brk01

However, KernelCI Native is sending LTP results split into several "suites", e.g. ltp-mm, ltp-pty, and ltp-timers, each with its own subtests e.g. ltp-mm.ksm02, ltp-pty.login, and ltp-timers.clock_settime. This prevents KCIDB from not only aggregating these together into a single ltp result, but also from correlating with results from other CI systems sending LTP results, e.g. ARM and Red Hat. A basic fix for this could be removing the -mm, -pty, -timers, and other suffixes from LTP suite names before sending them to KCIDB.

It looks like the igt-kms-exynos, igt-gpu-i915, igt-kms-rockchip, and so on, are all parts of the same test suite, even if (as it seems) they're built as separate executables. The appropriate name for the test suite could be igt, since it's widely-known, and the names above should be converted to igt.kms and igt.i915, and (what looks like) the details of the execution environment and target hardware (exynos, rockchip, and so on) should be put into the environment.misc structure until KCIDB can accommodate them properly. The test suite maintainers should be contacted for figuring out the exact details.

The baseline and baseline-nfs suites seem to be the same, but perhaps running with different setup/environment. Using the same name for them would allow KCIDB to correlate their results. Additionally, since it seems to be KernelCI specific, is not executed by anyone else, and might not be recognizable outside KernelCI context, we might need to name it kernelci_baseline.

There could be more non-complying names, but we could start with these three.

gctucker commented 2 years ago

I think it would be counter-productive to fix this in kernelci-backend as we're now focusing on getting the new API and pipeline up to speed. So for now, we could just implement a filter in kernelci-backend to only send the test results that have paths that comply with the convention used with KCIDB.