dheyay
commented
1 month ago I looked into this issue, looks like from the current coverage data collected from running the feature tests include all files (with imports). This creates a problem when trying to parse the coverage and map the files which the tests correspond to.
To fix this, we need to figure out a way to decide which files have actually executed lines and which files are just imports.
Please describe the scenario where the new feature would be useful
Now that we have the ability to collect coverage information for behave tests, we should technically have all the information we need to automatically select behave tests to run based on the changed files in a PR. It doesn't need to be perfect because we also run all tests daily in jenkins. This could potentially drastically decrease CI time by limiting the tests that run to only those that use the code that was changed.
Describe the solution you'd like
We should run all of the tests and collect coverage information from them. From that we can create a mapping from file -> test. Then in our github workflow, we can check the changed files and only run tests that use those files.
Current behavior
Currently all lxd-container tests run and a subset of cloud tests run. This is sometimes way too much and also often doesn't include pertinent tests.
Additional context
This doesn't need to be fully automated. But if we commit a static mapping to the repo for this, we should at least create a script to easily update it.