search

coreos / coreos-ci

CoreOS CI powers upstream testing for CoreOS projects.
https://jenkins-coreos-ci.apps.ocp.ci.centos.org/
6 stars 10 forks source link

Add `bodhi-testing.yaml` and support specifying tests to run #57

Closed jlebon closed 8 months ago

jlebon commented 8 months ago

jobs/bodhi-trigger: read SRPM list from bodhi-testing.yaml

Add a new bodhi-testing.yaml file containing the list of SRPMs to test. We'll augment the schema of this file later on to allow e.g. running only a subset of tests.

This will also in the short-term serve as the source of truth for which packages are gated on these tests or not. Long-term, we may use this file slightly differently, though likely the tests to run per component will still live there (including the default tests to run on any unlisted component).

While we're here, add a few more core packages to the list.

bodhi-testing.yaml: support specifying tests to run

For some packages, it's silly to run the full testsuite when they're not involved or tested at all in e.g. the testiso scenarios.

Let's allow configuring the list of tests to run per SRPM.

This is also prep for being able to run e.g. just basic for a much wider set of packages.

dustymabe commented 8 months ago

For some packages, it's silly to run the full testsuite when they're not involved or tested at all in e.g. the testiso scenarios.

Let's allow configuring the list of tests to run per SRPM.

Totally agree. Nice. I didn't think it would be this easy to do this.

This is also prep for being able to run e.g. just basic for a much wider set of packages.

Exactly! I think we should build and run basic tests for all packages in FCOS probably.

Maybe we should dynamically pick up the set of tests that are in FCOS by looking at the most recent build of FCOS (i.e. not hardcoding the list as it changes).

dustymabe commented 8 months ago

How will these changes interact with the current denylist in the FCOS config? May there be times where no tests get run because the test is denylisted?

dustymabe commented 8 months ago

Maybe we should dynamically pick up the set of tests that are in FCOS by looking at the most recent build of FCOS (i.e. not hardcoding the list as it changes).

Yeah, hardcoding test names here isn't great. Not sure I understand your suggestion. How would you get the list of tests that were run on an FCOS build, and how would it allow not hardcoding subsets of tests?

Silly me. I meant to say

Maybe we should dynamically pick up the set of rpms that are in FCOS

RPMs, not tests.