tmt testsuite reorganization needed (plans, tests and used tags)

[lukaszachy]

Motivation: Current centos-stream-8 has broken broken podman which makes CI fail. Reasonable action would be to disable (ignore) tests which do provision -h container as they cannot pass there. However with the current state of test suite it would be too much effort. Current tests are using provision methods as they want, there is no "external" (read adjustable) way how to limit them and so basically each test would need to be modified.

Has short brainstorm with @psss and this needs more thinking about what to do.

Current usage

1. Test picks provision method To verify functionality, test has picked one method and uses it. It doesn't make sense to run this test for other provision methods. Some of them (e.g. local) can be destructive. Example: /tests/prepare/require

2. Test needs to run on all provision methods Test knows that it should run on (all) possible provision methods as they can behave differently. Example: /tests/core/spaces

3. Test can use selected provision methods Test supports particular provision method. This is used in e.g. /plans/provision/virtual - Plan uses provision method from its name, installs tmt inside and lastly run tier:1 tests which can run inside such provisioned machine (e.g. nested VM are not expected so test which uses provision -h virtual can't run inside VM, similar for containers) E.g in plan /plans/provision/container

Current tag (e.g. virtual) means that test it can run in virtual

Concept of idea IMHO all of them can be covered if we have a global environment value containing provision methods (USE_PROVISION) to use and support way for a to specify that:

• Prefers to use some provision method ("prefers" for short)
• Is only for several provision methods ("needs")
• Cannot use some provision method ("cannot")
• Has to run for several provision methods ("several")

With this features

• Test picks provision method would set "prefers", and if it shouldn't run on some provision then "cannot" as well. So /prepare/require would set "prefers=container" and "cannot=local". If it runs with other provision it will not hurt but expected is that it runs with container (which would be its default value for USE_PROVISION.
• Test needs to run on all provision methods and Test can use selected provision methods would be tagged as "several". It would be possible to combine with "cannot" as well.
• If test needs some method it will set "needs" and will not be run anywhere else.

Looking at this "several" tag might be just an indication to check that it is executed on each relevant provision method.

So for current plans - /plans/provision/METHOD should run all tests (grouped by tier, including no tier) which "prefer" that method, or have "several" tag but exclude any test which "cannot" this method.

Other plans (integration, features) will run with provision as test needs, but author of the plan needs to make sure to limit by cannot according to what is available.

I'd prefer tag to specify what it can or cannot use as provision. Just because something runs in VM doesn't mean that provision virtual is impossible - one can have nested virtualization set up. Test doesn't need to care about that - it just should state that it needs to use virtual.

Duration problem

Currently, the test suite takes quite a long time to run (roughly an hour for all checks to pass). This is a problem when merging multiple PRs. Upon merging one PR, others need to be rebased and pushed and CI again waited for. This becomes even worse when more people want to merge a PR at the same time.

Possible causes:

• installing dependencies in containers/VM
• VM download/boot time

While reorganizing the test suite, we should also consider the CI use-case. Perhaps we could make it more parallel? Ideally, the CI run should not take more than 15 minutes (obviously shorter being better).

Downstream plugins

Some changes break API used by downstream plugins (use internal libraries and cannot be developed publicly) which we find only after change reaches 'main' branch. There should be a way how to run private test in packit.

[psss]

Thanks for the great summary! I'm not completely sure about the type 3. Is it expected to also cover this use case?

• Test can run under guest provisioned using given provision method

For example, /tests/core/docs should not be run under a container as man pages are intentionally not installed there.

[lukaszachy]

For example, /tests/core/docs should not be run under a container as man pages are intentionally not installed there.

In this case I'd say you either set "needs" to methods you know can contain man pages or you set both "cannot" and "prefers"

[psss]

To clarify even bit more, I see two types of the provision use cases:

• inside test defines under which provision methods it should be executed, e.g. /tests/core/docs
• outside test selects which provision methods it will be using to exercise a feature, e.g. /tests/core/spaces

Do I understand it correctly that needs would be used for the inside type and the rest of the tags for the outside use cases?

[lukaszachy]

Well, I'd argue that inside is not really necessary. If I pick at /tests/core/docs isn't that test missing 'man' as a require?

[psss]

No, including man would not help. The problem is that in containers dnf is configured in a special way so that it does not install man pages to save space and make container images as small a possible.

[lukaszachy]

Proposing for 1.17 - more people are merging, there is a lot of precious time wasted. Note that 'description' added 'Duration' problem and "Downstream plugins"

[lukaszachy]

@psss I think we can close this. You have fixed the problem with /provision/ reorganisation. Please reopen if not :)