Document test tags and mark beakerlib integration

[lukaszachy]

Probably we can get rid of following tags:

• artemis
• container
• integration
• virtual

Which would leave us with just provison-*, as_root (find a better name?) and beakerlib

[LecrisUT]

Actually I was thinking if instead a reusable plan can be provided? There are occasionally issues with the installation of dependencies as well, and it would be nice if these could be handled directly here. Basically a reverse-integration test that the tmt dependents should call.

[happz]

Probably we can get rid of following tags:

• artemis
• container
• virtual

Aren’t these candidates for getting provision- prefix? Or did they express something else?

[lukaszachy]

Aren’t these candidates for getting provision- prefix? Or did they express something else?

artemis very likely but the rest I'm not sure if the meaning was 'test runs in "virtual|container"' or "test can use ..."

[psss]

Aren’t these candidates for getting provision- prefix? Or did they express something else?

The container and virtual tags were originally used to mark tests which are suitable to run under container or a virtual provision. For example, it does not make sense to run /tests/core/docs under containers as man pages are not installed there:

https://github.com/teemtee/tmt/blob/ba120a2ba16fa1738927cae1511581928c384680/tests/core/docs/main.fmf#L4

But these were used in the original provision plans which exercised a selected set of tests under give provision method. Now that we have changed the approach, I guess, we could get rid of them.

The only remaining place, where the virtual tag is used is the upstream integration plan in the dist-git repo. It filters our only two tests which should work just fine. Let's try to get rid of it there as well:

• https://src.fedoraproject.org/rpms/tmt/pull-request/120

[lukaszachy]

Actually I was thinking if instead a reusable plan can be provided? There are occasionally issues with the installation of dependencies as well, and it would be nice if these could be handled directly here. Basically a reverse-integration test that the tmt dependents should call.

One of the options. But depends which package is in mind - e.g. testcloud should run '/plans/provision/virtual' while for podman '/plans/provision/container' etc.

For the beakerlib (the initial impulse for this PR) it does make sense run just those 14 tests, even though many more are written with it.

[lukaszachy]

Note to self: /tests/libraries/apache is provision-virtual and provision-only. Not sure how hard / relevant it is to run it as part of beakerlib integration as it would require TF with nested vm support

[psss]

Note to self: /tests/libraries/apache is provision-virtual and provision-only. Not sure how hard / relevant it is to run it as part of beakerlib integration as it would require TF with nested vm support

I would skip that one for beakerlib integration.

[psss]

What about something like 0b1dff5d?

[LecrisUT]

I've added a PR downstream about this. Feel free to force-push or re-open another one if needed: https://github.com/beakerlib/beakerlib/pull/171

[lukaszachy]

Removed tags artemis, container and virtual.

Now just 'integration' is questionable - it is used in plan imported from teemtee/tests. I'll try to get rid of that as well

[lukaszachy]

And I will remove the ref: tag-beakerlib-tests I commited by mistake :D

[LecrisUT]

Update: I tried it locally and ref seems to be needed for now. I've tried with 0b1dff5d4fa9a34e26841eee73c264a7cdf7d2ee and 2a540ae0da1a86f6af94829b8d1c4d9d17dd9239, and only the former worked :/. The f26b71b92b0f2328d14bd2d725b140561a94f230 filter seems to work as well.

My suspicion is that it's because of url

[psss]

Let's perhaps keep the tag-beakerlib-tests ref to verify that the remote plan setup works for @LecrisUT. Then we can remove it before merging the pull request.

[LecrisUT]

I can push some debugging commits downstream pointing to the different commits here. That's how I verified which variant worked with tmt run discover. I am curious to see the one without url

[psss]

I am curious to see the one without url

I already tried that one locally but it froze on path: / exploring the whole filesystem for fmf files...

[LecrisUT]

I am curious to see the one without url

I already tried that one locally but it froze on path: / exploring the whole filesystem for fmf files...

What about path: .? It is very unintuitive, but it seems to point relative to the git repo on the spglib implementation, i.e. when I have it as path: test it did not expand to .distro/test :shrug:

[lukaszachy]

Doesn't work for me with path: .

My testing plan is

plan:
    import:
        url: file:////tmp/r_150917_X8i/tmt
        ref: tag-beakerlib-tests
        path: /plans/friends
        name: /beakerlib

The file:////tmp/r_150917_X8i/tmt is git repo with commited beakerlib.fmf containing

discover:
    how: fmf
    path: .
    filter: "tag:beakerlib&tag:-provision-only"

But the tmt run discover ends with 0 tests selected. From the log it seems that it copied '.' but it was the parent plan's cwd.

[LecrisUT]

From the log it seems that it copied '.' but it was the parent plan's cwd.

Oh that is interesting and unexpected. My intuition would be that it should not take the parent's repo paths. It is interesting how that works on spglib then, because I would expect to be using the path relative to the fmf tree instead of the git repo.

Confirmed the same issue is with spglib. I used that as an example because it does not have the potential confusion of path: ..

[LecrisUT]

So far, downstream it seems to work: https://artifacts.dev.testing-farm.io/d2a9b1cf-d301-48ff-a4e3-5c8f449c90c4/

It will probably fail in various places due to yq missing. Probably that can be resolved by making a symlink to /plans/main.fmf in the other /plans/friends/main.fmf

[psss]

For now copy & pasted the relevant prepare step part in 284d910a. Hopefully the python-yq package will make it into Fedora repos soon and we will be able to get rid of these ugly workarounds.

[psss]

Tests in https://github.com/beakerlib/beakerlib/pull/171 are green.

[psss]

Just one #2744 failure, otherwise green.