search

teemtee / fmf

Flexible Metadata Format
GNU General Public License v2.0
22 stars 28 forks source link

Use `tree wget -F --dirsfirst` to clarify directory structure #173

Open abitrolly opened 2 years ago

abitrolly commented 2 years ago 
cd tmp
mkdir wget
cd wget
mkdir download protocols recursion smoke
cd protocols
mkdir ftp https http
cd ..
touch main.fmf
cd ..
tree wget -F --dirsfirst
abitrolly commented 2 years ago

I am not sure where runtest.sh should be. The doc doesn't mention it.

  1. Is it one runner for all dirs located in the root?
  2. Is it a runner that is placed in each dir?
lukaszachy commented 2 years ago

I am not sure where runtest.sh should be. The doc doesn't mention it.

1. Is it one runner for all dirs located in the root?

2. Is it a runner that is placed in each dir?

Ouch, example was oversimplified. That usage doesn't make much sense without tmt metadata specification. I guess we should come up with something generic enough and not dependent on tmt to stop always using fmf with tmt which makes it difficult to realise what is what.

If I take tmt into account (meaning that runtest.sh is command which should be executed for test and as such it should be part of "test" attribute):

abitrolly commented 2 years ago

Ouch, example was oversimplified.

Oversimplified is better than overcomplicated. :D

I guess we should come up with something generic enough and not dependent on tmt to stop always using fmf with tmt which makes it difficult to realise what is what.

Something artificial that is not being used in production won't make it better. Tests are at least common scenario both for coders and DevOps teams.

  • Each directory should have it's own 'runtest.sh' file Or you can have single runtest.sh but either path or test should be adjusted so process executing test has it on expected path (process changes dir according to 'path' attribute and executes whatever is written in 'test' attribute)

By path attribute, do you mean /download: part here?

/download:
    description: Check basic download options
    tester: Petr Šplíchal <psplicha@redhat.com>
    tags: [Tier2, TierSecurity]
    test: ./runtest.sh
    time: 3 min

So to clarify, for the wget/main.fmf example above, that script that will be executed for this test is wget/download/runtest.sh, and wget/runtest.sh will be ignored. Is that right?

And if the contents of the wget/main.fmf is changed to the below text, the wget/runtest.sh will be ignored too, and instead two separate scripts will be run. wget/download/runtest.sh and wget/recursion/runtest.sh

tester: Petr Šplíchal <psplicha@redhat.com>
tags: [Tier2, TierSecurity]
test: ./runtest.sh

/download:
    description: Check basic download options
    time: 3 min
/recursion:
    description: Check recursive download options
    time: 20 min

So the question now is how should I rewrite this wget/main.fmf to reuse wget/runtest.sh in both cases?

psss commented 2 years ago

Oh yeah, those examples were created long time ago, when we were still just dreaming about tmt ;-) Looking at it now I see many attributes have been renamed:

description --> summary
tester --> contact
tags --> tag
time --> duration

Let's fix those as well to prevent further confusion. For the structure I'd propose to keep directories only in the tree overview and then add an example of one directory which contains test.sh (instead of runtest.sh) and main.fmf.

By path attribute, do you mean /download: part here?

No, it's a dedicated tmt key path. But I'd say let's not use the path key in the fmf examples to make things simple. It does not demonstrate any fmf features anyway.

abitrolly commented 2 years ago

You should have write access to this branch. It should be the best way to put the content right.

lukaszachy commented 2 years ago

@psss, @abitrolly What if we linked tmt tests? For example "/tests/prepare/shell": Top main.fmf defines common attributes, prepare changes them for tests for prepare step and final test adds just link to coverage and summary.

It also shows that in "data" directory is another fmf root - which is hidden from the top one (another reason why .fmf/version exists)

lukaszachy commented 2 years ago

Another idea - would showing final metadata help more to understand what is happening? E.g. show that without inheritance one would need to spell out whole:

/download:
    description: Check basic download options
    time: 3 min
    tester: Petr Šplíchal <psplicha@redhat.com>
    tags: [Tier2, TierSecurity]
    test: ./runtest.sh
/recursion:
    description: Check recursive download options
    time: 20 min
    tester: Petr Šplíchal <psplicha@redhat.com>
    tags: [Tier2, TierSecurity]
    test: ./runtest.sh
psss commented 2 years ago

E.g. show that without inheritance one would need to spell out whole:

This is covered under the hierarchy section. So we have the comparison.

For example "/tests/prepare/shell":

The tangible real-life example sounds like a great idea!

abitrolly commented 2 years ago

@psss, @abitrolly What if we linked tmt tests?

That would create recursive link, which will only worsen the mess in my head.