search

avocado-framework / avocado

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
https://avocado-framework.github.io/
Other
345 stars 344 forks source link

[BUG - Job API] 'run.references' is not respected in some cases #4001

Closed willianrampazzo closed 4 years ago

willianrampazzo commented 4 years ago

Consider the following test:

#!/usr/bin/env python3

import os
import sys

from avocado import Test
from avocado.core.job import Job

class PassTest(Test):
    def test_foo(self):
        config = {'core.show': ['none'],
                  'run.references': ['/bin/true']}
        with Job(config) as j:
            j.run()

    def test_bar(self):
        config = {'core.show': ['none'],
                  'run.references': ['/bin/true']}
        with Job(config) as j:
            j.run()

if __name__ == '__main__':
    DIR = os.path.dirname(os.path.abspath(__file__))

    reference = '%s:PassTest.test_foo' % __file__
    config = {'run.references': [reference],
              'core.show': ['app'],
              'yaml_to_mux.files': [os.path.join(DIR, 'sample.yaml')]}

    with Job(config) as j:
        sys.exit(j.run())

With the following YAML file:

foo: !mux
  a:
    name: 'a'
  b:
    name: 'b'

bar: !mux
  c:
    name: 'c'
  d:
    name: 'd'

Running the test with python3 or avocado run works as desired ignoring test_bar as needed:

$ python3 /tmp/passjob_with_test.py 
JOB ID     : 651447d55e0749071826a74fa757e6ebf353e400
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.25-651447d/job.log
 (1/4) /tmp/passjob_with_test.py:PassTest.test_foo;c-a-c352:  JOB ID     : 8cb549b40cb830b902d36b88e1e0e5546b62d9f8
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.25-8cb549b/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.39 s)
 (2/4) /tmp/passjob_with_test.py:PassTest.test_foo;d-a-e000:  JOB ID     : 0a974750b58e2b173824b49cc664f9714587315b
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.25-0a97475/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.38 s)
 (3/4) /tmp/passjob_with_test.py:PassTest.test_foo;c-b-cc64:  JOB ID     : edd9c5520482755971b7f90c31ea73bde42ce11c
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.25-edd9c55/job.log
 (1/1) /bin/true: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.31 s)
 (4/4) /tmp/passjob_with_test.py:PassTest.test_foo;d-b-b767:  JOB ID     : fae50c14e7d420e722bd4925c90b6e13c8fad32e
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.25-fae50c1/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.37 s)
RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0

Renaming the tests to something longer:

$ cat /tmp/passjob_with_test.py 
#!/usr/bin/env python3

import os
import sys

from avocado import Test
from avocado.core.job import Job

class PassTest(Test):
    def test_foo_foo_foo_foo(self):
        config = {'core.show': ['none'],
                  'run.references': ['/bin/true']}
        with Job(config) as j:
            j.run()

    def test_foo_foo_foo_foo_bar(self):
        config = {'core.show': ['none'],
                  'run.references': ['/bin/true']}
        with Job(config) as j:
            j.run()

if __name__ == '__main__':
    DIR = os.path.dirname(os.path.abspath(__file__))

    reference = '%s:PassTest.test_foo_foo_foo_foo' % __file__
    config = {'run.references': [reference],
              'core.show': ['app'],
              'yaml_to_mux.files': [os.path.join(DIR, 'sample.yaml')]}

    with Job(config) as j:
        sys.exit(j.run())

Running with avocado run is still consistent:

$ avocado run /tmp/passjob_with_test.py 
/home/wrampazz/src/avocado/avocado.dev/avocado/plugins/run.py:283: FutureWarning: The following arguments will be changed to boolean soon: sysinfo, output-check, failfast and keep-tmp. 
  warnings.warn("The following arguments will be changed to boolean soon: "
JOB ID     : 9c9aff9100181739b18af982ba3866eb969ca1ad
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.30-9c9aff9/job.log
 (1/2) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo:  JOB ID     : 13f7a3cce7940d84f82bf64a6197e70a408ada8c
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.30-13f7a3c/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2020-07-16T18.30-13f7a3c/results.html
PASS (0.19 s)
 (2/2) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo_bar:  JOB ID     : c160661c4b52f01742b06085fde51bd81a020c3b
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.30-c160661/job.log
 (1/1) /bin/true: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2020-07-16T18.30-c160661/results.html
PASS (0.16 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2020-07-16T18.30-9c9aff9/results.html
JOB TIME   : 0.45 s

Running with pyhton3 it does not ignore the second test anymore:

$ python3 /tmp/passjob_with_test.py 
JOB ID     : 47f0051ef771a18da606647f9ad6dabb34bee472
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-47f0051/job.log
 (1/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo;c-a-c352:  JOB ID     : 6b25ba1f1ec0104d70365f6750db32a842ebecf8
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-6b25ba1/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.38 s)
 (2/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo;d-a-e000:  JOB ID     : 3ee48809c8ab8bf81049e314dd3ded5e6b0d8bc8
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-3ee4880/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.34 s)
 (3/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo;c-b-cc64:  JOB ID     : 8b1f90e494cb1d7940d23435fbc8cbefa34cd29a
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-8b1f90e/job.log
 (1/1) /bin/true: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.32 s)
 (4/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo;d-b-b767:  JOB ID     : dad2beb734ac55c06d83cedb460ece65139e5eb4
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-dad2beb/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.38 s)
 (5/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo_bar;c-a-c352:  JOB ID     : 836222406518a6d75d11a81a8c7232fd74a36f58
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-8362224/job.log
 (1/1) /bin/true: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.30 s)
 (6/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo_bar;d-a-e000:  JOB ID     : 652c3d110dfba23fbef7bd3bc5ddd0ac703c8479
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-652c3d1/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.40 s)
 (7/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo_bar;c-b-cc64:  JOB ID     : 5f5dbe78c1a1d0e6a3a93bed5d7f6764fb0f5437
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-5f5dbe7/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.37 s)
 (8/8) /tmp/passjob_with_test.py:PassTest.test_foo_foo_foo_foo_bar;d-b-b767:  JOB ID     : 4031a1ff95c52e1b11e66e31e903b01d61d856e2
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-07-16T18.31-4031a1f/job.log
 (1/1) /bin/true: PASS (0.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
PASS (0.38 s)
RESULTS    : PASS 8 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
beraldoleal commented 4 years ago

Running with avocado run is still consistent:

@willianrampazzo:

I did some tests here, and unless I did something wrong, it is consistent:

Looking to your output here seems to me that it is consistent. The method test_foo_foo_foo_foo_bar() it was called, or did miss something?

You asked to run tests that start with test_foo_foo_foo_foo, and this should be both. Looking at your logs I can see that both were executed. Maybe I'm lost with too many foo, bars. :)

The problem might be the following: Since that you have if __name__ == '__main__' this "parent job" is only executed when calling with python3. Makes sense?

willianrampazzo commented 4 years ago

Running with avocado run is still consistent:

@willianrampazzo:

I did some tests here, and unless I did something wrong, it is consistent:

Looking to your output here seems to me that it is consistent. The method test_foo_foo_foo_foo_bar() it was called, or did miss something?

You asked to run tests that start with test_foo_foo_foo_foo, and this should be both. Looking at your logs I can see that both were executed. Maybe I'm lost with too many foo, bars. :)

The problem might be the following: Since that you have if __name__ == '__main__' this "parent job" is only executed when calling with python3. Makes sense?

Okay, after looking again at the tests I added here and your comment, I concluded Avocado uses the test reference as a regex, so test_foo reference actually means test_foo*. This may be a feature, but may confuse testers if they are starting with Avocado.

I looked at https://avocado-framework.readthedocs.io/en/80.0/guides/user/chapters/concepts.html#test-references and thought the explanation confusing. We should state somewhere that INSTRUMENTED tests are resolved as test_*.

So, maybe this issue should cover a document update.

willianrampazzo commented 4 years ago

More precisely, update the documentation here: https://avocado-framework.readthedocs.io/en/80.0/guides/user/chapters/concepts.html#instrumented to state how the tests are resolved.

beraldoleal commented 4 years ago

@willianrampazzo I think that the first link you sent is the best place because talks about the references. I will submit a PR there.