search

avocado-framework / avocado

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
https://avocado-framework.github.io/
Other
345 stars 343 forks source link

avocado list --filter-by-tags broken on version >= 106 #6066

Open hjelmeland opened 2 weeks ago

hjelmeland commented 2 weeks ago

Describe the bug avocado list --filter-by-tags list all tests in directory, no matter what tag expression is used.

Last good version: 105

System information:

richtja commented 1 week ago

Hi @hjelmeland, thank you for opening this issue. Can you please provide some example with wrong behaviour? It will help me to reproduce this issue. Thank you very much.

hjelmeland commented 1 week ago

Create test.py:

from avocado import Test

class filterByTags(Test):
    def test_filterByTags(self):
        pass

Then do avocado list -t sometag test.py

With avocado 105 no test is listed, as expected.

With avocado >= 106 the test is listed, not as expected.

Adding a tag like """:avocado: tags=disabled""" to the test makes no difference on avocado >= 106.

hjelmeland commented 1 week ago

for the test I ran ubuntu 22:04 docker container and did 

pip install --force-reinstall avocado-framework==105
pip install --force-reinstall avocado-framework==106
richtja commented 1 week ago

Hi @hjelmeland, thank you very much for the reproducer. I was able to reproduce it.

It looks like that it is located only to the list plugin, because the overall tag filtering works well. Also when I use the --verbose option the results are still wrong, but the test summary is correct.

$ avocado -V list -t fast examples/tests/
Type                 Test                                                                       Uri                                                                        Resolver             Tag(s)
avocado-instrumented examples/tests/abort.py:AbortTest.test                                     examples/tests/abort.py:AbortTest.test                                     avocado-instrumented failure_expected
avocado-instrumented examples/tests/assert.py:Assert.test_assert_raises                         examples/tests/assert.py:Assert.test_assert_raises                         avocado-instrumented
avocado-instrumented examples/tests/assert.py:Assert.test_fails_to_raise                        examples/tests/assert.py:Assert.test_fails_to_raise                        avocado-instrumented
avocado-instrumented examples/tests/assets.py:Hello.test_gpg_signature                          examples/tests/assets.py:Hello.test_gpg_signature                          avocado-instrumented
avocado-instrumented examples/tests/assets.py:Hello.test_build_run                              examples/tests/assets.py:Hello.test_build_run                              avocado-instrumented
avocado-instrumented examples/tests/cabort.py:CAbort.test                                       examples/tests/cabort.py:CAbort.test                                       avocado-instrumented requires_c_compiler
avocado-instrumented examples/tests/cancel_on_exception.py:CancelOnException.test               examples/tests/cancel_on_exception.py:CancelOnException.test               avocado-instrumented
avocado-instrumented examples/tests/cancel_test.py:CancelTest.test_iperf                        examples/tests/cancel_test.py:CancelTest.test_iperf                        avocado-instrumented
avocado-instrumented examples/tests/cancel_test.py:CancelTest.test_gcc                          examples/tests/cancel_test.py:CancelTest.test_gcc                          avocado-instrumented
avocado-instrumented examples/tests/cancelonsetup.py:CancelOnSetupTest.test_wont_be_executed    examples/tests/cancelonsetup.py:CancelOnSetupTest.test_wont_be_executed    avocado-instrumented
avocado-instrumented examples/tests/canceltest.py:CancelTest.test                               examples/tests/canceltest.py:CancelTest.test                               avocado-instrumented
avocado-instrumented examples/tests/cit_parameters.py:CitParameters.test                        examples/tests/cit_parameters.py:CitParameters.test                        avocado-instrumented fast
exec-test            examples/tests/custom_env_variable.sh                                      examples/tests/custom_env_variable.sh                                      exec-test
...
TEST TYPES SUMMARY
==================
avocado-instrumented: 4

TEST TAGS SUMMARY
=================
fast: 4
net: 2