search

avocado-framework / avocado

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
https://avocado-framework.github.io/
Other
336 stars 335 forks source link

Recipe resolver resolves non-existent tests #5924

Closed richtja closed 3 weeks ago

richtja commented 2 months ago

Describe the bug When you make a mistake in a test uri in a recipe file, avocado will resolve this file without any issues. This bug makes avocado raise errors during test runtime.

Steps to reproduce recipe file with wrong uri:

{"kind": "avocado-instrumented", "uri": "examples/tests/passtes.py:PassTest.test"}

Expected behavior

$ avocado -V list recipe.json
Type Test Tag(s)

Resolver             Reference                 Info
avocado-instrumented examples/tests/passtes.py File "examples/tests/passtes.py" does not exist or is not a regular file
golang               examples/tests/passtes.py go binary not found
magic                examples/tests/passtes.py Word "examples/tests/passtes.py" is not a valid magic word
python-unittest      examples/tests/passtes.py File "examples/tests/passtes.py" does not exist or is not a regular file
robot                examples/tests/passtes.py File "examples/tests/passtes.py" does not end with ".robot"
rogue                examples/tests/passtes.py Word "examples/tests/passtes.py" is not the magic word
runnable-recipe      examples/tests/passtes.py File "examples/tests/passtes.py" does not end with ".json"
runnables-recipe     examples/tests/passtes.py File "examples/tests/passtes.py" does not end with ".json"
exec-test            examples/tests/passtes.py File "examples/tests/passtes.py" does not exist or is not a executable file
tap                  examples/tests/passtes.py File "examples/tests/passtes.py" does not exist or is not a executable file

TEST TYPES SUMMARY
==================
$avocado run recipe.json
No tests found for given test references: examples/tests/passtes.py
Try 'avocado -V list examples/tests/passtes.py' for details

Current behavior

$ avocado -V list recipe.json
Type                 Test                                    Tag(s)
avocado-instrumented examples/tests/passtes.py:PassTest.test

Resolver             Reference                                  Info
avocado-instrumented ../testing/avocado_instrumented_wrong.json File "../testing/avocado_instrumented_wrong.json" does not end with ".py"
golang               ../testing/avocado_instrumented_wrong.json go binary not found
magic                ../testing/avocado_instrumented_wrong.json Word "../testing/avocado_instrumented_wrong.json" is not a valid magic word
python-unittest      ../testing/avocado_instrumented_wrong.json File "../testing/avocado_instrumented_wrong.json" does not end with ".py"
robot                ../testing/avocado_instrumented_wrong.json File "../testing/avocado_instrumented_wrong.json" does not end with ".robot"
rogue                ../testing/avocado_instrumented_wrong.json Word "../testing/avocado_instrumented_wrong.json" is not the magic word

TEST TYPES SUMMARY
==================
avocado-instrumented: 1
$ avocado run recipe.json
Error running method "pre_tests" of plugin "fetchasset": [Errno 2] No such file or directory: 'examples/tests/passtes.py'

Reproduced traceback from: /home/janrichter/Avocado/avocado/avocado/core/extension_manager.py:229
Traceback (most recent call last):
  File "/home/janrichter/Avocado/avocado/avocado/plugins/assets.py", line 289, in pre_tests
    fetch_assets(
  File "/home/janrichter/Avocado/avocado/avocado/plugins/assets.py", line 236, in fetch_assets
    handler = FetchAssetHandler(test_file, test_file_parse_cache, klass, method)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/janrichter/Avocado/avocado/avocado/plugins/assets.py", line 61, in __init__
    test_file_parse_cache[file_name] = safeloader.find_avocado_tests(
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/janrichter/Avocado/avocado/avocado/core/safeloader/core.py", line 483, in find_avocado_tests
    return find_python_tests("avocado", "Test", _determine_match_python, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/janrichter/Avocado/avocado/avocado/core/safeloader/core.py", line 372, in find_python_tests
    module = PythonModule(path, target_module, target_class)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/janrichter/Avocado/avocado/avocado/core/safeloader/module.py", line 50, in __init__
    with open(self.path, encoding="utf-8") as source_file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'examples/tests/passtes.py'

JOB ID     : a1c095674ea40c7e4782309114a473dd0beefe9b
JOB LOG    : /home/janrichter/avocado/job-results/job-2024-05-02T11.36-a1c0956/job.log
 (1/1) examples/tests/passtes.py:PassTest.test: STARTED
 (1/1) examples/tests/passtes.py:PassTest.test: ERROR: [Errno 2] No such file or directory: '/home/janrichter/Avocado/avocado/examples/tests/passtes.py' (0.01 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/janrichter/avocado/job-results/job-2024-05-02T11.36-a1c0956/results.html
JOB TIME   : 1.62 s

Test summary:
1-examples/tests/passtes.py:PassTest.test: ERROR
clebergnu commented 3 weeks ago

So there are a few possibilities here which can even be complementary:

Show where the resolution is coming from

In the case of having a good (`/tmp/good.json'):

{"kind": "avocado-instrumented", "uri": "examples/tests/passtest.py:PassTest.test"}

and /tmp/bad.json:

{"kind": "avocado-instrumented", "uri": "examples/tests/passtes.py:PassTest.test"}

When doing a verbose list we currently have:

$ avocado -V list /tmp/good.json /tmp/bad.json 
Type                 Test                                     Tag(s)
avocado-instrumented examples/tests/passtest.py:PassTest.test
avocado-instrumented examples/tests/passtes.py:PassTest.test

What we could have is:

$ avocado -V list /tmp/good.json /tmp/bad.json 
Type                 Test                                       Resolver          Tag(s)
avocado-instrumented examples/tests/passtest.py:PassTest.test   runnable-recipe
avocado-instrumented examples/tests/passtes.py:PassTest.test    runnable-recipe

A two-phase resolver feature

It maybe limited to resolvers such as runnable-recipe and runnables-recipe. This will require the resolvers that will implement this to be able to load new resolver classes based on the kind of the resolution, and deduct the resolver that will perform the second phase. Further details bellow:

  1. runnable-recipe resolves bad.json into: 
    {"kind": "avocado-instrumented", "uri": "examples/tests/passtest.py:PassTest.test"}
  2. runnable-recipe asks for the qualified resolver for avocado-instrumented, receives a class that can perform the secondary resolution
  3. runnable-recipe replaces its initial resolution with that from the secondary resolution, whether sucessful or not
richtja commented 3 weeks ago

Hi @clebergnu thank you for detailed description of possible solutions. Even tho I would like to have more robust solution, I think that two-phase resolver feature would add more complexity to resolvers implementation, and it will increase learning curve for avocado users, therefore I am satisfied with the first solution resolved in #5951. IMO, we can close this and come back in the future if it's needed.

richtja commented 3 weeks ago

Closing as resolved in #5951.