search

taverntesting / tavern

A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax
https://taverntesting.github.io/
MIT License
1k stars 191 forks source link

Incorrect test file path printed when multiple targets are supplied #817

Open schooler-hpe opened 1 year ago

schooler-hpe commented 1 year ago

I originally submitted this as a pytest issue but they suggested that I report it here instead.

In each of two directories, I have one Tavern test file that will run with pytest:

C02VF3KDHTD8:api schooler$ ls 2-disruptive/test_hardware_negative.tavern.yaml 3-destructive/test_components.tavern.yaml
2-disruptive/test_hardware_negative.tavern.yaml 3-destructive/test_components.tavern.yaml

When I execute them both with one invocation, the summary output shows an incorrect test file path.

C02VF3KDHTD8:api schooler$ python3 -m pytest 2-disruptive/ 3-destructive/
========================================================================================================== test session starts ===========================================================================================================
platform darwin -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0
rootdir: <REDACTED>, configfile: pytest.ini
plugins: tavern-1.23.3
collected 2 items

2-disruptive/test_hardware_negative.tavern.yaml .                                                                                                                                                                                  [ 50%]
2-disruptive/test_components.tavern.yaml .                                                                                                                                                                                         [100%]

=========================================================================================================== 2 passed in 5.74s ============================================================================================================

The summary output should show 3-destructive/test_components.tavern.yaml as the last test and not 2-disruptive/test_components.tavern.yaml which doesn't exist.

Additional comment from the pytest thread that was closed: "I suggest to report this in the tavern repository, as I suspect there might be an issue on how they are creating the test items."

michaelboulton commented 1 year ago

It's definitely possible that it's using the wrong folder name but I can't see how, the code is in the tavern/testutils/pytesthook/file.py file but at a glance it seems to be doing the right thing

michaelboulton commented 1 year ago

I can't reproduce this locally

$ ls subfolder_*
subfolder_1:
test_auth_key.tavern.yaml

subfolder_2:
test_auth_key.tavern.yaml
$ py.test subfolder_*
...
subfolder_1/test_auth_key.tavern.yaml::Test 1 PASSED
subfolder_2/test_auth_key.tavern.yaml::Test 2 PASSED
$ py.test subfolder_* --collect-only
...
<YamlFile tests/integration/subfolder_1/test_auth_key.tavern.yaml>
  /home/michael/code/tavern/tests/integration/subfolder_1/test_auth_key.tavern.yaml
  <YamlItem Test 1>
    Test 1:
    1: Test stage
<YamlFile tests/integration/subfolder_2/test_auth_key.tavern.yaml>
  /home/michael/code/tavern/tests/integration/subfolder_2/test_auth_key.tavern.yaml
  <YamlItem Test 2>
    Test 2:
    1: Test stage