allure-framework / allure-python

Allure integrations for Python test frameworks
https://allurereport.org/
Apache License 2.0
719 stars 235 forks source link

Changing an id of a pytest parameter set breaks test's history #744

Closed delatrie closed 1 year ago

delatrie commented 1 year ago

I'm submitting a ...

What is the current behavior?

Assigning or changing an id of a parameter set in pytest's parametrize mark breaks the test's run/rerun history.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

Given we have the following test file:

import pytest

@pytest.mark.parametrize("v", [1])
def test_issue744_reproduction(v):
    pass

When we run pytest (without the --clean-alluredir argument), and then assign ids to the parameter sets:

import pytest

@pytest.mark.parametrize("v", [1], ids=["a"])
def test_issue744_reproduction(v):
    pass

When we run pytest again we end up with two test results with different historyId values.

If we generate the report it will look like this:

What is the expected behavior?

Both results should be considered retries (i.e., should have the same 'historyId' values). Assigning an id to a parameter set should only affects the default title of the test (where a portion of the nodeid is used).

What is the motivation / use case for changing the behavior?

More flexibility to refactor tests without losing the history of runs/reruns.

Please tell us about your environment:

Other information

Most probably will be fixed together with #743.