allure-framework / allure-python

Allure integrations for Python test frameworks
https://allurereport.org/
Apache License 2.0
730 stars 238 forks source link

Fix historyId to be dependent on dynamic allure parameters #743

Closed delatrie closed 1 year ago

delatrie commented 1 year ago

I'm submitting a ...

What is the current behavior?

Parameters added via the allure.dynamic.parameter inside a test body don't affect allure history of the test.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

Run the following example multiple times to produce multiple *-result.json files:

import allure
import time

def test_issue743_reproduction():
    allure.dynamic.parameter("time", time.perf_counter())

These results all have the same historyId. In such a case allure reporter shows us only one test case with several retries:

image

What is the expected behavior?

There should exists one test case per run, each with no retries in a way, similar to native pytest parameters.

  1. Take the nodeid of a test.
  2. Take all dynamic parameters of the test with excluded set to False.
  3. Sort the parameters alphabetically by their names (historyId should not depend on parameters order).
  4. Append a string representation of the values to the nodeid (use some separator to prevent collisions with other tests).
  5. Calculate hash of the resulting string and use it as historyId.

Related code: https://github.com/allure-framework/allure-python/blob/12085cd76d1c0ec78ef90a4981a31e7f8b4546b4/allure-pytest/src/listener.py#L102

Please tell us about your environment:

delatrie commented 1 year ago

This also blocks #430.