allure-framework / allure-python

Allure integrations for Python test frameworks
https://allurereport.org/
Apache License 2.0
722 stars 236 forks source link

using pytest.mark.parametrize will miss some testcase in report #778

Open Pegasus-Yang opened 10 months ago

Pegasus-Yang commented 10 months ago

I'm submitting a ...

What is the current behavior?

when i use pytest.mark.parametrize ,in specific data situations, i got same historyId for two diffent testcase, when report cameout, only one of two testcase show in report.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

Code to repeat :

data = [
    ["", "admin", 12345],
    ["admin", "", 12345],
]

@pytest.mark.parametrize("name, account, phone_number", data)
def test_add_member(name, account, phone_number):
    print(name, account, phone_number)

What is the expected behavior?

expected got 2 testcase in report

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Other information

Becourse in allure_pytest.util.get_history_id, only use values in original_values to generate historyId Maybe use both name and value will solve the problem

delatrie commented 10 months ago

Thank you for your report, @Pegasus-Yang ! There is a bug in how we hash arguments to create historyId. Due to this bug, empty strings don't affect the result, so the actual sequence of arguments that get into historyId is ["admin", 12345] in both cases. The same bug, I believe, causes some other collisions, like 1 and "1". I'll fix it soon.

saaj commented 3 months ago

Here is a workaround when you don't need the history feature, but just one-off report:

find allure-results -name '*-result.json' -print \
  | xargs -r -n1 -P8 -i -- sh -c 'jq ".historyId=.uuid" {} | sponge {}'