allure-framework / allure-python

Allure integrations for Python test frameworks
https://allurereport.org/
Apache License 2.0
718 stars 236 forks source link

Allure-behave reporting #518

Open subbro opened 3 years ago

subbro commented 3 years ago

Hi,

I am using allure-behave=2.8.19 version(latest) and I see issues with the status for few of the steps in behave.

I am using a behave + allure-behave formatter for json file generation. I also have retry logic in my code.

Whenever the x step fails in my code x+1,x,+2.. are marked as pass in 1st try. In the second try x+1x+2 is marked as skipped

I am not sure, what is happening. Need help!! Thanks

@sseliverstov

sseliverstov commented 3 years ago

@subbro did you retry steps or scenario? Can you share retry logic?

subbro commented 3 years ago

Hi, I am using the environment.py file to config my retry logic in behave framework

--environment.py-- before_feature(context,feature): ----for scenario in feature.scenarios: ----------if "autoretry" in scenario.effective_tags: ---------------patch_scenario_with_autoretry(scenario,max_attempts=2)

--test.feature-- @autoretry Feature: --Scenario Outline: ------Given let's say this step is successful ------When let's say this step has exception ------Then let's say this step is successful

So, every step of mine has try and except blocks --test.py-- @given(u'') def test1(): ----try: -------statements ----except Exception as e: -------raise AssertionError('{}'.format(type(e))) @when(u'') def test2(): ----try: -------statements ----except Exception as e: -------raise AssertionError('{}'.format(type(e)))

@then(u'') def test3(): ----try: -------statements ----except Exception as e: -------raise AssertionError('{}'.format(type(e)))

I expect the final report generated to be as below with retry logic in place: Status: Failed attempt 1: Given --- Passed When --- Failed Then ---Skipped

attempt 2: Given --- Passed When ---Failed Then -- Skipped

The report I see is as below: Status: Failed attempt 1: Given --- Passed When --- Failed Then ---Passed / sometimes Skipped

attempt 2: Given --- Passed When ---Failed Then -- Failed / sometimes skipped / sometimes passed

Let me know if this info helps !!

@sseliverstov

subbro commented 3 years ago

Hi, any update on this ticket please. Whether it is expected behaviour or a fix is required?

subbro commented 3 years ago

@sseliverstov any update?