search

allure-framework / allure2

Allure Report is a flexible, lightweight multi-language test reporting tool. It provides clear graphical reports and allows everyone involved in the development process to extract the maximum of information from the everyday testing process
https://allurereport.org/
Apache License 2.0
4.11k stars 703 forks source link

Using @flaky or @muted or @known tags in Cucumber Scenarios or Feature doesn't reflect in Allure report #1611

Open gnazgit opened 2 years ago

gnazgit commented 2 years ago

Describe the bug Using @flaky or @muted or @known tags in Cucumber Scenarios or Feature doesn't reflect in Allure report. No markers were present in the report when using these tags.

To Reproduce Steps to reproduce the behavior:

  1. Create any cucumber feature file
  2. Add flaky or muted or known tags to either scenario or feature
  3. Observe the report markers

Expected behavior When marked @flaky, scenario or feature should be marked as flaky and should able to filter using Markers in the allure report.

Screenshots Please refer screenshot for expected behavior.

Environment (please complete the following information):

Allure version 2.17.3
Test framework testng@7.5
Test framework cucumber@7
Allure adaptor allure-cucumber7-jvm@2.17.3

With flaky symbol

@baev - please suggest any possible solution. Thanks!

dineshmcac commented 2 years ago

Hi @baev we are also facing the same problem, please suggest

MSlaski commented 2 years ago

Hello @baev I'm also facing this issue with Cucumber6: allure-cucumber6-jvm

MSlaski commented 2 years ago

Hello @baev Do you have any plan to fix this issue?

MSlaski commented 2 years ago

@gnazgit @dineshmcac @baev

I found a workaround for this issue related to @flaky annotation and I will prepare small script to fix report (don't have time to find a root cause at allure's source code)

Basically after test inside allure-results directory you can find some files, the files which we have to modify are -result.json

If test will fail inside a root part of JSON there will be a structure called detailsStatus (sth like that), there all annotations are set to FALSE, but every step has also similar structure (JSON root > steps) and there you can find that, flaky value will be set top true if you added annotation to feature or scenario. Looks like the issue is to aggregate flaky status from steps and set proper value to overall flaky status

My script will be doing 4 things:

  1. Iterate over all *-result.json files (1 test will generate one result.json file)
  2. Check test status - if failed go further
  3. Check flaky status inside a first step - if set to true go further
  4. change value of overall flaky status

I will trigger this script at Jenkins stage before it will generate report

Manually all is working fine with this approach

itkhanz commented 1 year ago

Hi, Is there any solution for this issue as of today? I am running the tests with following dependecies

The Test Report does not properly mark the scenarios that are tagged with @flaky

Phlegethonyarre commented 1 year ago

Can someone help me out with the same issue? Allue 2.21.0, Cucumber 7, Junit 5.

mdaraz commented 1 year ago

Hi, the same issue occurs in allure-cucumber5-jvm.

sanflg commented 1 year ago

Still happening on allure-cucumber7-jvm 2.21.0. Hi @mpkorstanje there is any standard workaround for this? commenting to keep this issue open.

noomorph commented 1 year ago

I think that the maintainers removed the support for the @Flaky annotation without updating the docs.

dirkrooijakkers commented 1 year ago

We're looking for a solution too.

Allure 2.24.0. @badeball/cypress-cucumber-preprocessor 15.0.0. Cypress 12.14.0