search

allure-framework / allure-java

Allure integrations for Java test frameworks
Apache License 2.0
353 stars 223 forks source link

[allure-testng] Attachment on failed config is not added to the failed configuration method #776

Open baflQA opened 2 years ago

baflQA commented 2 years ago

Describe the bug There is a test with a configuration method (BeforeMethod) that fails with an exception. There is an IConfigurationListener implementation provided, that creates an attachment on failed configuration. An attachment is expected to be present in the report inside the failed configuration method container, but it is not. An attachment can be found in the allure-results directory but seems not to be attached to any test container.

To Reproduce Steps to reproduce the behaviour:

  1. Run https://github.com/baflQA/allure-java/blob/master/allure-testng/src/test/java/io/qameta/allure/testng/AllureTestNgTest.java#L710 test

Expected behaviour An attachment added on failed configuration is added to the failed configuration method.

Additional context Provided a fork to reproduce an issue https://github.com/baflQA/allure-java

baflQA commented 2 years ago

Hi @baev! Would You be able to have a look?

baev commented 2 years ago

I'm not sure we will ever support such behaviour since Allure uses listeners itself to collect data from TestNG and we can't control the listeners order

baev commented 2 years ago

in most cases as workaround you can use AllureLifecycle listeners instead of TestNG listeners

baflQA commented 2 years ago

Ok, I will have a look. But my idea was that MAYBE all the logic of collecting the results of a before configuration methods should be handled inside onConfigurationSuccess/onConfigurationFailure methods of IConfigurationListener.