search

serenity-bdd / serenity-cucumber

Cucumber integration for the Serenity BDD Reporting library
Other
78 stars 74 forks source link

Senerity console logger fails all subsequent tests in a scenario outline when one test fails. #161

Closed atifansari13 closed 5 years ago

atifansari13 commented 6 years ago

I have a scenario outline similar to this:

Scenario Outline: Add two numbers Given I have a calculator When I add and Then the result for and should be

Examples: | num1 | num2 | total | | -2 | 3 | 1 | | 10 | 15 | 3 | | 99 | -99 | 0 | | -1 | -10 | -11 |

I run this scenario outline and as expected, it will run 4 scenarios

When one of the scenarios (say scenario#2 in this case) fails, the serenity console logger says TEST FAILED for scenarios 2 , 3 and 4 even though 3 and 4 as passing and they are marked as passed in the serenity report.

2018-07-10T14:56:32.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:32.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:32.2690597Z __ ____ _ _ __
2018-07-10T14:56:32.2696272Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:32.2699629Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:32.2702138Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:32.2705488Z () | | || ||/ || ||// __|||_/ 2018-07-10T14:56:32.2708116Z \\
2018-07-10T14:56:32.2708801Z 2018-07-10T14:56:32.2709347Z TEST FAILED: Add two numbers 2018-07-10T14:56:32.2710054Z -------------------------------------------------------------------------- 2018-07-10T14:56:32.2710529Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:32.2710961Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging 2018-07-10T14:56:32.2711531Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:32.2711836Z ____ _ _ __
2018-07-10T14:56:32.2712176Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:32.2712666Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:32.2713431Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:32.2713871Z () | | || ||/ || ||// __|||___/ 2018-07-10T14:56:32.2714282Z \\
2018-07-10T14:56:32.2714580Z 2018-07-10T14:56:32.2714956Z TEST FAILED: Add two numbers 2018-07-10T14:56:32.2715372Z -------------------------------------------------------------------------- 2018-07-10T14:56:32.2715873Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:32.2716761Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging

2018-07-10T14:56:59.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:59.2690597Z __ ____ _ _ __
2018-07-10T14:56:59.2696272Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:59.2699629Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:59.2702138Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:59.2705488Z () | | || ||/ || ||// __|||_/ 2018-07-10T14:56:59.2708116Z \\
2018-07-10T14:56:59.2708801Z 2018-07-10T14:56:59.2709347Z TEST FAILED: Add two numbers 2018-07-10T14:56:59.2710054Z -------------------------------------------------------------------------- 2018-07-10T14:56:59.2710529Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:59.2710961Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging 2018-07-10T14:56:59.2711531Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:59.2711836Z ____ _ _ __
2018-07-10T14:56:59.2712176Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:59.2712666Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:59.2713431Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:59.2713871Z () | | || ||/ || ||// __|||___/ 2018-07-10T14:56:59.2714282Z \\
2018-07-10T14:56:59.2714580Z 2018-07-10T14:56:59.2714956Z TEST FAILED: Add two numbers 2018-07-10T14:56:59.2715372Z -------------------------------------------------------------------------- 2018-07-10T14:56:59.2715873Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:59.2716761Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging

2018-07-10T14:57:25.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:57:25.2690597Z __ ____ _ _ __
2018-07-10T14:57:25.2696272Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:57:25.2699629Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:57:25.2702138Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:57:25.2705488Z () | | || ||/ || ||// __|||_/ 2018-07-10T14:57:25.2708116Z \\
2018-07-10T14:57:25.2708801Z 2018-07-10T14:57:25.2709347Z TEST FAILED: Add two numbers 2018-07-10T14:57:25.2710054Z -------------------------------------------------------------------------- 2018-07-10T14:57:25.2710529Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:57:25.2710961Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging 2018-07-10T14:57:25.2711531Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:57:25.2711836Z ____ _ _ __
2018-07-10T14:57:25.2712176Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:57:25.2712666Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:57:25.2713431Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:57:25.2713871Z () | | || ||/ || ||// __|||___/ 2018-07-10T14:57:25.2714282Z \\
2018-07-10T14:57:25.2714580Z 2018-07-10T14:57:25.2714956Z TEST FAILED: Add two numbers 2018-07-10T14:57:25.2715372Z -------------------------------------------------------------------------- 2018-07-10T14:57:25.2715873Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:57:25.2716761Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging

This is the build i am using.

    <dependency>
        <groupId>net.serenity-bdd</groupId>
        <artifactId>serenity-core</artifactId>
        <version>1.9.19</version>
    </dependency>
    <dependency>
        <groupId>net.serenity-bdd</groupId>
        <artifactId>serenity-cucumber</artifactId>
        <version>1.9.7</version>
    </dependency>
wakaleo commented 6 years ago

This is as expected, as the test as whole has failed, even if some of the inidividual scenarios have passed.

atifansari13 commented 6 years ago

I understand that at the end of all scenarios, the test as a whole failed but once a test fails after every scenario (even though a scenario passes) we see a test failed message. Is that expected?

This is what i am seeing: Scenario 1 - pass TEST PASSED

Scenario 2 -fail TEST FAILED

Scenario 3 - pass TEST FAILED

Scenario 4 - pass TEST FAILED

final test outcome - fail TEST FAILED

Dont you think the ones highlighted should be TEST PASSED instead of TEST FAILED?

wakaleo commented 6 years ago

Yes, the console is just reporting the current state of the current test. Changing this would require major rework in the Serenity reporting API. Feel free to take a look, but it isn't considered a high priority for the core team at this stage.