Closed atifansari13 closed 5 years ago
This is as expected, as the test as whole has failed, even if some of the inidividual scenarios have passed.
I understand that at the end of all scenarios, the test as a whole failed but once a test fails after every scenario (even though a scenario passes) we see a test failed message. Is that expected?
This is what i am seeing: Scenario 1 - pass TEST PASSED
Scenario 2 -fail TEST FAILED
Scenario 3 - pass TEST FAILED
Scenario 4 - pass TEST FAILED
final test outcome - fail TEST FAILED
Dont you think the ones highlighted should be TEST PASSED instead of TEST FAILED?
Yes, the console is just reporting the current state of the current test. Changing this would require major rework in the Serenity reporting API. Feel free to take a look, but it isn't considered a high priority for the core team at this stage.
I have a scenario outline similar to this:
Scenario Outline: Add two numbers Given I have a calculator When I add and
Then the result for and should be
Examples: | num1 | num2 | total | | -2 | 3 | 1 | | 10 | 15 | 3 | | 99 | -99 | 0 | | -1 | -10 | -11 |
I run this scenario outline and as expected, it will run 4 scenarios
When one of the scenarios (say scenario#2 in this case) fails, the serenity console logger says TEST FAILED for scenarios 2 , 3 and 4 even though 3 and 4 as passing and they are marked as passed in the serenity report.
2018-07-10T14:56:32.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:32.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:32.2690597Z __ ____ _ _ __
2018-07-10T14:56:32.2696272Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:32.2699629Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:32.2702138Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:32.2705488Z () | | || ||/ || ||// __|||_/ 2018-07-10T14:56:32.2708116Z \\
2018-07-10T14:56:32.2708801Z 2018-07-10T14:56:32.2709347Z TEST FAILED: Add two numbers 2018-07-10T14:56:32.2710054Z -------------------------------------------------------------------------- 2018-07-10T14:56:32.2710529Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:32.2710961Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging 2018-07-10T14:56:32.2711531Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:32.2711836Z ____ _ _ __
2018-07-10T14:56:32.2712176Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:32.2712666Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:32.2713431Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:32.2713871Z () | | || ||/ || ||// __|||___/ 2018-07-10T14:56:32.2714282Z \\
2018-07-10T14:56:32.2714580Z 2018-07-10T14:56:32.2714956Z TEST FAILED: Add two numbers 2018-07-10T14:56:32.2715372Z -------------------------------------------------------------------------- 2018-07-10T14:56:32.2715873Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:32.2716761Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging
2018-07-10T14:56:59.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:59.2690597Z __ ____ _ _ __
2018-07-10T14:56:59.2696272Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:59.2699629Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:59.2702138Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:59.2705488Z () | | || ||/ || ||// __|||_/ 2018-07-10T14:56:59.2708116Z \\
2018-07-10T14:56:59.2708801Z 2018-07-10T14:56:59.2709347Z TEST FAILED: Add two numbers 2018-07-10T14:56:59.2710054Z -------------------------------------------------------------------------- 2018-07-10T14:56:59.2710529Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:59.2710961Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging 2018-07-10T14:56:59.2711531Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:56:59.2711836Z ____ _ _ __
2018-07-10T14:56:59.2712176Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:56:59.2712666Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:56:59.2713431Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:56:59.2713871Z () | | || ||/ || ||// __|||___/ 2018-07-10T14:56:59.2714282Z \\
2018-07-10T14:56:59.2714580Z 2018-07-10T14:56:59.2714956Z TEST FAILED: Add two numbers 2018-07-10T14:56:59.2715372Z -------------------------------------------------------------------------- 2018-07-10T14:56:59.2715873Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:56:59.2716761Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging
2018-07-10T14:57:25.2687866Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:57:25.2690597Z __ ____ _ _ __
2018-07-10T14:57:25.2696272Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:57:25.2699629Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:57:25.2702138Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:57:25.2705488Z () | | || ||/ || ||// __|||_/ 2018-07-10T14:57:25.2708116Z \\
2018-07-10T14:57:25.2708801Z 2018-07-10T14:57:25.2709347Z TEST FAILED: Add two numbers 2018-07-10T14:57:25.2710054Z -------------------------------------------------------------------------- 2018-07-10T14:57:25.2710529Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:57:25.2710961Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging 2018-07-10T14:57:25.2711531Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - 2018-07-10T14:57:25.2711836Z ____ _ _ __
2018-07-10T14:57:25.2712176Z / / | _| / | | | / \ | | | | _| \ 2018-07-10T14:57:25.2712666Z ()____| | | | | | \ \ | | | | / \ | || | | | | | | | 2018-07-10T14:57:25.2713431Z |__| | | | | | ) || | | / __ \ | || || || || | 2018-07-10T14:57:25.2713871Z () | | || ||/ || ||// __|||___/ 2018-07-10T14:57:25.2714282Z \\
2018-07-10T14:57:25.2714580Z 2018-07-10T14:57:25.2714956Z TEST FAILED: Add two numbers 2018-07-10T14:57:25.2715372Z -------------------------------------------------------------------------- 2018-07-10T14:57:25.2715873Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - TEST FAILED AT STEP Then the result for 10 and 15 should be 3 2018-07-10T14:57:25.2716761Z [main] ERROR net.thucydides.core.steps.ConsoleLoggingListener - something i am logging
This is the build i am using.