EmptyStackException when using Scenario outlines

[Susclick]

When I use scenario outlines and I get any exception in my Performable action (for example Assertion exception in case of test fails) there is an exception java.lang.EmptyStackException throws in net.thucydides.core.steps.BaseStepListener#updateExampleLineNumber After this exception all test suite will break of execution.

I tried to debug this situation. There is net.thucydides.core.steps.BaseStepListener#getCurrentStep method which tries to get curent step in case when stack of steps is empty. There can be 3 reasons of this behavior:

1. In some cases there is skipping of stack push operation
2. In some cases there is some excess stack pop operation
3. In some reasons there is some excess of BaseStepListener#updateExampleLineNumber method call

Unfortunately I can not attach the code of my project because of security of my company. If to find the reason without code is hard, maybe it is possible to add some additional check for stack is not empty before call net.thucydides.core.steps.BaseStepListener#updateExampleLineNumber ??

Thanks

[wakaleo]

The best would be to debug the code that you have (since you can reproduce the issue) and figure out what the issue is, and propose a Pull Request.

[Susclick]

Hello! Unfortunetly I can not create pull requests form my workplace. I have deeply investigated the issue and found the reason:

1. When we use scenario outlines with examples, for every example there will be created new highLevel step in stack of steps in net.thucydides.core.steps.BaseStepListener#currentStepStack

2. Then this stack will grow for every step and nested substep level in scenario for current example. So for 1 current example we have stack of steps: Example (stack level 0) -> Scenario step (stack level 1) -> Scenario substep for performable (stack level 2) -> etc for all nested substeps

3. Every step/nested step execution is performed in method net.thucydides.core.steps.StepInterceptor#runTestStep. If any step/nested step throws exception, then this exception will be processed in method net.thucydides.core.steps.StepInterceptor#runTestStep in catch block by calling method net.thucydides.core.steps.StepInterceptor#logStepFailure

4. This method will call net.thucydides.core.steps.StepEventBus#stepFailed which will call method stepFailed for subscribed listeners. One of listeners is net.thucydides.core.steps.BaseStepListener. So it will be called method net.thucydides.core.steps.BaseStepListener#stepFailed which will call another method net.thucydides.core.steps.BaseStepListener#currentStepDone which will remove one level of stack net.thucydides.core.steps.BaseStepListener#currentStepStack

So the call of method net.thucydides.core.steps.BaseStepListener#currentStepDone will remove one element from steps stack

5. After removing one element from stack the initial exception of nested low level step will be Re-throw again in net.thucydides.core.steps.StepInterceptor#notifyOfStepFailure And then this exception will be catch again in method net.thucydides.core.steps.StepInterceptor#runTestStep and processing will repeats from step 3 to step 5 until stack net.thucydides.core.steps.BaseStepListener#currentStepStack will be empty.

After all repeats we have an empty stack net.thucydides.core.steps.BaseStepListener#currentStepStack.

6. And after all of this actions cucumber4-jvm runner makes event TestStepStarted for next "skipped" step which locates in Scenario Outline after the step which throw exception. And this event will call the method cucumber.runtime.formatter.SerenityReporter#handleTestStepStarted which tries to execute block: if (this.getContext().isAScenarioOutline()) { int lineNumber = event.getTestCase().getLine(); this.getContext().stepEventBus().updateExampleLineNumber(lineNumber); }

Then calling of this.getContext().stepEventBus().updateExampleLineNumber tries to get current step from empty stack net.thucydides.core.steps.BaseStepListener#currentStepStack

So there is a reason of an exception EmptyStack

[Susclick]

Maybe (really I dont know all subtleties of Serenity implementation) the solution is that in case when we have some steps in scenario outline which are located after the step which throw an exception (some skipped future steps), we should not remove the root level step in stack which was created for "Example" of scenario outline.

[Susclick]

Hello. To reproduce this problem you can change starter scenario to scenario outline from my example and change the url of DuckDuckGo to any incorrect url to generate 404 error in browser for example And after it just run mvn clean verify

You will see the error

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.889 s <<< FAILURE! - in Search by keyword
[ERROR] Feature: Search by keyword  Time elapsed: 2.889 s  <<< ERROR!
java.util.EmptyStackException

Examle of scenario outline for reproducing:

  Scenario Outline: Searching for a term
    Given Sergey is on the DuckDuckGo home page
    When he searches for "<search>"
    Then all the result titles should contain the word "cucumber"

    Examples:
      | search |
      | 12345 |
      | 12347 |

[Susclick]

The problem reproducing in screenplay branch only