Scenarios run multiple times

kerryb commented 7 years ago

Tested with Cabbage 0.3.3 and Elixir 1.5.0.

When there are n scenarios in a feature file, it seems that each scenario is run n times. I've created a minimal test case here, but basically what happens is this …

Starting with a single scenario, all seems fine:

Feature: Tests get run multiple times
  Scenario: Scenario 1
    Given a precondition

$ mix test

18:10:43.675 [debug] Parsing line: "Feature: Tests get run multiple times"

18:10:43.676 [debug] Parsing line: "Scenario: Scenario 1"

18:10:43.680 [debug] Parsing line: "Given a precondition"

18:10:43.682 [debug] Parsing line: ""

18:10:43.740 [info]  Line 2:  Scenario: Scenario 1

18:10:43.741 [info]         Given a precondition
.

Finished in 0.1 seconds
1 test, 0 failures

Add a second scenario, and the number of tests jumps to four:

Feature: Tests get run multiple times
  Scenario: Scenario 1
    Given a precondition

  Scenario: Scenario 2
    Given a precondition

$ mix test

18:12:58.887 [debug] Parsing line: "Feature: Tests get run multiple times"

18:12:58.889 [debug] Parsing line: "Scenario: Scenario 1"

18:12:58.892 [debug] Parsing line: "Given a precondition"

18:12:58.894 [debug] Parsing line: ""

18:12:58.894 [debug] Parsing line: "Scenario: Scenario 2"

18:12:58.894 [debug] Parsing line: "Given a precondition"

18:12:58.894 [debug] Parsing line: ""

18:12:59.001 [info]  Line 2:  Scenario: Scenario 1

18:12:59.001 [info]         Given a precondition
.
18:12:59.001 [info]  Line 2:  Scenario: Scenario 1

18:12:59.001 [info]         Given a precondition
.
18:12:59.002 [info]  Line 5:  Scenario: Scenario 2

18:12:59.002 [info]         Given a precondition
.
18:12:59.002 [info]  Line 5:  Scenario: Scenario 2

18:12:59.002 [info]         Given a precondition
.

Finished in 0.1 seconds
4 tests, 0 failures

With three scenarios, the number jumps to nine, and so on.

I've added code to the step definition that writes to a file, to verify that the tests are actually running multiple times (rather than it being a problem with reporting). I've also verified that the behaviour is the same regardless of whether or not the scenarios share steps.

mgwidmann commented 7 years ago

😮 o that sucks... have you tried #41 to see if it fixes your issue?

kerryb commented 7 years ago

Yes, I tried pointing mix to your branch yesterday and it fixes the issue. Thanks :-)

mgwidmann commented 7 years ago

While that branch doesn't have your problem, it will have another issue until its updated. Once he fixes the comment I made I'll merge and release a new version... I will also remove the versions that had this issue as well once I've done that.

rawkode commented 7 years ago

Sorry about this, @kerryb. Silly mistake! I've pushed the latest fix.

cabbage-ex / cabbage

Scenarios run multiple times #40