Structure of Gherkin DSL

[davidsowerby]

I've been trying out the Gherkin DSL, and just seen the docs update at #493.

I love the idea of this structure but it does cause some issues with the way tests are counted.

• The Given, When, Then and And are all at the same level of the structure, which means they all get counted as tests - if you use all 4, then it counts as 4 successful tests for what I would see as 1 test! (Your example in the documentation counts as 8 tests - which does seem excessive)

• More importantly, I recall there was some debate about whether the order of tests would be guaranteed - I don't know what the outcome of that was, so there may be no issue here, but if the order of Given, When, Then and And is not guaranteed then that would cause problems.

• has the given, on, it of Spek v1 been dropped completely?

[raniejade]

The Given, When, Then and And are all at the same level of the structure, which means they all get counted as tests - if you use all 4, then it counts as 4 successful tests for what I would see as 1 test! (Your example in the documentation counts as 8 tests - which does seem excessive).

I understand your concern here, but I wouldn't want introduce another concept just for this. I think the issue here is how the tests are reported?

More importantly, I recall there was some debate about whether the order of tests would be guaranteed - I don't know what the outcome of that was, so there may be no issue here, but if the order of Given, When, Then and And is not guaranteed then that would cause problems.

Yep, I'm going to introduce something for this. It will be opt-in and by default execution order is not guaranteed.

has the given, on, it of Spek v1 been dropped completely?

Yes, it is replaced by the gherkin style.

[davidsowerby]

@raniejade thanks - I'm not too worried about the number of tests - it is a bit mis-leading but otherwise not too much of a problem.

Yep, I'm going to introduce something for this. It will be opt-in and by default execution order is not guaranteed.

Do you have an issue I can track this on?

[davidsowerby]

Thinking about it further, without guaranteed execution order this structure is invalid isn't it?

At any time the test could suddenly go from Given, When, Then to Then,Given, When (or any other combination) - which clearly would not make sense and would lead to random, incorrectly failing tests.

It seems to me that with the current structure, the Scenario is really the 'test' - the Given When Then elements are just steps within the test.

[punkisdead]

IIRC cucumber doesn't differentiate between Given, When, Then, And and treats them all as "steps" when tracking execution. When you look at the output from cucumber it will state how many "steps" were executed, how many successful, how many skipped, how many failed, etc.

You would be correct in thinking that the logical grouping of a scenario should guarantee the execution order of the various "steps" defined within it.

[davidsowerby]

@punkisdead thanks for that confirmation. I think this highlights the core of this issue. The current Gherkin DSL in Spek is treating each test step as a test.

That leads to both the issues I see here - the lack of guarantee of test step order, and incorrect counting.

The principle of encouraging test isolation - by making no guarantee of test execution order - is fine, but only if it is applied to what is really a test.

[raniejade]

the lack of guarantee of test step order

This should be fixable and is already planned.

, and incorrect counting.

We rely on JUnit 5 as a test runner (which includes reporting as well) and it doesn't have this concept of "steps". I see it as only a problem on how it's being reported not core Spek itself.

[eric-labelle]

I've been trying the gherkin style and must agree that a lot is confusing with the style. As @davidsowerby pointed out, the number of test is not representative but most importantly, since Given When Then are consider tests, the jUnit reports are quite confusing as there is no way to see the scenario in the report and they are usually what describe best the test The before/after EachTest { } is also really confusing as one would expect Scenario are test, not Given When Then