JUnit report wrong when using nested scenarioOutline

[ckeiner]

Spectrum only offers Data-Driven-Tests with the scenarioOutline-method defined in the gherkinDSL. However, while they can be nested like SpecificationDSL's describe, the JUnit report isn't correct when more than one example is used. The report shows all examples but cannot assign a status to it, i.e show that it passed or failed.

Here's a minimal example:

@RunWith(Spectrum.class)
public class NestedScenarioOutline
{
    {
        feature("A feature", () ->
            {
                scenarioOutline("The background with an example", (browser) ->
                    {
                        scenarioOutline("Eating less cucumbers than I have ", (have, eat, remaining) ->
                            {
                                final AtomicInteger cukes = new AtomicInteger();
                                given("I have " + have + " cucumbers", () ->
                                    {
                                        cukes.set(have);
                                    });
                                when("I eat " + eat + " cucumbers", () ->
                                    {
                                        cukes.addAndGet(-eat);
                                    });
                            }, withExamples(example(12, 5, 7), example(20, 5, 15)));

                    }, withExamples(example("Chrome"), example("Firefox")));
            });
    }
}

And the corresponding report in Eclipse:

The first example, Chrome, does not have any status, neither do the children of it.

In addition, if an error occurs, the test suite fails but no test shows any failure. Here's an example, which fails when the outer scenarioOutline receives "Chrome" as example:

@RunWith(Spectrum.class)
public class NestedScenarioOutline
{
    {
        feature("A feature", () ->
            {
                scenarioOutline("The background with an example", (browser) ->
                    {
                        scenario("Eating less cucumbers than I have ", () ->
                            {
                                final AtomicInteger cukes = new AtomicInteger();
                                given("I have 12 cucumbers", () ->
                                    {
                                        System.out.println("Current Browser is " + browser);
                                        cukes.set(12);
                                        if (browser.equals("Chrome"))
                                        {
                                            throw new AssertionError();
                                        }
                                    });
                                when("I eat 5 cucumbers", () ->
                                    {
                                        cukes.addAndGet(-5);
                                    });
                                then("I have 7 cucumbers left", () ->
                                    {
                                        assertThat(cukes.get(), equalTo(7));
                                    });
                            });
                        scenarioOutline("Eating less cucumbers than I have ", (have, eat, remaining) ->
                            {
                                final AtomicInteger cukes = new AtomicInteger();
                                given("I have " + have + " cucumbers", () ->
                                    {
                                        cukes.set(have);
                                    });
                                when("I eat " + eat + " cucumbers", () ->
                                    {
                                        cukes.addAndGet(-eat);
                                    });
                                then("I have " + remaining + " cucumbers left", () ->
                                    {
                                        assertThat(cukes.get(), equalTo(remaining));
                                    });
                            }, withExamples(example(12, 5, 7), example(20, 5, 15)));

                    }, withExamples(example("Chrome"), example("Firefox")));
            });
    }
}

The JUnit report as shown in Eclipse:

The report further shows, that the when and then step of the scenario in the scenarioOutline were ignored, which isn't true for the Firefox example.

Since a scenarioOutline can already contain other scenario(Outline)s, it would be nice if the report were correct. This would also add the ability to supply test data to Spectrum's fixtures (beforeEach, beforeAll, etc.) , which is often very helpful.

[bvkatwijk]

I ran into the same issue. To me it seems that specs having the same description in suites with the same description causes this incorrect reporting. I worked around it by using a delegation base class and appending a random number to the spec descriptions.