Brainstorming: How to handle consecutive steps scenario

[nikowitt]

Hi Jan,

currently, I'm trying to create a table based scenario that depends on consecutive steps:

    // virtual "dataprovider" as we want don't want to execute an own scenario for each record, but to execute it consecutively
    private static Object[][] create_correspondence_with_reference_number_category_provider() {
        return $$(
                $(CODE, CODE, null, 1),
                $(CODE, CODE, null, 2),
                $(CODE, CODE, CATEGORY_NAME_1, 1),
                $(CODE, CODE, null, 3),
                $(CODE, CODE, CATEGORY_NAME_1, 2),
                $(CODE, CODE, CATEGORY_NAME_2, 1),
                $(CODE, CODE, CATEGORY_NAME_1, 3),
                $(CODE, CODE, CATEGORY_NAME_2, 2));
    }

    @Test
    public void create_correspondence_with_reference_number_category() throws Exception {

        given().logged_in_system_user().and().a_new_correspondence_code(CODE).and()
                .new_reference_number_categories(CATEGORY_NAME_1, CATEGORY_NAME_2).and().a_refno_template_with_category();

        assert_that().sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(
                create_correspondence_with_reference_number_category_provider());

    }

(...)
    public void sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(@Table(columnTitles = { "sender code",
            "recipient code",
            "reference category",
            "expected sequence" }) Object[][] data)
            throws Exception {
        for (Object[] o : data) {
            reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
            reference_sequential_number_is((int) o[3]);
        }

    }

When everything is fine and no errors are raised, I can create the following output:

Test Class: com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest

 Scenario: create correspondence with reference number category

         Given logged in system user
           And a new correspondence code
           And new reference number categories category1, category2
           And a refno template with category
   Assert that sequence is correct when correspondence with reference is created consecutively

     | sender code | recipient code | reference category | expected sequence |
     +-------------+----------------+--------------------+-------------------+
     | NEWCODE     | NEWCODE        | null               |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 2 |
     | NEWCODE     | NEWCODE        | category1          |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 3 |
     | NEWCODE     | NEWCODE        | category1          |                 2 |
     | NEWCODE     | NEWCODE        | category2          |                 1 |
     | NEWCODE     | NEWCODE        | category1          |                 3 |
     | NEWCODE     | NEWCODE        | category2          |                 2 |

This is a much better approach than to create a when/then for every case as the scenario grows pretty fast as soon as one additional case is added.

But of course, this way, an error case cannot be displayed in the table itself. To do that, I'd have to be able to catch errors, mark the row where an error occurs and then throw a global error afterwards.

An error case currently looks like this

Test Class: com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest

 Scenario: create correspondence with reference number category

         Given logged in system user
           And a new correspondence code
           And new reference number categories category1, category2
           And a refno template with category
   Assert that sequence is correct when correspondence with reference is created consecutively

     | sender code | recipient code | reference category | expected sequence |
     +-------------+----------------+--------------------+-------------------+
     | NEWCODE     | NEWCODE        | null               |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 2 |
     | NEWCODE     | NEWCODE        | category1          |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 3 |
     | NEWCODE     | NEWCODE        | category1          |                 2 |
     | NEWCODE     | NEWCODE        | category2          |                 1 |
     | NEWCODE     | NEWCODE        | category1          |                 3 |
     | NEWCODE     | NEWCODE        | category2          |                22 |

FAILED: 
Expected: is <22>
     but: was <2>

It is still readable, But only when the assert fails on a unique sequence so I can tell from which row the error is raised without debugging.

Any idea? :)

Best regards, Niko

[janschaefer]

A simple solution would be to add an index to the rows and add a custom error message to your assertion message. Then at least you should be able to correlate the error with the row.

[nikowitt]

Oh yes, you are right :)

[nikowitt]

Interesting, adding "numberedRows"=true throws an exception in my case:

java.lang.UnsupportedOperationException
    at java.util.AbstractList.add(Unknown Source)
    at com.tngtech.jgiven.report.model.DataTable.addColumn(DataTable.java:85)
    at com.tngtech.jgiven.report.model.StepFormatter.addNumberedRows(StepFormatter.java:206)
    at com.tngtech.jgiven.report.model.StepFormatter.toTableValue(StepFormatter.java:189)
    at com.tngtech.jgiven.report.model.StepFormatter.getRemainingArguments(StepFormatter.java:179)
    at com.tngtech.jgiven.report.model.StepFormatter.buildFormattedWordsInternal(StepFormatter.java:145)
    at com.tngtech.jgiven.report.model.StepFormatter.buildFormattedWords(StepFormatter.java:98)
    at com.tngtech.jgiven.impl.ScenarioModelBuilder.createStepModel(ScenarioModelBuilder.java:102)
    at com.tngtech.jgiven.impl.ScenarioModelBuilder.addStepMethod(ScenarioModelBuilder.java:84)
    at com.tngtech.jgiven.impl.ScenarioModelBuilder.stepMethodInvoked(ScenarioModelBuilder.java:215)
    at com.tngtech.jgiven.impl.StandaloneScenarioExecutor$MethodHandler.handleMethod(StandaloneScenarioExecutor.java:127)
    at com.tngtech.jgiven.impl.intercept.StepMethodInterceptor.doIntercept(StepMethodInterceptor.java:57)
    at com.tngtech.jgiven.impl.intercept.StandaloneStepMethodInterceptor.intercept(StandaloneStepMethodInterceptor.java:30)
    at com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest$BOReferenceNumberStage$$EnhancerByCGLIB$$e5a1d644.sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(<generated>)
    at com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest.create_correspondence_with_reference_number_category(BOReferenceNumberTest.java:83)
(...)org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:174)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

This is probably related to my Object[][]?

[janschaefer]

Oh. That can be regarded as a defect :-(

[nikowitt]

I've encountered another issue: I'm using assertions that are not thrown and cannot be intercepted properly in the loop. My idea then is to set a flag on the Object[][](just add another column), but it seems that changes on the object that are done in the method itself are not reflected when the object is formatted. Is this a technical limitation for some reason?

[janschaefer]

That is a limitation, because the table is copied internally when the step is executed. I think this is not possible to change.

[nikowitt]

Oh, what a pity. Then I have to check how to catch the assertions properly.

[janschaefer]

A possible solution would be that you print the whole table again when an assertion fails. Then you can mark the corresponding line.

[nikowitt]

One more question related to that topic: in a test step, I sometimes call further test steps that also could be invoked directly via given(), when(), then(). Is my assumption correct that in these cases, exceptions are not thrown, but the interceptor instead stops the execution directly?

[janschaefer]

There should actually be no difference how you invoke the steps. In any case the exception is first captured by the interceptor, then the following steps are executed in a special skip mode and at the end the captured exception is thrown.

[nikowitt]

Let me explain my issue with more details:

In addition to the intial post:

public void sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(
                @Table(numberedRows = true, columnTitles = { "sender code",
                        "recipient code",
                        "reference category",
                        "expected sequence" }) List<Object[]> data)
                throws Exception {

            for (int i = 0; i < data.size(); i++) {
                Object[] o = data.get(i);

                try {
                    reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
                    reference_sequential_number_is((int) o[3]);
                } catch (Throwable e) {
                    throw new TestStageException("Error in row " + i);
                }

            }

        }

This is the original approach that does not work.

Checking more closely:

    public BOReferenceNumberStage reference_sequential_number_is(int i) throws Exception {
            field_$_of_$_$_match(ReferenceNumber.SEQUENTIALCODE, correspondence.getRefNo(), true, is(i));
            return self();
        }

This method is actually only a wrapper for a technical test. The method is also directly usable in a technical then step.

public T field_$_of_$_$_match(@Quoted String field, IDatabaseEntity entity, @IsIsNot boolean matches,
            Matcher<?>... matcher)
            throws Exception {
        Object value = ReflectionUtils.getRelatedObjectByFieldName(entity, field);

        for (Matcher<?> single : matcher) {
            Matcher<Object> m = (Matcher<Object>) (matches ? single : not(single));
            LOGGER.debug("MATCHER={} on object {}", m, value);
            assertThat(value, m);

        }

        return self();
    }

The assertThat failure is not thrown, but intercepted as explained by you. When I directly replace the technical matcher with the assert, it works as expected:

public void sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(...)
(...)
        try {
                    reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
                    assertThat(correspondence.getRefNo().getSequentialCode(), is((int) o[3]));
                } catch (Throwable e) {
                    throw new TestStageException("Error in row " + i);
                }

In collection handling checks, this also causes issues when I want to catch/collect failures from another step method. So at least for me the default approach of intercepting all exceptions is not in some corner cases.

Maybe some mechanism to tell the interceptor to further throw an exception instead of intercepting it can be introduced. Maybe some global flag can be applied to pass exceptions when I'm in a batch mode? E.g. something like

try {
                    reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
                    deactivateInterception();
                    reference_sequential_number_is((int) o[3]);
                    activateInterception();
                } catch (Throwable e) {
                    throw new TestStageException("Error in row " + i,e);
                }

[janschaefer]

I have to check the code, but as far as I know, exceptions should only be captured on the outer calls of step methods and not on the inner ones. If this is not the case, this can be easily fixed. Which also would totally make sense.