Failed assertions in async containers do not fail the test (Citrus 3)

[bthdimension]

Citrus Version 3.0.0

Expected behavior When a validation within an async container fails, I expect the whole test to fail.

Actual behavior I can see the failures in the log, however, the main thread doesn't seem to be bothered with the failures and completes successfully.

Test case sample

@Test
@CitrusTest
public void test() {
    run(async().actions(
        applyBehavior(new ServerMockBehavior())
    ));
    // Execute a function in a bean I'm testing (calls the mocked server via RestTemplate)
    myBean.execute(execution);
}
---
public class ServerMockBehavior implements TestBehavior {
    @Override
    public void apply(TestActionRunner r) {
            r.run(http().server("myService")
                    .receive().post().path("/some/path").message().type(MessageType.JSON)
                    .body(new ClassPathResource("data/request.json"))
            );
            r.run(http().server("myService")
                    .send().response().message().status(HttpStatus.NO_CONTENT)
            );
    }
}

Workaround for the sample above I'm currently just catching the validation exceptions in the receive-Action and let my mocked servers respond with a 500 error, so the application in the main thread fails.

[christophd]

when the receive action fails the send is not executed. The server then uses the fallback response that is sent after configured timeout. By default the fallback response is a 200 OK. You can adjust the default fallback and send a 500 error instead.

[bthdimension]

Hi Christoph, thank you for your reply. This sounds similar to my own workaround but doesn't solve the problem: When the receive action fails because of a validation error, then the test should fail. At the moment, it doesn't, which is not the expected behavior (in my opinion). Or is that by design?

[christophd]

I am confused how your workaround should fix the issue then. How does the 500 error response cause the test to fail?

Another question is why do you need the async here? I can remember that there has been an issue with tests finishing before the async action is done. This would explain the test reporting success instead of failure. I would need some more information on your test setup to understand and reproduce on my side.

[bthdimension]

How does the 500 error response cause the test to fail?

When the Citrus Mock Server sends a 500 to my tested application, the tested application throws an error and causes the test to fail.

Another question is why do you need the async here?

That's because I'm directly and synchronously calling the Java API of my application instead of using a REST Client with .fork(true). My application then synchronously calls the mock server and waits for the response. So the mock sever needs to already be waiting asynchronously in another thread.

I can remember that there has been an issue with tests finishing before the async action is done. This would explain the test reporting success instead of failure.

That's probably the case here.

[bthdimension]

Hi @christophd , we're now on Citrus 3.4.0 and we still have this problem. The test is green even though in the async container, the request could not be validated and a validation error is logged.

We've improved our workaround by setting .defaultStatus(HttpStatus.INTERNAL_SERVER_ERROR) for the HttpServer-Bean. But that still just means that the tested application needs to fail in order for the test to fail. If our application ignores or handles errors from the servers it calls, then all our tests are green. This renders async containers kinda useless but they're required in some of our test cases.

I guess, the problem is that async containers run in a different thread and when an assertion fails there, it doesn't affect the main thread. Could that be the case?