Allow Postman contract-testing support

[lbroudoux]

Reason/Context

As of today (version 0.1.4 of this library) we just support essential features of Microcks provided by the main Microcks container. The list of supported features is the following:

• Mocking of REST APIs using different kinds of artifacts,
• Contract-testing of REST APIs using OPEN_API_SCHEMA runner/strategy,
• Mocking and contract-testing of SOAP WebServices,
• Mocking and contract-testing of GraphQL APIs,
• Mocking and contract-testing of gRPC APIs.

POSTMAN contract-testing strategy is not supported at the moment as it requires an additional microcks-postman-runner container that must be linked to the main container.

This feature is required by some users (see #21) and would allow to implement Different levels of API contract testing in the Inner Loop with Testcontainers!

Description

We'd like to implement some kind of wrapper that allows starting/stopping the 2 containers together: the main MicrocksContainer but also the Postman runner container (a GenericContainer should be enough for that as we don't have to provide specific configuration methods).

Ideally, this should be transparent from the user point of view: just start/configure/stop an ensemble of Microcks related containers in a single operation (no manual configuration of each and every containers).

Implementation ideas

Different options to consider:

• Use Docker-compose based test containers (by embedding a docker-compose file in our lib?)
• Use LinkableContainer and reuse the same shared network
• Create a simple Java wrapper but it must respond to Tetscontainers lifecycle events

Some additional requirements:

• Managed containers should be configurable to access the host network
• MicrocksContainer operations should remain accessible
• Images names and versions must be configurable to a certain degree (notion of platform version to introduce here?)

[lbroudoux]

In order to support multi-containers in Microcks, I created a new base class named a MicrocksContainersEnsemble. An ensemble allows you to configure and control the multiple containers needed for Microcks advanced fetaures.

So instead of creating a single MicrocksContainer, you now have to create a MicrocksContainersEnsemble like this:

MicrocksContainersEnsemble ensemble = new MicrocksContainersEnsemble("quay.io/microcks/microcks-uber:1.8.0");

An ensemble conforms to Testcontainers lifecycle method, so you can start it like this:

ensemble.start();

and it will take care of starting multiple containers (in our case, Microcks main one + the microcks-postman-runtime) and link them together.

A MicrocksContainer is wrapped by an ensemble and is still available to import artifacts and execute test methods. You have to access it using:

MicrocksContainer microcks = ensemble.getMicrocksContainer();
microcks.importAsMainArtifact(...);

You can check the testPostmanContractTestingFunctionality() test to check how to use it.

[ericwyles]

Hi @lbroudoux .. I have some initial feedback. Here's where I'm at...

My own sample app is failing using the Ensemble along with the POSTMAN runner. I suspect it is my own mistake because this is my first time making a POSTMAN collection in order to use in Microcks in this way, so I have probably messed something up. But the errors that return from in the TestResult from ensemble.getMicrocksContainer().testEndpoint(testRequest) are pretty opaque and hard to tell what is going on. I can see success = false and some basics but other than that I can't tell what is going wrong.

So to narrow it down, I applied the new version with the Ensemble and the POSTMAN runner in the microcks/api-lifecycle/shift-left-demo/spring-boot-order-service app in the OrderControllerContractTests

It fails with this output:

{
  "id" : "6531938178011e5620564b20",
  "version" : 1,
  "testNumber" : 1,
  "testDate" : 1697747841393,
  "testedEndpoint" : "http://host.testcontainers.internal:56851/api",
  "serviceId" : "6531936f78011e5620564b07",
  "timeout" : 5000,
  "elapsedTime" : -1,
  "success" : false,
  "inProgress" : true,
  "runnerType" : "POSTMAN",
  "testCaseResults" : [ {
    "success" : false,
    "elapsedTime" : -1,
    "operationName" : "POST /orders",
    "testStepResults" : [ ]
  } ]
}

If I put in a breakpoint and watch the logs of the microcks-uber container before it goes away, I can see the problem is this:

20:37:22.747 ERROR 1 --- [    task-4] i.g.m.service.TestRunnerService          : Throwable while testing operation POST /orders

java.io.IOException: Postman collection file cannot be found or parsed
    at io.github.microcks.util.postman.PostmanTestStepsRunner.runTest(PostmanTestStepsRunner.java:120)
    at io.github.microcks.util.postman.PostmanTestStepsRunner.runTest(PostmanTestStepsRunner.java:58)

But without catching the docker container logs I'm not sure how to determine that.

So I guess 2 issues: 1) I think the test setup in the spring-boot-order-service may have some issue that isn't completely configured to run correctly with the postman runner. 2) It seems difficult to tell the specific result when something like this goes wrong using the ensemble.getMicrocksContainer().testEndpoint(testRequest) -- Is there something I'm missing on how to get better output here without catching the transient docker container logs?

I still have some issue with my own application that I haven't configured correctly, but I backed off to get the sample working to eliminate variables first. I should get back to my own app most likely next week (I'm ooo tomorrow).

Thanks!

[ericwyles]

If I'm still stuck on my application after I resolve the sample app, I'll push a demo if that helps.

[lbroudoux]

Hey Eric!

Thanks for your test. You actually need an extra setup step to get this working on the spring-boot-order-service: a valid Postman Collection.

I've written such a collection, and here it is attached: order-service-postman-collection.json

In this collection, you'll find a single test script that allows the Postman runner to evaluate "behavioral or business conformance" rules for your API - remember that the pure "syntactic conformance" is already managed at the OPEN_API_SCHEMA level (and checked when you're using the OpenAPI test runner).

So imagine I have this business rule saying that "on successful order creation, the order should contain exactly the same products and quantities as in the incoming request". I can write a Postman script like this one to check these expectations:

var requestProductQuantities = JSON.parse(pm.request.body.raw).productQuantities;

if (pm.response.code === 201) {
    pm.test("Correct products and quantities in order", function () {
        var order = pm.response.json();
        var productQuantities = order.productQuantities
        pm.expect(productQuantities).to.be.an("array");
        pm.expect(productQuantities.length).to.eql(requestProductQuantities.length);
        for (let i=0; i<requestProductQuantities.length; i++) {
            var productQuantity = productQuantities[i];
            var requestProductQuantity = requestProductQuantities[i];
            pm.expect(productQuantity.productName).to.eql(requestProductQuantity.productName)
            pm.expect(productQuantity.quantity).to.eql(requestProductQuantity.quantity)
        }
    });
}

To finalize the setup and execute the test, you need:

• To put the above order-service-postman-collection.json collection into the src/main/resources folder of the Spring app,
• Add an extra microcksEnsemble.getMicrocksContainer().importAsSecondaryArtifact(new File("target/classes/order-service-postman-collection.json")); directive in the setup() method of your test, after having imported the main artifact
• Execute this snippet into your JUnit test:

// Ask for a Postman Collection script conformance to be launched.
TestRequest testRequest = new TestRequest.Builder()
        .serviceId("Order Service API:0.1.0")
        .runnerType(TestRunnerType.POSTMAN.name())
        .testEndpoint("http://host.testcontainers.internal:" + port + "/api")
        .build();

TestResult testResult = microcksEnsemble.getMicrocksContainer().testEndpoint(testRequest);

That way, you should get the following TestResult expressed as JSON:

{
  "id" : "653276461335696f78a996ae",
  "version" : 2,
  "testNumber" : 1,
  "testDate" : 1697805894759,
  "testedEndpoint" : "http://host.testcontainers.internal:54302/api",
  "serviceId" : "653276451335696f78a99694",
  "timeout" : 5000,
  "elapsedTime" : 229,
  "success" : true,
  "inProgress" : false,
  "runnerType" : "POSTMAN",
  "testCaseResults" : [ {
    "success" : true,
    "elapsedTime" : 229,
    "operationName" : "POST /orders",
    "testStepResults" : [ {
      "success" : true,
      "elapsedTime" : 172,
      "requestName" : "invalid_order"
    }, {
      "success" : true,
      "elapsedTime" : 57,
      "requestName" : "valid_order"
    } ]
  } ]
}

You now have the 2 requests coming from the OpenAPI file (valid_order and invalid_order) being executed by the Postman runner and evaluated against the script defined in the collection.

NB 1: to have the same test result, you'll have to update your local quay.io/microcks/microcks-postman-runtime:latest as I have found a minor bug while testing (see #28)

NB 2: I agree that troubleshooting can be complex without catching the logs of containers (BTW this can be done using the microcksEnsemble.getMicrocksContainer().getLogs(). I have also added the microcksEnsemble.getPostmanContainer().getLogs() method to get the same thing on Postman runtime). I think we'll have to evolve the test creation process to do extra verification checks (like Postman collection presence) and return more diagnostic messages...

Thanks again for your tests!

[ericwyles]

Hey Laurent... I am able to test the order service successfully with your postman collection (thanks for that) and I was able to observe the bug in #28 and then the fix by pulling a new image.

I'll move back to my own demo app and also use the getPostmanContainer().getLogs() and try to isolate what my issue is now that I have a working example on the other app from you. Thanks again! I will let you know.

[ericwyles]

@lbroudoux Got everything working on my end now. Between the getPostmanContainer().getLogs() and the fix for #28, it was a lot easier to determine what was going wrong with my own postman collection. It seems to be working ok on my end (just a sample app, so just keep that in mind).

Thanks for the help. I've got everything working together and I can see how the pieces fit together and can work in a shift-left scenario too which is what I was really after.

It's yet to be seen how much we end up using the POSTMAN runner in practice. We already have a pretty extensive set of java integration tests doing behavior testing. So, we may get the most value out of the microcks OPEN_API_SCHEMA test runner and not as much from the POSTMAN one).

[yada]

@ericwyles, this is great and very valuable feedback for the community; thank you. Please keep us posted on your shift-left scenario/usage, and I guess it can become great material for blog posts or talks 😎 We are happy to discuss this further when you are ready 👀

[lbroudoux]

Hey @ericwyles,

Thanks for the feedback. That's excellent news, and I'm going to close this issue as it now works as expected.

TBH, I share your thoughts on POSTMAN runner usage in your case. If you have OpenAPI schema conformance tests + existing tests suite that already test the behavior at the code level, additional Postman collections have low value. Waiting to have the API exposition layer to test the business behavior of a component is lousy practice IMHO.

I think using the Postman runner has great value in 2 cases, though:

• Your QA already invests a lot in Postman scripts - maybe mixing semantic and behavioral checks - and you don't necessarily have OpenAPI or Swagger at hand (or in sync with the implementation)
• You're developing an implementation of an API that is defined by someone else (think of a standard like OpenBanking, BIAN, etc...), and the API owner distributes its test suites as Postman Collection

Happy to discuss your journey to shift-left!