Inconsistent pact manifest generation when using gRPC responseMetadata

[ysatarov]

Hi!

We encountered a strange problem when using with gRPC and responseMetadata. The contents of the pact manifest in these cases depends on the order in which the tests are run.

Below is a slightly modified example of area_calculator test for reproduction: https://github.com/YOU54F/pact-ruby-ffi/commit/78a2a2af17b5567739a46c3d69e6d443c39ccfae

There are two identical tests added, one of them at the beginning and the other at the end of spec-file. The response format in the final pact manifest differs for these two added specs when running specs in random order, which leads to an error publishing contract to pact broker with pact-cli:

Cannot change the content of the pact for SOME-CONSUMER version 2821c525 and provider SOME_PROVIDER, as race conditions will cause unreliable results for can-i-deploy. Each pact must be published with a unique consumer version number. Some Pact libraries generate random data when a concrete value for a type matcher is not specified, and this can cause the contract to mutate - ensure you have given example values for all type matchers. For more information see https://docs.pact.io/go/versioning

Tried protobuf-plugin v0.4 / v0.5.4, pact-ffi v0.4.22 (linux-x86_64 and macos-aarch64), ruby v3.1.2 / v3.3.6

[YOU54F]

Hey @ysatarov.

First off nice work to you and the Kuper-Tech team for your work on sbmt-pact. I've been meaning to catch up for a while.

I took it for a spin recently and started testing it out against some of our existing ruby projects which use pact.

Needed a few tweaks as we don't use rails.

My delta is here. https://github.com/Kuper-Tech/sbmt-pact/compare/master...YOU54F:sbmt-pact:feat/pact-ruby

Anyway back to your question in point.

The order of interactions should not matter. If you do re-order them to the same as a previous publish, does that rectify the publishing issue?

Is that the only delta between runs (interaction order)?

If so, that is determined by the pact-broker on publish.

Here is where the check starts in the pact_publish api resource

https://github.com/pact-foundation/pact_broker/blob/96532124f3a53a499276c69ff2df785b8377588e/lib/pact_broker/api/resources/publish_contracts.rb#L25

which calls into the contract_service

https://github.com/pact-foundation/pact_broker/blob/96532124f3a53a499276c69ff2df785b8377588e/lib/pact_broker/contracts/service.rb#L49-L64

the calculated pact_version will be important, as it will be used to find existing pacts, by that version number.

https://github.com/pact-foundation/pact_broker/blob/96532124f3a53a499276c69ff2df785b8377588e/lib/pact_broker/contracts/service.rb#L131-L142

The Pact Broker isn't aware of the V4 Pact specification, so it may not be able to determine that these pact files are the same.

Maybe that is enough of a cookie trail to take a peek at the broker and cast your Rubyist eyes across it

[ysatarov]

My delta is here

Thanks! It may take a little bit of time, but we'll definitely look into your changes. Feel free to send a pull request if you'd like

Is that the only delta between runs (interaction order)?

Well, not exactly. Maybe it was the screenshot that added some confusion ) The point is the same test example using responseMetadata gives different serialization of response.contents in generated pact manifest.

If it runs first, response.contents would be

...
      "response": [
        {
          "contents": {
            "content": ""
          },
         ...
        }
      ],
...

And if we move the same test example into the end of the spec file and run, response.contents will contain 3 more fields:

...
      "response": [
        {
          "contents": {
            "content": "",
            "contentType": "application/protobuf;message=AreaResponse",
            "contentTypeHint": "BINARY",
            "encoded": "base64"
          },
         ...
        }
      ],
...

[YOU54F]

hmm curious. If the middle test is not there, does the same occur? I wonder if the interactions are not fully cleared in between tests.

[ysatarov]

If the middle test is not there, does the same occur?

if I run only that one specific test, it generates:

...
      "response": [
        {
          "contents": {
            "content": "",
            "contentType": "application/protobuf;message=AreaResponse",
            "contentTypeHint": "BINARY",
            "encoded": "base64"
          },
         ...
        }
      ],
...