pact-foundation / pact-reference

Reference implementations for the pact specifications
https://pact.io
MIT License
91 stars 46 forks source link

Verification results across multiple pacts accumulate, publishing invalid results #231

Closed mefellows closed 1 year ago

mefellows commented 1 year ago

If a Pact provider verification sources multiple pact files from a broker to verify, the test results accumulate for each pass, sending additional detail along with the published results. Crucially, if any failed interactions happen all subsequent verification publishes will also have an overall status of failed.

Given

And

verification fails for X on interaction A

Then

Two results will publish to the broker, with the following statuses:

Pact file Overall status A B C D E F
X - - -
Y

Note that despite the three interactions in Y that passed, the overall status is failed because of the accumulated result from the first pact file.