search

pact-foundation / pact-jvm

JVM version of Pact. Enables consumer driven contract testing, providing a mock service and DSL for the consumer project, and interaction playback and verification for the service provider project.
https://docs.pact.io
Apache License 2.0
1.08k stars 479 forks source link

When publishing verification results, only the errors are sent #1266

Closed uglyog closed 2 years ago

uglyog commented 3 years ago

When publishing verification results, only the errors are being sent. This causes the Pact broker to see the passing interactions as un-verified.

For a test with 3 interactions, if two pass and one fails, the following entries are logged:

Received test result 'au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160' for Pact pactflow-example-provider-springboot-pactflow-example-consumer and a request to get a product (Pact Broker https://testdemo.pactflow.io/pacts/provider/pactflow-example-provider-springboot/consumer/pactflow-example-consumer/pact-version/4a6c2017e4f913c5b18ee0b526eae842567bf489)
2020-12-08 12:27:12.818 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Number of interactions #3 and results: [au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160]
2020-12-08 12:27:12.818  WARN 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Not all of the 3 were verified. The following were missing:
2020-12-08 12:27:12.818  WARN 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   :     a request to get a product
2020-12-08 12:27:12.818  WARN 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   :     a request to get all products
2020-12-08 12:27:13.029 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Received test result 'au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160' for Pact pactflow-example-provider-springboot-pactflow-example-consumer and a request to get all products (Pact Broker https://testdemo.pactflow.io/pacts/provider/pactflow-example-provider-springboot/consumer/pactflow-example-consumer/pact-version/4a6c2017e4f913c5b18ee0b526eae842567bf489)
2020-12-08 12:27:13.030 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Number of interactions #3 and results: [au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160, Failed(results=[{interactionId=ba08b56e3fecb8a49c7f7a2d509d9fe3183a44cf, attribute=status, description=expected status of 404 but was 200}], description=Response status did not match), au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160]
2020-12-08 12:27:13.030 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : All interactions for Pact pactflow-example-provider-springboot-pactflow-example-consumer have a verification result

Then when the results are published:

2020-12-08 12:27:13.447 DEBUG 33876 --- [    Test worker] a.c.d.p.c.pactbroker.PactBrokerClient    : Test result = Failed(results=[{interactionId=ba08b56e3fecb8a49c7f7a2d509d9fe3183a44cf, attribute=status, description=expected status of 404 but was 200}], description=Response status did not match)
2020-12-08 12:27:13.448 DEBUG 33876 --- [    Test worker] a.c.dius.pact.core.pactbroker.HalClient  : Posting JSON to https://testdemo.pactflow.io/pacts/provider/pactflow-example-provider-springboot/consumer/pactflow-example-consumer/pact-version/4a6c2017e4f913c5b18ee0b526eae842567bf489/verification-results
{"providerApplicationVersion":"cdbbf3f+1607390821","success":false,"testResults":[{"interactionId":"ba08b56e3fecb8a49c7f7a2d509d9fe3183a44cf","mismatches":[{"attribute":"status","description":"expected status of 404 but was 200"}],"success":false}]}

Only the error is sent.

anto-ac commented 3 years ago

On which version did this start to happen?

uglyog commented 3 years ago

Not sure, but it has been reported with 4.0.10 and 4.1.8.

uglyog commented 3 years ago

4.1.12 released

PhilHardwick commented 3 years ago

Hi, thanks for fixing this ^ I've just opened https://github.com/pact-foundation/pact-jvm/pull/1276 to fix a small part of the refactor which came from this issue.

Rafaesp commented 3 years ago

@uglyog I'm still seeing something related to this in 4.1.14. I have a case where a pending pact is failing and another pact tagged with master is passing. It seems that what gets published is the following:

Published verification result of 'Failed(results=[{exception=java.lang.RuntimeException: Failed to invoke provider method 'updateTest', description=Request to provider method failed with an exception, interactionId=d7b7da1207e7db9dd47db7f01a1334baef01c303}, {interactionId=443bb987c3f94c905c58a38769407ed00725add1}, {interactionId=434ed5a5dbd99c9625d6424af897cc6bc26d32e8}, {interactionId=380cb209e8d00e36a330e77133b5091b30ef76a5}], description=Request to provider method failed with an exception)' for consumer 'Consumer(name=api)

Which in pact broker marks all interactions as failed even when all individual tests for the interaction marked as master are marked green and passing. This leads to can-i-deploy failing the build even when master pacts are passing.

When I use pact 4.1.6, the results are published one by one and I don't hit this problem.

Rafaesp commented 3 years ago

https://github.com/pact-foundation/pact-jvm/issues/1266#issuecomment-762383209 More logs of the issue above:

With Pact 4.1.14: https://pastebin.com/cTqBdx4V With Pact 4.1.6: https://pastebin.com/94m9g70D

mefellows commented 3 years ago

@Rafaesp the pastebin logs appear to have been removed, are you able to please re-add them?

Rafaesp commented 3 years ago

@mefellows Done! With Pact 4.1.14: https://pastebin.com/cTqBdx4V With Pact 4.1.6: https://pastebin.com/94m9g70D Thank you

uglyog commented 3 years ago

@Rafaesp are you using JUnit 4 or 5?

Rafaesp commented 3 years ago

JUnit5

uglyog commented 3 years ago

@Rafaesp I've released 4.1.19 which hopefully resolves your issue

Rafaesp commented 3 years ago

@uglyog Thank you very much. We have upgraded to the latest and made a few tests. The issue is hard to reproduce, the current status of our pacts is stable and we don't have new contracts so we can't be 100% sure it was fixed. However, we can still see the following behaviour in the Pact Broker that I expected 4.1.19 to have fixed:

As you can see, Pact Broker doesn't know about the status of the interactions and I'm concerned this could lead to the same problem we were seeing when we had new contracts and pending pacts. Did you expect Pact Broker to show a green badge for successful pact interactions?

uglyog commented 3 years ago

Unknown status means the broker did not receive a result for that interaction.

Rafaesp commented 3 years ago

@uglyog I can confirm that's still happening with 4.1.19 😞

Rafaesp commented 3 years ago

@uglyog

Unknown status means the broker did not receive a result for that interaction.

Does that make sense? The broker has marked the pact as verified (it's shown in the screenshot) and the logs show that the verifications have been published. But still, all individual interactions are marked as unknown.

@bethesque @mefellows Did you expect that? All interactions in our Pactflow instance (https://imanage.pact.dius.com.au/) are marked as unknown. However, can-i-deploy works as expected but this is very concerning :(

mefellows commented 3 years ago

@bethesque @mefellows Did you expect that? All interactions in our Pactflow instance (https://imanage.pact.dius.com.au/) are marked as unknown. However, can-i-deploy works as expected but this is very concerning :(

I can't speak for the implementation details of Pact JVM. I'm hopeful that perhaps there is a bug or detail I'm missing about why interaction level success/fail is not sent through, but I would assume the overall verification cannot be "successful" if the requests are not satisfied.

This feedback aside, are you seeing any evidence of the pact verification actually failing but still passing can-i-deploy?

Rafaesp commented 3 years ago

@mefellows When verifications fail, can-i-deploy also fails. That works as expected.

uglyog commented 3 years ago

This is actually a Pactflow issue. The verification results have a status for the verification, and then a status for each interaction. The UI is displaying an unknown result where the overall verification is a success, which does not make sense to me. If the overall verification is a success, all the interactions have to be successful, regardless if they have an individual status or not.

bethesque commented 3 years ago

I've made a Pactflow trello card to fix this.

mefellows commented 2 years ago

I believe this is fixed now, are you able to confirm?

Rafaesp commented 2 years ago

I can confirm the verification status is not unknown anymore! thanks!

mefellows commented 2 years ago

Awesome, thanks Rafa!