Closed uglyog closed 2 years ago
On which version did this start to happen?
Not sure, but it has been reported with 4.0.10 and 4.1.8.
4.1.12 released
Hi, thanks for fixing this ^ I've just opened https://github.com/pact-foundation/pact-jvm/pull/1276 to fix a small part of the refactor which came from this issue.
@uglyog I'm still seeing something related to this in 4.1.14. I have a case where a pending pact is failing and another pact tagged with master is passing. It seems that what gets published is the following:
Published verification result of 'Failed(results=[{exception=java.lang.RuntimeException: Failed to invoke provider method 'updateTest', description=Request to provider method failed with an exception, interactionId=d7b7da1207e7db9dd47db7f01a1334baef01c303}, {interactionId=443bb987c3f94c905c58a38769407ed00725add1}, {interactionId=434ed5a5dbd99c9625d6424af897cc6bc26d32e8}, {interactionId=380cb209e8d00e36a330e77133b5091b30ef76a5}], description=Request to provider method failed with an exception)' for consumer 'Consumer(name=api)
Which in pact broker marks all interactions as failed even when all individual tests for the interaction marked as master are marked green and passing. This leads to can-i-deploy failing the build even when master pacts are passing.
When I use pact 4.1.6, the results are published one by one and I don't hit this problem.
https://github.com/pact-foundation/pact-jvm/issues/1266#issuecomment-762383209 More logs of the issue above:
With Pact 4.1.14: https://pastebin.com/cTqBdx4V With Pact 4.1.6: https://pastebin.com/94m9g70D
@Rafaesp the pastebin logs appear to have been removed, are you able to please re-add them?
@mefellows Done! With Pact 4.1.14: https://pastebin.com/cTqBdx4V With Pact 4.1.6: https://pastebin.com/94m9g70D Thank you
@Rafaesp are you using JUnit 4 or 5?
JUnit5
@Rafaesp I've released 4.1.19 which hopefully resolves your issue
@uglyog Thank you very much. We have upgraded to the latest and made a few tests. The issue is hard to reproduce, the current status of our pacts is stable and we don't have new contracts so we can't be 100% sure it was fixed. However, we can still see the following behaviour in the Pact Broker that I expected 4.1.19 to have fixed:
When there is a failure and the Pact status is marked as Red:
When the verifications passed (there might be pending pacts that fail) and Pact status is marked as Green:
As you can see, Pact Broker doesn't know about the status of the interactions and I'm concerned this could lead to the same problem we were seeing when we had new contracts and pending pacts. Did you expect Pact Broker to show a green badge for successful pact interactions?
Unknown status means the broker did not receive a result for that interaction.
@uglyog I can confirm that's still happening with 4.1.19 😞
@uglyog
Unknown status means the broker did not receive a result for that interaction.
Does that make sense? The broker has marked the pact as verified (it's shown in the screenshot) and the logs show that the verifications have been published. But still, all individual interactions are marked as unknown.
@bethesque @mefellows Did you expect that? All interactions in our Pactflow instance (https://imanage.pact.dius.com.au/) are marked as unknown. However, can-i-deploy works as expected but this is very concerning :(
@bethesque @mefellows Did you expect that? All interactions in our Pactflow instance (https://imanage.pact.dius.com.au/) are marked as unknown. However, can-i-deploy works as expected but this is very concerning :(
I can't speak for the implementation details of Pact JVM. I'm hopeful that perhaps there is a bug or detail I'm missing about why interaction level success/fail is not sent through, but I would assume the overall verification cannot be "successful" if the requests are not satisfied.
This feedback aside, are you seeing any evidence of the pact verification actually failing but still passing can-i-deploy
?
@mefellows When verifications fail, can-i-deploy also fails. That works as expected.
This is actually a Pactflow issue. The verification results have a status for the verification, and then a status for each interaction. The UI is displaying an unknown result where the overall verification is a success, which does not make sense to me. If the overall verification is a success, all the interactions have to be successful, regardless if they have an individual status or not.
I've made a Pactflow trello card to fix this.
I believe this is fixed now, are you able to confirm?
I can confirm the verification status is not unknown anymore! thanks!
Awesome, thanks Rafa!
When publishing verification results, only the errors are being sent. This causes the Pact broker to see the passing interactions as un-verified.
For a test with 3 interactions, if two pass and one fails, the following entries are logged:
Then when the results are published:
Only the error is sent.