Ignore or skip pact verification

[wedla]

Hello all, I'm having an issue that I want to share with you.

The scenario: The development team had to add more fields to existing endpoints of an API, but because this breaks the contract between consumers and provider, they used the concept of feature flag. So I'm in charge of executing the tests with the flag turned on and off, but I'm having some issues. I read about pending pacts, but because the QA team is so small, I'm having trouble to develop it by myself applying Pact Broker. Nowadays, we are using a maven project, with a father pom and consumer and provider poms as its children. When the contract files are generated, they are copied to a source in the provider project folder, and the tests run without problems.

I was studying and found the concept of conditional tests in JUnit, especifically this one and I was trying to put this in the @Test methods, but the problem lies in the @PactVerification annotation, because when the assumeTrue from JUnit returns false or vice-versa, JUnit doesn't skip the test, it fails. I found out it was the annotation because when I commented the code line, JUnit was able to skip the test and run the others.

It's like Pact works, in this case, as a intermediator, and it throws the exception when it should just skip it. I want to know from you if it's possible to ignore some tests from running, without using the pending pacts feature, in an elegant way, because I don't want to put if and elses to verify the flag.

Thank you very much for all the attention :)

Some informations:

<groupId>au.com.dius.pact.consumer</groupId>
<artifactId>junit</artifactId>
<version>4.1.18</version>

The error Pact throws: java.lang.AssertionError: Pact Test function failed with an exception, possibly due to ExpectedButNotReceived(expectedRequests=[ method: POST path: * query: {} headers: {Authorization=[Bearer token], content-type=[application/json]} matchers: MatchingRules(rules={}) generators: Generators(categories={}) body: PRESENT({body}) at au.com.dius.pact.consumer.junit.JUnitTestSupport.validateMockServerResult(JUnitTestSupport.kt:55) at au.com.dius.pact.consumer.junit.BaseProviderRule.validateResult(BaseProviderRule.java:186) at au.com.dius.pact.consumer.junit.BaseProviderRule$1.evaluate(BaseProviderRule.java:88) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:542) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:770) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:464) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210) Caused by: org.junit.AssumptionViolatedException: got: , expected: is at org.junit.Assume.assumeThat(Assume.java:95) at org.junit.Assume.assumeTrue(Assume.java:41) at com.ipp.sinapse.sales.test.TransmissaoVendaAmPmConsumidorTest.testeSuccesso(TransmissaoVendaAmPmConsumidorTest.java:105) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19) at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239) at au.com.dius.pact.consumer.junit.BaseProviderRule.lambda$runPactTest$2(BaseProviderRule.java:171) at au.com.dius.pact.consumer.BaseMockServer.runAndWritePact(MockHttpServer.kt:126) at au.com.dius.pact.consumer.ConsumerPactRunnerKt.runConsumerTest(ConsumerPactRunner.kt:16) at au.com.dius.pact.consumer.junit.BaseProviderRule.runPactTest(BaseProviderRule.java:169) at au.com.dius.pact.consumer.junit.BaseProviderRule.access$100(BaseProviderRule.java:30) at au.com.dius.pact.consumer.junit.BaseProviderRule$1.evaluate(BaseProviderRule.java:87) ... 16 more

[Yazon2006]

It looks more like help request but not an issue of a pact itself. You can discuss it in slack: https://slack.pact.io/

[uglyog]

How the Pact verification works, it checks that all the expected requests have been received. But because you used the assume annotation, the test was not run so the requests were not made to the mock server. Hence the ExpectedButNotReceived error.

JUnit 5 might be better to use for this type of test behaviour.

[wedla]

Hi, @uglyog, thank you so much for your kind response, as always. :) In this case, I used a bad strategy but it works for now (I run the tests by packages, so i have the deprecated one and the main one). I'll have to study and implement Pact Broker because in the near future some features can be added to the endpoints already consumed, and this strategy is not scalable and maintainable.

Now, correct me if I'm wrong, but the JUnit used in this context of the pact consumer is bellow 5, am I right?! so how could I use Junit5 in this context that I use Pact already integrated with JUnit?

[uglyog]

If your strategy works, then it is not bad.

The Pact library you are using is for JUnit 4. There is a JUnit 5 one au.com.dius.pact.consumer:junit5. To use it you will have to convert you test to JUnit 5 and then use that library instead of the JUnit 4 one. There are lots of blogs available online on how to convert from JUnit 4 to Junit 5.