pact-foundation / pact-jvm

JVM version of Pact. Enables consumer driven contract testing, providing a mock service and DSL for the consumer project, and interaction playback and verification for the service provider project.
https://docs.pact.io
Apache License 2.0
1.08k stars 479 forks source link

Unable to publish verification result to broker #1567

Open bwgjoseph opened 2 years ago

bwgjoseph commented 2 years ago

Hi,

I am trying out Pact as an exploration project, and right now, I am facing issue to publish the verification result to the broker using gradle plugin. You can refer to my demo repo for my setup.

Disclaimer: I am still trying to learn Pact so I might be wrong in certain understanding, so please forgive me if I explain it wrongly.


In summary

My only problem is that I can't publish the result to the broker via ./gradlew pactVerify command

My provider setup is as such

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Provider("ProfileProvider")
@PactBroker
public class PactProfileConsumerVerificationTest {
    @LocalServerPort
    private int port;

    @BeforeEach
    void setup(PactVerificationContext context) {
        context.setTarget(new HttpTestTarget("localhost", port));
    }

    @TestTemplate
    @ExtendWith(PactVerificationSpringProvider.class)
    void pactVerificationTestTemplate(PactVerificationContext context) {
        context.verifyInteraction();
    }

    // omitted
}

My application properties

server.port=9090

pactbroker.host: localhost
pactbroker.port: 9292
pactbroker.auth.username: pact
pactbroker.auth.password: pact

When running the test (./gradlew -i clean test)

This is the logs (partial)

2022-06-04 11:45:10.798  INFO 20092 --- [    Test worker] .b.p.PactProfileConsumerVerificationTest : Started PactProfileConsumerVerificationTest in 4.372 seconds (JVM running for 6.008)   

PactProfileConsumerVerificationTest > pactVerificationTestTemplate(PactVerificationContext) > com.bwgjoseph.pactprovider.PactProfileConsumerVerificationTest.pactVerificationTestTemplate(PactVerificationContext)[1] STANDARD_OUT
    2022-06-04 11:45:12.253  INFO 20092 --- [    Test worker] p.j.PactVerificationStateChangeExtension : Invoking state change method 'profiles 1 exists':SETUP

    Verifying a pact between ProfileConsumer (0.0.1-SNAPSHOT) and ProfileProvider

      Notices:
        1) The pact at http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961 is being verified because the pact content belongs to the consumer version matching the following criterion:
        * latest version of ProfileConsumer from the main branch 'main' (0.0.1-SNAPSHOT)

      [from Pact Broker http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961/metadata/c1tdW2N2XT05]
      Given profiles 1 exists
      get profile with id 1
    2022-06-04 11:45:12.354  WARN 20092 --- [       Thread-4] au.com.dius.pact.core.support.Metrics    :
                Please note: we are tracking events anonymously to gather important usage statistics like JVM version
                and operating system. To disable tracking, set the 'pact_do_not_track' system property or environment
                variable to 'true'.

    2022-06-04 11:45:12.450  INFO 20092 --- [o-auto-1-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
    2022-06-04 11:45:12.450  INFO 20092 --- [o-auto-1-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
    2022-06-04 11:45:12.452  INFO 20092 --- [o-auto-1-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
        returns a response which
          has status code 200 (OK)
          has a matching body (OK)
    2022-06-04 11:45:12.743  WARN 20092 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Not all of the 2 were verified. The following were missing:
    2022-06-04 11:45:12.744  WARN 20092 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   :     get all profiles

PactProfileConsumerVerificationTest > pactVerificationTestTemplate(PactVerificationContext) > com.bwgjoseph.pactprovider.PactProfileConsumerVerificationTest.pactVerificationTestTemplate(PactVerificationContext)[2] STANDARD_OUT
    2022-06-04 11:45:12.775  INFO 20092 --- [    Test worker] p.j.PactVerificationStateChangeExtension : Invoking state change method 'profiles exists':SETUP

    Verifying a pact between ProfileConsumer (0.0.1-SNAPSHOT) and ProfileProvider

      Notices:
        1) The pact at http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961 is being verified because the pact content belongs to the consumer version matching the following criterion:
        * latest version of ProfileConsumer from the main branch 'main' (0.0.1-SNAPSHOT)

      [from Pact Broker http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961/metadata/c1tdW2N2XT05]
      Given profiles exists
      get all profiles
        returns a response which
          has status code 200 (OK)
          has a matching body (OK)
    2022-06-04 11:45:12.881  WARN 20092 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Skipping publishing of verification results as it has been disabled (pact.verifier.publishResults is not 'true')

PactProviderApplicationTests STANDARD_OUT
    2022-06-04 11:45:12.913  INFO 20092 --- [    Test worker] .b.t.c.SpringBootTestContextBootstrapper : Neither @ContextConfiguration nor @ContextHierarchy found for test class [com.bwgjoseph.pactprovider.PactProviderApplicationTests], using SpringBootContextLoader

For some reason, it states Skipping publishing of verification results as it has been disabled but I have it configured to pact.verifier.publishResults=true in gradle.properties

Well, I tried to pass the properties via gradle but encounter errors

./gradlew -i clean test -Ppact.verifier.publishResults=true
* What went wrong:
Your project is misconfigured, was expecting a 'pact' configuration in the build, but got a String with value '' instead. Make sure there is no property that is overriding 'pact'.

I checked, and the closest seem to be this issue 738 but even after setting the protocol, project.version, it still doesn't work out

But if I were to run the following command

// pact.verifier.publishResults=true is defined in gradle.properties
./gradlew clean pactVerify

The test would fail with the following output

> Task :pactVerify_ProfileProvider FAILED
Caching disabled for task ':pactVerify_ProfileProvider' because:
  Build cache is disabled
Task ':pactVerify_ProfileProvider' is not up-to-date because:
  Task has not declared any outputs despite executing actions.

Verifying a pact between ProfileConsumer (0.0.1-SNAPSHOT) and ProfileProvider

  Notices:
    1) The pact at http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961 is being verified because the pact content belongs to the consumer version matching the following criterion:
    * latest version tagged 'main' (0.0.1-SNAPSHOT)

  [from Pact Broker http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961/metadata/c1tdW3RdPW1haW4mc1tdW2xdPXRydWUmc1tdW2N2XT05]
  Given profiles 1 exists
         WARNING: State Change ignored as there is no stateChange URL
  get profile with id 1
      Request Failed - Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information
  Given profiles exists
         WARNING: State Change ignored as there is no stateChange URL
  get all profiles
      Request Failed - Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information
Published verification result of 'Failed(results=[{interactionId=345959413de4179e15ce5e6f46f3f8e3716d222e, exception=org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information, description=Request to provider endpoint failed with an exception}, {interactionId=1f183cf4873b5acf3ad4d4e91bc521c899bc24b2, exception=org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information, description=Request to provider endpoint failed with an exception}], description=Request to provider endpoint failed with an exception)' for consumer 'Consumer(name=ProfileConsumer)'

Failures:

1) Verifying a pact between Pact between ProfileConsumer (0.0.1-SNAPSHOT) and ProfileProvider - get profile with id 1 Given profiles 1 exists

    1.1) Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information

    1.2) Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information

:pactVerify_ProfileProvider (Thread[included builds,5,main]) completed. Took 0.101 secs.

FAILURE: Build failed with an exception.

However, the failure result would still be published to the broker with the error like such

image

I did come across this question in SO and have tried to configure via system property but it doesn't work out as expected as well.

I'm not quite sure what is causing the issue, do let me know if there's something I did wrong, or if I need to provide more information.

Thanks!

rholshausen commented 2 years ago

You are using a JUnit test, so you don't need the Pact Gradle plugin command, the normal Gradle test will do.

The problem is that Gradle will run your tests in a separate JVM process, so the properties from the command line won't be applied. See https://stackoverflow.com/questions/21406265/how-to-give-system-property-to-my-test-via-gradle-and-d how to set the properties correctly for the test.

bwgjoseph commented 2 years ago

Hey @rholshausen, thanks for getting back.

You are using a JUnit test, so you don't need the Pact Gradle plugin command, the normal Gradle test will do.

If I get you correctly, you meant that as long as I ran ./gradlew test, it should automatically upload the verification to pact server?

I tried a couple of ways, but still unable to get it uploaded.


  1. Specify System.setProperty manually under test task
tasks.named('test') {
    useJUnitPlatform()

    System.setProperty("pact.verifier.publishResults", "true");
}

And I ran ./gradlew clean test. But nothing gets uploaded, although the report are generated.


  1. As per SO link you mentioned
tasks.named('test') {
    useJUnitPlatform()

    systemProperty "pact.verifier.publishResults", project.getProperty("pact.verifier.publishResults")
}

And I ran ./gradlew clean test -Ppact.verifier.publishResults=true, I only face with this error

* What went wrong:
Your project is misconfigured, was expecting a 'pact' configuration in the build, but got a String with value '' instead. Make sure there is no property that is overriding 'pact'

Also tried ./gradlew pactVerify which ran successfully but nothing happens. Then I tried ./gradlew pactVerify -Ppact.verifier.publishResults=true which faced the same error of was expecting a 'pact' configuration in the build...

Not exactly sure where was configured wrongly, do enlighten. Thank you

rholshausen commented 2 years ago

Your project is misconfigured, was expecting a 'pact' configuration in the build

Option 2 is the correct way, don't know why you are getting that error. Try removing the Pact Gradle plugin (you don't need it), or just add an empty pact block.

bwgjoseph commented 2 years ago

I made a newbie mistake earlier, and was running the command from the consumer project.

Now that I have switched to the right project, and here's what I tried


  1. Same as 1 above, but I removed gradle plugin, same result as before

  1. Same as 2 above, removed gradle plugin, ran ./gradlew clean test -Ppact.verifier.publishResults=true but encounter
* What went wrong:
Task '.verifier.publishResults=true' not found in root project 'pact-provider'

Somehow, pact was stripped off. Wondering if it's a terminal issue, so I switched to use standard command prompt instead of powershell, and this time it works. Any idea?


Also wondering, when would I need to use gradle plugin then if that is not required?

rholshausen commented 2 years ago

Ah, looks like a Gradle doesn't like you using pact in the property name.

BTW, the command line value doesn't have to be the same. Try

tasks.named('test') {
    useJUnitPlatform()

    systemProperty "pact.verifier.publishResults", project.publishResults
}

and then run ./gradlew clean test -PpublishResults=true

rholshausen commented 2 years ago

Even better, something like

tasks.named('test') {
    useJUnitPlatform()

        if (project.hasProperty("publishResults")) {
        systemProperty "pact.verifier.publishResults", project.publishResults
        }
}
rholshausen commented 2 years ago

Here is an example you can look at https://github.com/pactflow/example-provider-springboot

bwgjoseph commented 2 years ago

Thanks! That works if I rely on junit, what if I wanted to use gradle plugin as my original intention via pactVerify? Where if I have it configured as such

tasks.named('test') {
    useJUnitPlatform()
       // nothing set here
}

pact {
    broker {
        pactBrokerUrl = 'http://localhost:9292'

        pactBrokerUsername = 'pact'
        pactBrokerPassword = 'pact'
    }

    serviceProviders {
        ProfileProvider {

            fromPactBroker {
                selectors = latestTags('main')
            }

            providerTags = [project.pactBrokerTag]
        }
    }

    reports {
      defaultReports() // adds the standard console output
    }
}

And I run gradlew pactVerify -Ppact.verifier.publishResults=true in command prompt, powershell still hates -Ppact for some reason.

And I still encounter this error

Verifying a pact between ProfileConsumer (0.0.1-SNAPSHOT) and ProfileProvider

  Notices:
    1) The pact at http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961 is being verified because the pact content belongs to the consumer version matching the following criterion:
    * latest version tagged 'main' (0.0.1-SNAPSHOT)

  [from Pact Broker http://localhost:9292/pacts/provider/ProfileProvider/consumer/ProfileConsumer/pact-version/28b2b6450b198331505dab7cbd7fd53d0b867961/metadata/c1tdW3RdPW1haW4mc1tdW2xdPXRydWUmc1tdW2N2XT05]
  Given profiles 1 exists
         WARNING: State Change ignored as there is no stateChange URL
  get profile with id 1
      Request Failed - Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information
  Given profiles exists
         WARNING: State Change ignored as there is no stateChange URL
  get all profiles
      Request Failed - Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information

Failures:

1) Verifying a pact between Pact between ProfileConsumer (0.0.1-SNAPSHOT) and ProfileProvider - get profile with id 1 Given profiles 1 exists        

    1.1) Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information        

    1.2) Connect to http://localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: no further information        

FAILURE: Build failed with an exception.

* What went wrong:
There were 2 non-pending pact failures for provider ProfileProvider

What is causing this to fail?

Here is an example you can look at https://github.com/pactflow/example-provider-springboot

I've seen that example before, but it also brings in gradle plugin but not using it since it declares systemProperty under test task as well. So I'm not entirely sure the difference between using and not using the gradle plugin

rholshausen commented 2 years ago

The Pact Gradle plugin provides the pactVerify task, you only need it if you want to run that. But there is no harm in having it in your project.

I don't know much about Powershell, I've never used it.

If you want to use gradlew pactVerify, you need to have your provider running before. See https://github.com/pact-foundation/pact-jvm/tree/master/provider/gradle#starting-and-shutting-down-your-provider

bwgjoseph commented 2 years ago

It states that Gradle plugin for verifying pacts against a provider but I'm actually trying to use my provider to verify against the consumer contracts. I'm actually trying to do this - https://github.com/pact-foundation/pact-jvm/tree/master/provider/gradle#verifying-pact-files-from-a-pact-broker

Why would I need to start up or shut-down the provider? I'm already running as the provider?

rajnavakotiikea commented 2 years ago

@rholshausen I have the similar issue with Maven. I want to use the plugin for 'verify_changed_pacts' workflow.

We are planning to have to 2 work flows for the provider (similar to https://github.com/pactflow/example-provider-springboot)

  1. Build : mvn clean test - executes all unit tests (This will also run the provider contract tests)
  2. Verify_changed_pact : mvn pact:verify -Dpact.filter.consumers=foo -Dpact.filter.pacturl=bar - execute only those tests that impacted by changed pact specified in the webhook

As per your suggestion, we don't need to use plugin we can run mvn clean test -Dpact.filter.consumers-foo -Dpact.filter.pacturl=bar correct? But this will run all the unit tests rit ? not just tests related to changed pact specified in the webhook. Please correct me if i'm wrong.

I can further refine the mvn clean test command to look into the contract test folder in the src but it will still run all the contract tests in the folder rit not just tests related to changed pact?