search

pact-foundation / pact-ruby

Enables consumer driven contract testing, providing a mock service and DSL for the consumer project, and interaction playback and verification for the service provider project.
https://pact.io
MIT License
2.16k stars 215 forks source link

Cannot publish verification for "" as not all interactions have been verified #221

Closed jmortlock closed 4 years ago

jmortlock commented 4 years ago

Hi

I am running into an issue where I am constantly hitting the following warning when publishing pact verifications

WARN: Cannot publish verification for PetStore as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.

The service provider is a standard ruby application using a setup like the following.

    consumer_version_tags [
      { name: 'production', all: false, fallback: 'master' },
      'master'
    ]

The consumer is a Javascript application and we are using pact-broker to share pacts.

What I think is happening is when the "production" and "master" tag point to the same commit sha# pact only verifies one lot of tags and is then failing to publish with the incomplete warning..

The following shows both pacts are for the same consumer version (I have slightly redacted the log)

INFO: Fetching pacts for PetStore from * for tags: latest production (or master if not found), latest master
INFO: Reading pact at /pacts/provider/PetStore/consumer/PetStore-Web/version/4aeec9b870979a8a8b66108f7382669b85556796
INFO: Reading pact at /pacts/provider/PetStore/consumer/PetStore-Web/version/4aeec9b870979a8a8b66108f7382669b85556796
mefellows commented 4 years ago

Can you please share what environment variables you have? I don't know for certain, but any exported variables like PACT_DESCRIPTION and PACT_PROVIDER_STATE will filter what interactions are validated - and so publishing results won't cover the whole contract

jmortlock commented 4 years ago

We have the following env-vars with PACT in the name

PACT_BROKER_USERNAME
PACT_BROKER_PASSWORD
PACT_BROKER_URL
PACT_BROKER_BASE_URL
PACT_PARTICIPANT
PACT_CLIENT_ID
PACT_CLIENT_SECRET

I left out of the original description the fact that this project also verifies against another consumer(ruby) which does not have the same issue (although master / tag very unlikely to every match), in case this helps at all.

mefellows commented 4 years ago

Awesome thanks, I hope it does! I'll leave it with Beth as she'll have a better idea.

bethesque commented 4 years ago

Can you reproduce the issue by forking and modifying the code that I've put on this branch? Make sure all the version numbers match what you are using on the provider side.

https://github.com/pact-foundation/pact-ruby-e2e-example/tree/issues/pact-ruby-221

bethesque commented 4 years ago

The command to reproduce is bundle exec rake pact:bethtest

jmortlock commented 4 years ago

I was not able to reproduce using that repo (I did make a change to run the task pact:verify:foobar)


Tagging version 1.0.0 of Foo as "production"
Tagging version 1.0.0 of Foo as "master"
Publishing Foo/Bar pact to pact broker at https://test.pactflow.io
A pact for this consumer version is already published. Overwriting. (Note: Overwriting pacts is not recommended as it can lead to race conditions. Best practice is to provide a unique consumer version number for each publication.)
The latest version of this pact can be accessed at the following URL (use this to configure the provider verification):
https://test.pactflow.io/pacts/provider/Bar/consumer/Foo/latest

SPEC_OPTS='' /home/mortlock/.rbenv/versions/2.4.4/bin/ruby -S pact verify --pact-helper ./provider/spec/pact_helper.rb --pact-uri ./consumer/spec/pacts/foo-bar.json --backtrace
INFO: Reading pact at ./consumer/spec/pacts/foo-bar.json

Verifying a pact between Foo and Bar
  A retrieve thing request
    with GET /thing
      returns a response which
        has status code 200
        has a matching body
        includes headers
          "Content-Type" which equals "application/json"

1 interaction, 0 failures
WARN: Cannot publish verification for Foo as there is no link named pb:publish-verification-results in the pact JSON. If you are using a pact broker, please upgrade to version 2.0.0 or later.
jmortlock commented 4 years ago

Running some debug locally I think this may be the cause

From: /home/mortlock/src/auth/vendor/ruby/2.6.0/gems/pact-1.51.0/lib/pact/provider/verification_results/create.rb:33 Pact::Provider::VerificationResults::Create#publishable?:

30: def publishable?
31:   require 'pry'
32:   binding.pry
33:   executed_interactions_count == all_interactions_count && all_interactions_count > 0
34: end

[6] pry(#<Pact::Provider::VerificationResults::Create>)> executed_interactions_count
=> 2
[7] pry(#<Pact::Provider::VerificationResults::Create>)> all_interactions_count
=> 1

As you can see the executed_interactions_count is more than all_interactions which causes the publish to fail.

For this particular example I would expected there to be one test executed, as both master / production are at the same point

jmortlock commented 4 years ago

Seems like the purpose of this check is to ensure all interactions have been executed, so I have a pseduo-code type solution

def all_executed
  executed_ids = examples_for_pact_uri.map { |o| o[:pact_interaction]._id }.uniq
  all_ids = pact_source.pact_hash['interactions'].map { |o| o["_id"] }
  (all_ids - executed_ids).count == 0
end

Obviously not very familiar with the internal workings of all this, and the underscore for id kinda implies its private so not sure if this is a correct approach or there is something else not correct in my local setup

bethesque commented 4 years ago

I can't work out why there are two lines for "INFO: Reading pact at /pacts/provider/PetStore/consumer/PetStore-Web/version/4aeec9b870979a8a8b66108f7382669b85556796". The latest Pact Broker/Pact Ruby combination deduplicates the pacts, so you should only ever have that line once, and the verification should only run once. Are you on the latest version of everything?

jmortlock commented 4 years ago

Not running the very latest, pact-broker is 2.52.2-1 and pact-ruby is 1.51.0.

I will get the pact-broker upgraded and see how we go

bethesque commented 4 years ago

Those versions should be fine. 

SPEC_OPTS='' /home/mortlock/.rbenv/versions/2.4.4/bin/ruby -S pact verify --pact-helper ./provider/spec/pact_helper.rb --pact-uri ./consumer/spec/pacts/foo-bar.json --backtrace
INFO: Reading pact at ./consumer/spec/pacts/foo-bar.json

That's not reading the pact from the correct location. Are you running the bethtest task?

jmortlock commented 4 years ago

That may be because I changed the task from task 'pact:bethtest' => ['pact:set_publish_verification_results_flag', 'pact:publish:remote', 'pact:verify'] to task 'pact:bethtest' => ['pact:set_publish_verification_results_flag', 'pact:publish:remote', 'pact:verify:foobar']

Running with the original task I get the following

mortlock@mortlock:~/src/pact-ruby-e2e-example (issues/pact-ruby-221)*$ bundle exec rake pact:bethtest

Tagging version 1.0.0 of Foo as "production"
Tagging version 1.0.0 of Foo as "master"
Publishing Foo/Bar pact to pact broker at https://test.pactflow.io
A pact for this consumer version is already published. Overwriting. (Note: Overwriting pacts is not recommended as it can lead to race conditions. Best practice is to provide a unique consumer version number for each publication.)
The latest version of this pact can be accessed at the following URL (use this to configure the provider verification):
https://test.pactflow.io/pacts/provider/Bar/consumer/Foo/latest

SPEC_OPTS='' /home/mortlock/.rbenv/versions/2.4.4/bin/ruby -S pact verify --pact-helper /home/mortlock/src/pact-ruby-e2e-example/consumer/spec/pact_helper.rb
/home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/pact-1.51.1/lib/pact/cli/run_pact_verification.rb:74:in `run_with_configured_pacts_from_pact_helper': Please configure a pact to verify (RuntimeError)
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/pact-1.51.1/lib/pact/cli/run_pact_verification.rb:52:in `run_specs'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/pact-1.51.1/lib/pact/cli/run_pact_verification.rb:21:in `call'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/pact-1.51.1/lib/pact/cli/run_pact_verification.rb:13:in `call'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/pact-1.51.1/lib/pact/cli.rb:31:in `verify'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/gems/pact-1.51.1/bin/pact:4:in `<top (required)>'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/bin/pact:23:in `load'
    from /home/mortlock/src/pact-ruby-e2e-example/vendor/bundle/bin/pact:23:in `<main>'
bethesque commented 4 years ago

Gah. "It works on my machine" but fails with yet another error on Travis. https://travis-ci.org/github/pact-foundation/pact-ruby-e2e-example/builds/722418000

bethesque commented 4 years ago

Ok, passing build, using the ./test.sh script.

https://travis-ci.org/github/pact-foundation/pact-ruby-e2e-example/builds/725092832

SPEC_OPTS='' /home/travis/.rvm/rubies/ruby-2.4.10/bin/ruby -S pact verify --pact-helper /home/travis/build/pact-foundation/pact-ruby-e2e-example/provider/spec/pact_helper.rb INFO: Fetching pacts for Bar from https://test.pactflow.io with the selection criteria: latest for tag master, latest for tag production INFO: Reading pact at https://dXfltyFMgNOFZAxr8io9wJ37iUpY42M:*****@test.pactflow.io/pacts/provider/Bar/consumer/Foo/pact-version/418323d7e84d3e08ce42c4bba2f11714a4c25f47 DEBUG: The pact at https://test.pactflow.io/pacts/provider/Bar/consumer/Foo/pact-version/418323d7e84d3e08ce42c4bba2f11714a4c25f47 is being verified because it matches the following configured selection criteria: latest pact for a consumer version tagged 'master', latest pact for a consumer version tagged 'production' (both have the same content)

I've worked out why your ruby client is not using the new API. You probably don't have the feature toggle on for the Pact Broker. I'm about to put out a release without the feature toggle (officially releasing the feature!). Grab a new copy of the broker. It will be 2.60.

jmortlock commented 4 years ago

So I updated the pact-broker to 2.60 but now I get a new error when running the pact verification.

{:name=>"production", :all=>false, :fallback=>"master"}, latest for tag master /opt/atlassian/pipelines/agent/build/vendor/bundle/ruby/2.6.0/gems/pact-1.51.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:75:in query': undefined methodany?' for "1.17.x":String (NoMethodError)

jmortlock commented 4 years ago

Rolling pact-broker back to 2.59.2.0 resolves the error

bethesque commented 4 years ago

There was a bug in the href of the "beta:pacts-for-verification" relation that I believe was causing the error you saw. Try upgrading to 2.60.1.0.

jmortlock commented 4 years ago

Unfortunately I've hit the same error when running against pact broker 2.60.1.0

mefellows commented 4 years ago

Hmm this showed up on our forums this morning and sounds very similar:

Screen Shot 2020-09-11 at 4 25 35 pm

bethesque commented 4 years ago

The API that supports the pending pact, WIP pacts, and consumer version selectors during verification has only just been officially released. There are some versions of the pact-ruby-standalone (the underlying shared pact code) that use a beta version of the API that is now no longer supported (the original API used a GET, but the parameters turned out to be too complex, so it was changed to a POST). The latest version of the libraries use the POST API.

bethesque commented 4 years ago

The error on line 75 is:

q["providerVersionTags"] = provider_version_tags if provider_version_tags.any?

Tags is an array.

bethesque commented 4 years ago

If you grab the latest ruby gem, I've updated the tags to accept a single string.

bethesque commented 4 years ago

Oh, you can't because you're on pact-js. Well, try changing the tags to an array in the js.

jmortlock commented 4 years ago

I've actually built CI tooling to handling creation of pact tags / publishing so I use https://github.com/pact-foundation/pact-ruby-standalone/releases which looks like you have also just released with the gem update. I will be able to test all this on Monday.

bethesque commented 4 years ago

I've lost track of who has which problem at this stage! Trying to support too many issues at the same time.

Here's a working example with the latest Pactflow and rubygem code.

Code: https://github.com/pact-foundation/pact-ruby-e2e-example/tree/issues/pact-ruby-221-2 Build: https://travis-ci.org/github/pact-foundation/pact-ruby-e2e-example/builds/726487591

jmortlock commented 4 years ago

Looks like this is going to be a big job for me to test, as to move to the v2.60 pact-broker I first need to update the pact-ruby gem in all projects.

Happy to close this ticket off for now and can re-open if I hit further issues with this.

bethesque commented 4 years ago

You shouldn't need to upgrade everything, as the Broker/gem are meant to be forwards/backwards compatible with each other (as in, you may not get all the features, but it should at least not break). Because I don't have the right code to reproduce your original issue, I can't help any further I'm afraid. I was hoping you'd be able to use the ruby-e2e example to find the exact combination of code/configuration that causes it.

Let us know how you go.

jmortlock commented 4 years ago

In this case there is missing compatibility due to the pact-broker no longer returning an array for our tags.

So pact-broker 2.60 pact broker will fail for us unless we also update the pact gem to the latest version.

So with pact broker >= 2.60 we get this error. /fetch_pact_uris_for_verification.rb:75:in query': undefined method any?' for "1.17.x":String (NoMethodError)

Rolling back pact broker back to 2.59 we no longer get the error.

Updating the pact gem in the provider project fixes the issue as it will implicitly converts all tags into an array on the client side (https://github.com/pact-foundation/pact-ruby/commit/16866f440afdad355266e0eb9685529d8a995de3)

Not sure when or why pact-broker is not returning the tags as an array, we have always used the pact-cli to publish.

bethesque commented 4 years ago

I'm still unclear as to how to recreate the problem, because I can't tell what parts you're using the Ruby library directly, and what you're using the CLI for. Can you provide me the command and code versions that you're using with the pact-cli that demonstrates the issue for you? And when you say, "using the pact-cli to publish" do you mean, publish pacts, or publish verification results?

svobom57 commented 4 years ago

@bethesque We're running into the same issue where we're running verification against multiple consumer version tags:

Pact.service_provider 'provider-service' do
  CONSUMER_PACT_TAG = ENV['CONSUMER_PACT_TAG'] || `git branch --show`.chomp
  PROVIDER_VERSION = ENV['PROVIDER_VERSION'] || `git rev-parse HEAD`.chomp

  publish_verification_results true
  app_version PROVIDER_VERSION

  honours_pacts_from_pact_broker do
    pact_broker_base_url ENV['PACT_BROKER_URL'], username: ENV['PACT_BROKER_USERNAME'], password: ENV['PACT_BROKER_PASSWORD']
    consumer_version_tags [{ name: CONSUMER_PACT_TAG, fallback: 'master' }, { name: 'master' }]
  end
end

This produces following warnings when running pact:verify on the provider-service:

WARN: Cannot publish verification for consumer-service-1 as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.
WARN: Cannot publish verification for consumer-service-2 as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.
WARN: Cannot publish verification for consumer-service-3 as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.

However, if we remove the { name: 'master' } consumer version tag we'll get a successful publish.

bethesque commented 4 years ago

@svobom57 can you please follow the instructions in the README in this branch to recreate the issue using https://github.com/pact-foundation/pact-ruby-e2e-example/tree/feat/docker-compose

You can see the currently passing build here: https://travis-ci.org/github/pact-foundation/pact-ruby-e2e-example/builds/730426907

I really want to get this fixed, but I cannot do anything more until I have a way of recreating the issue.

@svobom57 I did just put out a new version of the ruby gem today. There was a regression where the hash consumer_version_tags weren't being properly converted into the format required for the new API. If you update, hopefully you will find your issue is fixed, but otherwise, I look forward to seeing your repro code.

svobom57 commented 4 years ago

@bethesque I've succeeded simulating the issue on your branch, thanks for setting up dockerised approach 👏 .

All I did was to downgrade pact-broker to 2.54.0.0 (the version we're running in our organisation).

You can run ./run.sh on my forked branch of yours: https://github.com/svobom57/pact-ruby-e2e-example/commit/f8af0a54f6018ff2df26a370086e80707eacba2e

I noticed that you're using consumer_version_selectors which I'm not familiar with in the pact_helper.rb here: https://github.com/svobom57/pact-ruby-e2e-example/blob/7c6fa241a8feca71d793e191ff67f74b6e47e6ff/provider/spec/pact_helper.rb#L13 We're using consumer_version_tags as I suggested in the example above. Regardless, the issue is similar. I wasn't able to get the example working with consumer_version_tags at all. I might take another stab at it but we have a starting point now:

verify-pacts_1   | INFO: Reading pact at http://pact-broker:9292/pacts/provider/Bar/consumer/Foo/version/1
verify-pacts_1   | INFO: Reading pact at http://pact-broker:9292/pacts/provider/Bar/consumer/Foo2/version/2
verify-pacts_1   | INFO: Reading pact at http://pact-broker:9292/pacts/provider/Bar/consumer/Foo/version/1
verify-pacts_1   | INFO: Reading pact at http://pact-broker:9292/pacts/provider/Bar/consumer/Foo2/version/2
verify-pacts_1   | 
verify-pacts_1   | 
verify-pacts_1   | Verifying a pact between Foo and Bar
verify-pacts_1   |   A retrieve thing request
verify-pacts_1   |     with GET /thing
verify-pacts_1   |       returns a response which
verify-pacts_1   |         has status code 200
verify-pacts_1   |         has a matching body
verify-pacts_1   |         includes headers
verify-pacts_1   |           "Content-Type" which equals "application/json"
verify-pacts_1   | 
verify-pacts_1   | 
verify-pacts_1   | Verifying a pact between Foo2 and Bar
verify-pacts_1   |   Another retrieve thing request
verify-pacts_1   |     with GET /thing2
verify-pacts_1   |       returns a response which
verify-pacts_1   |         has status code 200
verify-pacts_1   |         has a matching body
verify-pacts_1   |         includes headers
verify-pacts_1   |           "Content-Type" which equals "application/json"
verify-pacts_1   |   Another retrieve thing request 2
verify-pacts_1   |     with GET /thing2
verify-pacts_1   |       returns a response which
verify-pacts_1   |         has status code 200
verify-pacts_1   |         has a matching body
verify-pacts_1   |         includes headers
verify-pacts_1   |           "Content-Type" which equals "application/json"
verify-pacts_1   | 
verify-pacts_1   | 
verify-pacts_1   | Verifying a pact between Foo and Bar
verify-pacts_1   |   A retrieve thing request
verify-pacts_1   |     with GET /thing
verify-pacts_1   |       returns a response which
verify-pacts_1   |         has status code 200
verify-pacts_1   |         has a matching body
verify-pacts_1   |         includes headers
verify-pacts_1   |           "Content-Type" which equals "application/json"
verify-pacts_1   | 
verify-pacts_1   | 
verify-pacts_1   | Verifying a pact between Foo2 and Bar
verify-pacts_1   |   Another retrieve thing request
verify-pacts_1   |     with GET /thing2
verify-pacts_1   |       returns a response which
verify-pacts_1   |         has status code 200
verify-pacts_1   |         has a matching body
verify-pacts_1   |         includes headers
verify-pacts_1   |           "Content-Type" which equals "application/json"
verify-pacts_1   |   Another retrieve thing request 2
verify-pacts_1   |     with GET /thing2
verify-pacts_1   |       returns a response which
verify-pacts_1   |         has status code 200
verify-pacts_1   |         has a matching body
verify-pacts_1   |         includes headers
verify-pacts_1   |           "Content-Type" which equals "application/json"
verify-pacts_1   | 
verify-pacts_1   | 6 interactions, 0 failures
verify-pacts_1   | 
verify-pacts_1   | WARN: Cannot publish verification for Foo as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.
verify-pacts_1   | WARN: Cannot publish verification for Foo2 as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.
verify-pacts_1   | WARN: Cannot publish verification for Foo as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.
verify-pacts_1   | WARN: Cannot publish verification for Foo2 as not all interactions have been verified. Re-run the verification without the filter parameters or environment variables to publish the verification.
bethesque commented 4 years ago

Awesome sauce. I'll have a look at it as soon as I have a moment.

bethesque commented 4 years ago

Ok, the thing that was causing the issue was that when you verify a pact with the same URL twice, the number of interactions in this pact is equal to n, and the number of executed interactions that came from this pact URL was n * 2. Because publishable? was checking for equality, these two obviously didn't equal each other. I was never able to reproduce it because I was using the new 'pacts for verification' API that deduplicates the pacts before they even reach the client. This is why the problem didn't exist in the example I set up.

I've updated the logic in the pact gem to be smarter about how it checks if all the interactions have been run (it now checks that each interaction has an example associated with it, rather than doing a count) so the latest pact gem will work with the old broker.

I do recommend upgrading your broker though, as the new 'pact for verification' API will give you some cool new features.

https://docs.pact.io/pact_broker/advanced_topics/provider_verification_results#pacts-for-verification