search

pact-foundation / pact-reference

Reference implementations for the pact specifications
https://pact.io
MIT License
91 stars 46 forks source link

pact_verifier_cli process exits with status code 0 even if verification has failed #258

Closed sergeyklay closed 1 year ago

sergeyklay commented 1 year ago

Hi,

Just realized that pact_verifier_cli exits with status 0 even if the pact verification fails.

I deployed a new broker from scratch, no pact in it yet. Then I published the first contract in it and made the very first verification attempt (it failed). And verification process ended with a status code of 0. This makes it difficult to configure the CI/CD pipeline.

Could someone advise if there is a way around this without resorting grep output?

% pact_verifier_cli --version
pact verifier version   : v0.10.3
pact specification      : v4.0
models version          : v1.0.4

The script I use to reproduce the issue at my local workstation looks as follows:

# -e  Exit immediately if a command exits with a non-zero status.
# -u  Treat unset variables as an error when substituting.

set -eu
set -o pipefail

PROJECT_ROOT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)

if [ -z "${PYENV_VIRTUAL_ENV}" ]; then
  PYTHON="${PYTHON:-$(command -v python3 2>/dev/null)}"
else
  PYTHON="${PYTHON:-$(pyenv which python 2>/dev/null)}"
fi

# To get the most out of the Pact Broker, it should either be the git sha
# (or equivalent for your repository), be a git tag name, or it should
# include the git sha or tag name as metadata if you are using semantic
# versioning eg. 1.2.456+405b31ec6.
#
# See https://docs.pact.io/pact_broker/pacticipant_version_numbers for more
# details.
APP_VERSION="$($PYTHON setup.py --version)"
APP_VERSION+="+$(git -C "$PROJECT_ROOT" rev-parse --short HEAD)"

# For the purposes of this example, the broker is started up using docker
# compose (see '.github/workflows/test-contracts.yaml'). For normal usage this
# would be self-hosted or using PactFlow.
PACT_BROKER_USERNAME="${PACT_BROKER_USERNAME:-pactbroker}"
PACT_BROKER_PASSWORD="${PACT_BROKER_PASSWORD:-pactbroker}"
PACT_BROKER_BASE_URL="${PACT_BROKER_BASE_URL:-http://localhost}"
export PACT_BROKER_USERNAME PACT_BROKER_PASSWORD PACT_BROKER_BASE_URL

# For the purposes of this example, the Flask server will be started up as a
# part of this script when running the tests. Alternatives could be,
# for example running a Docker container with a DB of test data configured.
# This is the "real" provider to verify against.
PROVIDER_HOST="${PROVIDER_HOST:-127.0.0.1}"
PROVIDER_PORT="${PROVIDER_PORT:-5001}"
PROVIDER_TRANSPORT="${PROVIDER_TRANSPORT:-http}"

# This will disable console messages for the 'tests/app.py'.
TEST_MODE="verify"
export TEST_MODE

# Run the Flask server, using the 'tests/app.py' as the app to be able to
# inject the '-pact/provider-states' endpoint
FLASK_APP="$PROJECT_ROOT/tests/app.py"
export FLASK_APP

$PYTHON -m flask run \
  --port "$PROVIDER_PORT" \
  --host "$PROVIDER_HOST" &
FLASK_PID=$!

# Make sure the Flask server is stopped when finished to avoid blocking the port
function teardown {
  echo "Tearing down Flask server: ${FLASK_PID}"
  kill -9 $FLASK_PID 2>/dev/null || true
}
trap teardown EXIT

# Wait a little in case Flask isn't quite ready
while ! nc -z "$PROVIDER_HOST" "$PROVIDER_PORT"; do
  sleep 0.1
done

# Turn off the annoying warning in the output
PACT_DO_NOT_TRACK=true
export PACT_DO_NOT_TRACK

# Consumer version selectors:
#
# mainBranch:         Recommended. Returns the pacts for consumers configured
#                     mainBranch property.
# deployedOrReleased: Recommended. Returns the pacts for all versions of the
#                     consumer that are currently deployed or released and
#                     currently supported in any environment.
pact_verifier_cli \
  --provider-name ProductService \
  --provider-version "$APP_VERSION" \
  --provider-branch="$(git -C "$PROJECT_ROOT" rev-parse --abbrev-ref HEAD)" \
  --hostname "$PROVIDER_HOST" \
  --transport "$PROVIDER_TRANSPORT" \
  --port "$PROVIDER_PORT" \
  --disable-ssl-verification \
  --state-change-url "${PROVIDER_TRANSPORT}://${PROVIDER_HOST}:${PROVIDER_PORT}/-pact/provider-states" \
  --header User-Agent=PactBroker \
  --publish \
  --enable-pending \
  --consumer-version-selectors '{"mainBranch": true}' \
  --consumer-version-selectors '{"deployedOrReleased": true}' \
  --loglevel info
The output of this script

``` % ./verify-contracts.sh Connection to 127.0.0.1 port 5001 [tcp/commplex-link] succeeded! 2023-03-08T16:56:06.237793Z INFO main pact_verifier::pact_broker: Fetching path '/' from pact broker 2023-03-08T16:56:06.268221Z INFO main pact_verifier::pact_broker: Fetching path '/pacts/provider/ProductService/for-verification' from pact broker 2023-03-08T16:56:06.357725Z INFO main pact_verifier::pact_broker: Fetching path '/pacts/provider/ProductService/consumer/ProductServiceClient/pact-version/a5620bf979253b375bb2b03bfb4feab67446a82c/metadata/c1tdW2N2XT04OSZwPXRydWU' from pact broker Address already in use Port 5001 is in use by another program. Either identify and stop that program, or start the server with a different port. 2023-03-08T16:56:06.524051Z INFO main pact_verifier: Running setup provider state change handler 'there is a product with ID 1' for 'a request for a product' 2023-03-08T16:56:07.770291Z ERROR main pact_verifier: Provider setup state change for 'there is a product with ID 1' has failed - MismatchResult::Error("Invalid status code: 500", Some("847128512b93c963c678babfdfc617d67081eb3b")) 2023-03-08T16:56:07.917970Z INFO main pact_verifier: Running setup provider state change handler 'there is no product with ID 7777' for 'a request for a product' 2023-03-08T16:56:08.041469Z INFO main pact_verifier: Running provider verification for 'a request for a product' 2023-03-08T16:56:08.042868Z INFO main pact_verifier::provider_client: Sending request to provider at http://127.0.0.1:5001 2023-03-08T16:56:08.042874Z INFO main pact_verifier::provider_client: Sending request HTTP Request ( method: GET, path: /v1/products/7777, query: None, headers: None, body: Missing ) 2023-03-08T16:56:08.043546Z INFO main pact_verifier::provider_client: Received response: HTTP Response ( status: 404, headers: Some({"content-length": ["34"], "date": ["Wed", "08 Mar 2023 16:56:08 GMT"], "connection": ["close"], "server": ["Werkzeug/2.2.3 Python/3.11.0"], "content-type": ["application/json"]}), body: Present(34 bytes, application/json) ) 2023-03-08T16:56:08.043811Z INFO main pact_matching: comparing to expected response: HTTP Response ( status: 404, headers: Some({"Content-Type": ["application/json"]}), body: Present(72 bytes) ) 2023-03-08T16:56:08.169018Z INFO main pact_verifier: Running setup provider state change handler 'there is no product with ID 7777' for 'a request to delete a product' 2023-03-08T16:56:08.290803Z INFO main pact_verifier: Running provider verification for 'a request to delete a product' 2023-03-08T16:56:08.290838Z INFO main pact_verifier::provider_client: Sending request to provider at http://127.0.0.1:5001 2023-03-08T16:56:08.290840Z INFO main pact_verifier::provider_client: Sending request HTTP Request ( method: DELETE, path: /v1/products/7777, query: None, headers: None, body: Missing ) 2023-03-08T16:56:08.291469Z INFO main pact_verifier::provider_client: Received response: HTTP Response ( status: 404, headers: Some({"server": ["Werkzeug/2.2.3 Python/3.11.0"], "date": ["Wed", "08 Mar 2023 16:56:08 GMT"], "content-type": ["application/json"], "content-length": ["34"], "connection": ["close"]}), body: Present(34 bytes, application/json) ) 2023-03-08T16:56:08.291483Z INFO main pact_matching: comparing to expected response: HTTP Response ( status: 404, headers: Some({"Content-Type": ["application/json"]}), body: Present(72 bytes) ) 2023-03-08T16:56:08.413341Z INFO main pact_verifier: Running setup provider state change handler 'there are no products' for 'a request to get list of products' 2023-03-08T16:56:08.535136Z INFO main pact_verifier: Running provider verification for 'a request to get list of products' 2023-03-08T16:56:08.535173Z INFO main pact_verifier::provider_client: Sending request to provider at http://127.0.0.1:5001 2023-03-08T16:56:08.535175Z INFO main pact_verifier::provider_client: Sending request HTTP Request ( method: GET, path: /v1/products, query: None, headers: None, body: Missing ) 2023-03-08T16:56:08.535691Z INFO main pact_verifier::provider_client: Received response: HTTP Response ( status: 404, headers: Some({"content-length": ["34"], "content-type": ["application/json"], "connection": ["close"], "server": ["Werkzeug/2.2.3 Python/3.11.0"], "date": ["Wed", "08 Mar 2023 16:56:08 GMT"]}), body: Present(34 bytes, application/json) ) 2023-03-08T16:56:08.535705Z INFO main pact_matching: comparing to expected response: HTTP Response ( status: 200, headers: Some({"ETag": ["\"a36c1fae7588366925a982e9a026b1d9\""], "Content-Type": ["application/json"]}), body: Present(291 bytes) ) 2023-03-08T16:56:08.657054Z INFO main pact_verifier: Running setup provider state change handler 'there are few products' for 'a request to get expanded list of products' 2023-03-08T16:56:08.787238Z INFO main pact_verifier: Running provider verification for 'a request to get expanded list of products' 2023-03-08T16:56:08.787291Z INFO main pact_verifier::provider_client: Sending request to provider at http://127.0.0.1:5001 2023-03-08T16:56:08.787293Z INFO main pact_verifier::provider_client: Sending request HTTP Request ( method: GET, path: /v1/products, query: Some({"expanded": ["1"]}), headers: None, body: Missing ) 2023-03-08T16:56:08.787876Z INFO main pact_verifier::provider_client: Received response: HTTP Response ( status: 404, headers: Some({"server": ["Werkzeug/2.2.3 Python/3.11.0"], "date": ["Wed", "08 Mar 2023 16:56:08 GMT"], "content-length": ["34"], "content-type": ["application/json"], "connection": ["close"]}), body: Present(34 bytes, application/json) ) 2023-03-08T16:56:08.787900Z INFO main pact_matching: comparing to expected response: HTTP Response ( status: 200, headers: Some({"ETag": ["\"a36c1fae7588366925a982e9a026b1d9\""], "Content-Type": ["application/json"]}), body: Present(817 bytes) ) 2023-03-08T16:56:08.914593Z INFO main pact_verifier: Running setup provider state change handler 'there are few products' for 'a request to get collapsed list of products' 2023-03-08T16:56:09.037900Z INFO main pact_verifier: Running provider verification for 'a request to get collapsed list of products' 2023-03-08T16:56:09.037939Z INFO main pact_verifier::provider_client: Sending request to provider at http://127.0.0.1:5001 2023-03-08T16:56:09.037941Z INFO main pact_verifier::provider_client: Sending request HTTP Request ( method: GET, path: /v1/products, query: Some({"expanded": ["0"]}), headers: None, body: Missing ) 2023-03-08T16:56:09.038488Z INFO main pact_verifier::provider_client: Received response: HTTP Response ( status: 404, headers: Some({"content-length": ["34"], "connection": ["close"], "content-type": ["application/json"], "date": ["Wed", "08 Mar 2023 16:56:09 GMT"], "server": ["Werkzeug/2.2.3 Python/3.11.0"]}), body: Present(34 bytes, application/json) ) 2023-03-08T16:56:09.038503Z INFO main pact_matching: comparing to expected response: HTTP Response ( status: 200, headers: Some({"Content-Type": ["application/json"], "ETag": ["\"a36c1fae7588366925a982e9a026b1d9\""]}), body: Present(385 bytes) ) 2023-03-08T16:56:09.038809Z INFO main pact_verifier: Publishing verification results back to the Pact Broker 2023-03-08T16:56:09.165645Z INFO main pact_verifier::pact_broker: Fetching path '/pacticipants/ProductService' from pact broker 2023-03-08T16:56:09.332413Z INFO main pact_verifier: Results published to Pact Broker The pact at http://localhost/pacts/provider/ProductService/consumer/ProductServiceClient/pact-version/a5620bf979253b375bb2b03bfb4feab67446a82c is being verified because the pact content belongs to the consumer version matching the following criterion: * latest version from the main branch 'main' (1.0.0+c75b8af) This pact is in pending state for this version of ProductService because a successful verification result for a version of ProductService from branch 'feature/flask-smorest' has not yet been published. If this verification fails, it will not cause the overall build to fail. Read more at https://docs.pact.io/go/pending Verifying a pact between ProductServiceClient and ProductService a request for a product (312ms loading, 1s 374ms verification) Given there is a product with ID 1 Request Failed - One or more of the setup state change handlers has failed a request for a product (312ms loading, 275ms verification) Given there is no product with ID 7777 returns a response which has status code 404 (OK) includes headers "Content-Type" with value "application/json" (OK) has a matching body (FAILED) a request to delete a product (312ms loading, 245ms verification) Given there is no product with ID 7777 returns a response which has status code 404 (OK) includes headers "Content-Type" with value "application/json" (OK) has a matching body (FAILED) a request to get list of products (312ms loading, 244ms verification) Given there are no products returns a response which has status code 200 (FAILED) includes headers "ETag" with value ""a36c1fae7588366925a982e9a026b1d9"" (FAILED) "Content-Type" with value "application/json" (OK) has a matching body (FAILED) a request to get expanded list of products (312ms loading, 252ms verification) Given there are few products returns a response which has status code 200 (FAILED) includes headers "ETag" with value ""a36c1fae7588366925a982e9a026b1d9"" (FAILED) "Content-Type" with value "application/json" (OK) has a matching body (FAILED) a request to get collapsed list of products (312ms loading, 250ms verification) Given there are few products returns a response which has status code 200 (FAILED) includes headers "Content-Type" with value "application/json" (OK) "ETag" with value ""a36c1fae7588366925a982e9a026b1d9"" (FAILED) has a matching body (FAILED) This pact is still in pending state for any version of ProductService from branch 'feature/flask-smorest' as a successful verification result has not yet been published Pending Failures: 1) Verifying a pact between ProductServiceClient and ProductService Given there is a product with ID 1 - a request for a product - One or more of the setup state change handlers has failed 2) Verifying a pact between ProductServiceClient and ProductService Given there is no product with ID 7777 - a request for a product 2.1) has a matching body $.status -> Expected '404' to be equal to 'Not Found' $ -> Actual map is missing the following keys: description, title 3) Verifying a pact between ProductServiceClient and ProductService Given there is no product with ID 7777 - a request to delete a product 3.1) has a matching body $ -> Actual map is missing the following keys: description, title $.status -> Expected '404' to be equal to 'Not Found' 4) Verifying a pact between ProductServiceClient and ProductService Given there are no products - a request to get list of products 4.1) has a matching body $ -> Actual map is missing the following keys: links, pagination, products 4.2) has status code 200 expected 200 but was 404 4.3) includes header 'ETag' with value '"\"a36c1fae7588366925a982e9a026b1d9\""' Expected header 'ETag' to have value '"\"a36c1fae7588366925a982e9a026b1d9\""' but was '' 5) Verifying a pact between ProductServiceClient and ProductService Given there are few products - a request to get expanded list of products 5.1) has a matching body $ -> Actual map is missing the following keys: links, pagination, products 5.2) has status code 200 expected 200 but was 404 5.3) includes header 'ETag' with value '"\"a36c1fae7588366925a982e9a026b1d9\""' Expected header 'ETag' to have value '"\"a36c1fae7588366925a982e9a026b1d9\""' but was '' 6) Verifying a pact between ProductServiceClient and ProductService Given there are few products - a request to get collapsed list of products 6.1) has a matching body $ -> Actual map is missing the following keys: links, pagination, products 6.2) has status code 200 expected 200 but was 404 6.3) includes header 'ETag' with value '"\"a36c1fae7588366925a982e9a026b1d9\""' Expected header 'ETag' to have value '"\"a36c1fae7588366925a982e9a026b1d9\""' but was '' There were 6 non-fatal pact failures on pending pacts or interactions (see docs.pact.io/pending for more information) ```

YOU54F commented 1 year ago 
The pact at http://localhost/pacts/provider/ProductService/consumer/ProductServiceClient/pact-version/a5620bf979253b375bb2b03bfb4feab67446a82c is being verified because the pact content belongs to the consumer version matching the following criterion:
    * latest version from the main branch 'main' (1.0.0+c75b8af)
This pact is in pending state for this version of ProductService because a successful verification result for a version of ProductService from branch 'feature/flask-smorest' has not yet been published. If this verification fails, it will not cause the overall build to fail. Read more at https://docs.pact.io/go/pending
Verifying a pact between ProductServiceClient and ProductService
Pending Failures:
1) 
There were 6 non-fatal pact failures on pending pacts or interactions (see docs.pact.io/pending for more information)

So this is due to the --enable-pending flag being set to true and is expected, setting this flag to false, would cause the provider build to fail, which would be a less than ideal solution, as this would prevent the provider from deploying through no fault of their own.

More info :- https://docs.pact.io/pact_broker/advanced_topics/pending_pacts#purpose :)

sergeyklay commented 1 year ago

Thank you for helping me understand what the problem was!

mefellows commented 1 year ago

Thank you for the beautiful bug report and detail - this really helps!