pact-foundation / pact-js

JS version of Pact. Pact is a contract testing framework for HTTP APIs and non-HTTP asynchronous messaging systems.
https://pact.io
Other
1.63k stars 347 forks source link

Nested arrayContaining causes mocked server to return null for the inner array #841

Closed wabrit closed 1 year ago

wabrit commented 2 years ago

Software versions

Expected behaviour

Using arrayContaining matcher should work regardless of nesting level (e.g. arrayContaining including a nested property which is also matched by arrayContaining)

Actual behaviour

An arrayContaining nested inside another arrayContaining causes the mocked server to return null for the inner array.

Steps to reproduce

Sample (GraphQL) query

query badQuery($id: ID!) {
        foo(id: $id) {
            bar {
              results {
                  __typename
              }
            }
        }
    }

Sample expectation:

foo: {
  bar: arrayContaining(
    {
      results: arrayContaining({
        __typename: equal('Result')
       })
     }
   )
  }

Relevant log files

DEBUG output from consumer test:

[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.variables
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.variables.id
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] detected pact:matcher:type, will configure a matcher
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.data
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.data.foo
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.data.foo.bar
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] detected pact:matcher:type, will configure a matcher
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.results
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] detected pact:matcher:type, will configure a matcher
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.__typename
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] detected pact:matcher:type, will configure a matcher
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.results
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.results[*]
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.results[*].__typename
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] detected pact:matcher:type, will configure a matcher
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Skipping the matching rule (skip_matchers == true)
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.data.foo.bar
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.data.foo.bar[*]
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Configuring a normal object
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Path = $.data.foo.bar[*].results
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] detected pact:matcher:type, will configure a matcher
[2022-03-16T10:07:04Z DEBUG pact_ffi::mock_server::bodies] Skipping the matching rule (skip_matchers == true)
[2022-03-16T10:07:04Z DEBUG pact_plugin_driver::catalogue_manager] Updated catalogue entries:
    core/interaction/http
    core/interaction/https
[2022-03-16T10:07:04Z DEBUG pact_plugin_driver::catalogue_manager] Updated catalogue entries:
    core/content-generator/binary
    core/content-generator/json
    core/content-matcher/json
    core/content-matcher/multipart-form-data
    core/content-matcher/text
    core/content-matcher/xml
[2022-03-16T10:07:04Z DEBUG pact_plugin_driver::catalogue_manager] Updated catalogue entries:
    core/matcher/v1-equality
    core/matcher/v2-max-type
    core/matcher/v2-min-type
    core/matcher/v2-minmax-type
    core/matcher/v2-regex
    core/matcher/v2-type
    core/matcher/v3-content-type
    core/matcher/v3-date
    core/matcher/v3-datetime
    core/matcher/v3-decimal-type
    core/matcher/v3-includes
    core/matcher/v3-integer-type
    core/matcher/v3-null
    core/matcher/v3-number-type
    core/matcher/v3-time
    core/matcher/v4-array-contains
    core/matcher/v4-equals-ignore-order
    core/matcher/v4-max-equals-ignore-order
    core/matcher/v4-min-equals-ignore-order
    core/matcher/v4-minmax-equals-ignore-order
    core/matcher/v4-not-empty
    core/matcher/v4-semver
[2022-03-16T10:07:04Z DEBUG pact_mock_server::mock_server] Started mock server on 127.0.0.1:65384
[2022-03-16T10:07:04Z DEBUG hyper::proto::h1::io] parsed 7 headers 
[2022-03-16T10:07:04Z DEBUG hyper::proto::h1::conn] incoming body is content-length (178 bytes) 
[2022-03-16T10:07:04Z DEBUG hyper::proto::h1::conn] incoming body completed 
[2022-03-16T10:07:04Z DEBUG pact_mock_server::hyper_server] Creating pact request from hyper request
[2022-03-16T10:07:04Z DEBUG pact_mock_server::hyper_server] Extracting query from uri /graphql
[2022-03-16T10:07:04Z INFO  pact_mock_server::hyper_server] Received request HTTP Request ( method: POST, path: /graphql, query: None, headers: Some({"host": ["127.0.0.1:65384"], "content-type": ["application/json"], "content-length": ["178"], "user-agent": ["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"], "accept": ["*/*"], "connection": ["close"], "accept-encoding": ["gzip", "deflate"]}), body: Present(178 bytes, application/json) )
[2022-03-16T10:07:04Z DEBUG pact_mock_server::hyper_server]      body: '{"operationName":"badQuery","variables":{"id":"1"},"query":"query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}"}'
[2022-03-16T10:07:04Z INFO  pact_matching] comparing to expected HTTP Request ( method: POST, path: /graphql, query: None, headers: Some({"Content-Type": ["application/json"]}), body: Present(178 bytes, application/json) )
[2022-03-16T10:07:04Z DEBUG pact_matching]      body: '{"operationName":"badQuery","query":"query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}","variables":{"id":"1"}}'
[2022-03-16T10:07:04Z DEBUG pact_matching]      matching_rules: MatchingRules { rules: {BODY: MatchingRuleCategory { name: BODY, rules: {DocPath { path_tokens: [Root, Field("variables"), Field("id")], expr: "$.variables.id" }: RuleList { rules: [Type], rule_logic: And, cascaded: false }} }, PATH: MatchingRuleCategory { name: PATH, rules: {} }} }
[2022-03-16T10:07:04Z DEBUG pact_matching]      generators: Generators { categories: {} }
[2022-03-16T10:07:04Z DEBUG pact_matching::matchers] String -> String: comparing '/graphql' to '/graphql' using Equality (false)
[2022-03-16T10:07:04Z DEBUG pact_matching] expected content type = 'application/json', actual content type = 'application/json'
[2022-03-16T10:07:04Z DEBUG pact_matching] content type header matcher = 'RuleList { rules: [], rule_logic: And, cascaded: false }'
[2022-03-16T10:07:04Z DEBUG pact_plugin_driver::catalogue_manager] Looking for a content matcher for application/json
[2022-03-16T10:07:04Z DEBUG pact_matching] No content matcher defined for content type 'application/json', using core matcher implementation
[2022-03-16T10:07:04Z DEBUG pact_matching] Using body matcher for content type 'application/json'
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare: Comparing path $
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare_maps: Comparing maps at $: {"operationName": String("badQuery"), "query": String("query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}"), "variables": Object({"id": String("1")})} -> {"operationName": String("badQuery"), "query": String("query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}"), "variables": Object({"id": String("1")})}
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare: Comparing path $.operationName
[2022-03-16T10:07:04Z DEBUG pact_matching::json] JSON -> JSON: Comparing '"badQuery"' to '"badQuery"' using Equality -> Ok(())
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare_values: Comparing 'String("badQuery")' to 'String("badQuery")' at path '$.operationName' -> Ok(())
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare: Comparing path $.query
[2022-03-16T10:07:04Z DEBUG pact_matching::json] JSON -> JSON: Comparing '"query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}"' to '"query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}"' using Equality -> Ok(())
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare_values: Comparing 'String("query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}")' to 'String("query badQuery($id: ID!) {\n  foo(id: $id) {\n    bar {\n      results {\n        __typename\n      }\n    }\n  }\n}")' at path '$.query' -> Ok(())
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare: Comparing path $.variables
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare_maps: Comparing maps at $.variables: {"id": String("1")} -> {"id": String("1")}
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare: Comparing path $.variables.id
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare_values: Calling match_values for path $.variables.id
[2022-03-16T10:07:04Z DEBUG pact_matching::json] JSON -> JSON: Comparing '"1"' to '"1"' using Type -> Ok(())
[2022-03-16T10:07:04Z DEBUG pact_matching::json] compare_values: Comparing 'String("1")' to 'String("1")' at path '$.variables.id' -> Ok(())
[2022-03-16T10:07:04Z DEBUG pact_matching] --> Mismatches: []
[2022-03-16T10:07:04Z DEBUG pact_mock_server::hyper_server] Test context = {"mockServer": Object({"href": String("http://127.0.0.1:65384/"), "port": Number(65384)})}
[2022-03-16T10:07:04Z INFO  pact_mock_server::hyper_server] Request matched, sending response HTTP Response ( status: 200, headers: Some({"Content-Type": ["application/json"]}), body: Present(43 bytes, application/json) )
[2022-03-16T10:07:04Z DEBUG pact_mock_server::hyper_server]      body: '{"data":{"foo":{"bar":[{"results":null}]}}}'
[2022-03-16T10:07:04Z DEBUG hyper::proto::h1::io] flushed 355 bytes 
[2022-03-16T10:07:04Z INFO  pact_mock_server::mock_server] Writing pact out to 'C:\Dev\git_repos\apps\case_notes_on_the_go\pacts\ena-app-apigateway.json'
[2022-03-16T10:07:04Z DEBUG pact_models::pact] Merging pact with file "C:\\Dev\\git_repos\\apps\\case_notes_on_the_go\\pacts\\ena-app-apigateway.json"
[2022-03-16T10:07:04Z WARN  pact_models::pact] Note: Existing pact is an older specification version (V3), and will be upgraded
[2022-03-16T10:07:04Z DEBUG pact_mock_server::server_manager] Shutting down mock server with port 65384
[2022-03-16T10:07:04Z DEBUG pact_mock_server::server_manager] Shutting down mock server with port 65384 - MockServerMetrics { requests: 1 }
[2022-03-16T10:07:04Z DEBUG pact_mock_server::mock_server] Mock server cf98a3b4-4aa0-4836-8bcb-9f144d34a04b shutdown - MockServerMetrics { requests: 1 }
[2022-03-16T10:07:04Z DEBUG hyper::server::shutdown] signal received, starting graceful shutdown 

Error: expect(received).resolves.toEqual(expected) // deep equality

- Expected  - 5
+ Received  + 1

  Object {
    "foo": Object {
      "bar": Array [
        Object {
-         "results": Array [
-           Object {
-             "__typename": "Result",
-           },
-         ],
+         "results": null,
        },
      ],
    },
  }
mefellows commented 1 year ago

I wonder if this relates to #662 where the nesting didn't work @uglyog ?

diestrin commented 1 year ago

I can confirm this is still happening on 12.1.0

diestrin commented 1 year ago

I'm really happy! I've spent the last day just understanding Pact architecture (and learning a bit about rust too) and I think I nailed this down to just a single change.

When an arrayContains operator is used, the flag skip_matchers is being set to true, causing the inner content not to use the second arrayContains. See https://github.com/pact-foundation/pact-reference/blob/master/rust/pact_ffi/src/mock_server/bodies.rs#L111

match rule {
  MatchingRule::ArrayContains(_) => (obj.get("variants"), true),
    _ => (obj.get("value"), false)
}

I'm not gonna submit a PR cause I'm not sure if this flag is set to false because of a reason, but I can provide the test I created to reproduce the issue and to validate the fix.

In https://github.com/pact-foundation/pact-reference/blob/master/rust/pact_ffi/tests/tests.rs

fn array_contains_matcher() {
  let consumer_name = CString::new("array_contains_matcher-consumer").unwrap();
  let provider_name = CString::new("array_contains_matcher-provider").unwrap();
  let pact_handle = pactffi_new_pact(consumer_name.as_ptr(), provider_name.as_ptr());
  let description = CString::new("array_contains_matcher").unwrap();
  let interaction = pactffi_new_interaction(pact_handle.clone(), description.as_ptr());

  let content_type = CString::new("application/json").unwrap();
  let path = CString::new("/book").unwrap();
  let json = json!({
    "pact:matcher:type": "array-contains",
    "variants": [
      {
        "group": {
          "value": "A"
        },
        "users": {
          "pact:matcher:type": "array-contains",
          "variants": [
            {
              "id": {
                "value": 1
              }
            },
            {
              "id": {
                "value": 2
              }
            },
          ]
        }
      },
    ]
  });
  let body = CString::new(json.to_string()).unwrap();
  let address = CString::new("127.0.0.1:0").unwrap();
  let method = CString::new("GET").unwrap();

  pactffi_upon_receiving(interaction.clone(), description.as_ptr());
  pactffi_with_request(interaction.clone(), method.as_ptr(), path.as_ptr());
  pactffi_with_body(interaction.clone(), InteractionPart::Response, content_type.as_ptr(), body.as_ptr());
  pactffi_response_status(interaction.clone(), 200);

  let port = pactffi_create_mock_server_for_pact(pact_handle.clone(), address.as_ptr(), false);

  expect!(port).to(be_greater_than(0));

  let client = Client::default();
  let result = client.get(format!("http://127.0.0.1:{}/book", port).as_str())
    .header("Content-Type", "application/json")
    .send();

  pactffi_cleanup_mock_server(port);

  match result {
    Ok(ref res) => {
      expect!(res.status()).to(be_eq(200));
    },
    Err(err) => {
      panic!("expected 200 response but request failed: {}", err);
    }
  };

  let json: Value = result.unwrap().json().unwrap();
  let users = json.as_array().unwrap().first().unwrap().as_object()
    .unwrap().get("users").unwrap();

  if users.is_null() {
    panic!("'users' field is null in JSON");
  }
}
mefellows commented 1 year ago

Awesome work Diego - glad you had some fun and learning in the process!

Would you mind raisin an issue, with that reproducible test as part of it here: https://github.com/pact-foundation/pact-reference/issues

We can then link the issues and when the upstream issue is fixed, it will also be fixed here.

diestrin commented 1 year ago

The upstream issue was fixed on FFI v0.4.9 https://github.com/pact-foundation/pact-reference/blob/master/rust/pact_ffi/CHANGELOG.md#049---bugfix-release

mefellows commented 1 year ago

Thanks! I'll release a new version of the core that uses the FFI today.

Really appreciate your deep dive into this @diestrin! 🙏

mefellows commented 1 year ago

Fixed via https://github.com/pact-foundation/pact-js-core/releases/tag/v14.0.5.