Match each not contains / match contains fails unexpectedly for partial objects

[tomwetjens]

There appears to be an issue with the JSON array contains / not contains step, when trying to partially match objects.

Given the following scenario:

Feature: Bug Report
  Scenario: Each not contains
    * def ar = [{a: 1, b: 2}, {a:3, b:4}]
    * match each ar !contains {a:1, b:4}

This fails with: com.intuit.karate.exception.KarateException: path: $[0].a, actual: 1, NOT expected: 1, reason: actual value contains expected

While I would expect this to pass.

Other scenario:

Feature: Bug Report
  Scenario: Array contains
    * def ar = [{a: 1, b: 2}, {a:3, b:4}]
    * match ar contains {a:1}

This fails with: com.intuit.karate.exception.KarateException: path: $[*], actual: [{"a":1,"b":2},{"a":3,"b":4}], expected: {a=1}, reason: actual value does not contain expected

While I would expect this to pass.

[ptrthomas]

Think about it, you are trying two levels of contains. The array. And then the "partial" JSON. And you are mixing each into all this. Please read this section in the documentation carefully: https://github.com/intuit/karate#contains-short-cuts

So your first example should be:

* def part = {a: 1, b: 4}
* def ar = [{a: 1, b: 2}, {a: 3, b: 4}]
* match each ar != '#(^part)'
* match ar !contains '#(^part)'

And your second:

* def part = {a: 1}
* def ar = [{a: 1, b: 2}, {a: 3, b: 4}]
* match ar contains '#(^part)'

Also refer to this answer on Stack Overflow: https://stackoverflow.com/a/51285925/143475

[tomwetjens]

Thanks for your answer. However the syntax you are pointing out does not feel very idiomatic to me. IMO the behavior of 'contains' seems different when using it on non-arrays vs arrays with 'each'. This led me to expecting the above outcome. Maybe something can be improved here?

[ptrthomas]

@tomwetjens sure ! can you provide suggestions :) also point me to any framework that does this better.

[ptrthomas]

@tomwetjens my apologies, you indeed have found a bug, I will look into this. here's the simplified case, which when fixed should also fix the problem you refer to.

* def foo = {a: 1, b: 2}
* match foo !contains {a: 1, b: 4}

05:58:30.616 [main] ERROR com.intuit.karate - assertion failed: path: $.a, actual: 1, NOT expected: 1, reason: actual value contains expected

[tomwetjens]

@ptrthomas no problem :-) Really happy with Karate in general, very little bugs and good documentation. We have a workaround for now.

[ptrthomas]

fixed in dev, now this also works:

* def ar = [{a: 1, b: 2}, {a: 3, b: 4}]
* match each ar !contains {a: 1, b: 4}

[ericdriggs]

@tomwetjens sure ! can you provide suggestions :) also point me to any framework that does this

When I'm validating sparse fields in nested json, I'm a fan of the syntax in JsonUnit, especially with the IGNORING_EXTRA_FIELDS and IGNORING_ARRAY_OPTIONS. That way I can specify only the fields I wish to match against, and use inline regex when needed for bounded non-deterministic data (e.g. time).

A sparse assertion preserves the structure and readability of the expected json so much better that I only use match if I only need to assert on 1 or 2 values.

https://github.com/lukas-krecan/JsonUnit

usage: net.javacrumbs.jsonunit.JsonAssert.assertJsonEquals(expected, actual, when(IGNORING_EXTRA_FIELDS, IGNORING_ARRAY_ORDER));

[ptrthomas]

A sparse assertion preserves the structure and readability of the expected json so much better

@ericdriggs kindly provide the equivalent for the below in JsonUnit:

* def part = {a: 1}
* def ar = [{a: 1, b: 2}, {a: 3, b: 4}]
* match ar contains '#(^part)'

[ericdriggs]

jsonunit.java

import static net.javacrumbs.jsonunit.core.Option.; import static net.javacrumbs.jsonunit.JsonAssert.;

@SuppressWarnings({"unused", "WeakerAccess"}) public class JsonUnit {

public static void assertJsonEqualsIgnoringExtra(final String expected, final String actual) {
    assertJsonEquals(expected, actual, when(IGNORING_EXTRA_FIELDS, IGNORING_ARRAY_ORDER));
}

}

jsonunit.feature

Feature:

Background:

• def JsonUnit = Java.type('JsonUnit')

  Scenario: match part

  for matching arrays, will need to match on count

• def part = [{a: 1}, {}]

• def ar = [{a: 1, b: 2}, {a: 3, b: 4}]

  lvalue for assert only there to satisfy lvalue syntax requirement -- method is void

• def assert = JsonUnit.assertJsonEqualsIgnoringExtra(part, ar )

• def part2 = [ {a:1}, {a:3} ]

• def assert = JsonUnit.assertJsonEqualsIgnoringExtra(part2, ar )

[ericdriggs]

More detailed scenarios:

Scenario: match deeply nested:

#text preferred for java interop since auto-cast from def to java string will manage as {I=[{"A":"I.A","B":"I.B"}], II={A=II.A, B=II.B}, III={A={1={a={i=III.A.1.a.i., ii=III.A.1.a.ii}}}, B={1={a={i=III.B.1.a.i., ii=III.B.1.a.ii}}}}}
* text deepJson =
"""
{
  "I" : [
     { "A" : "I.A", "B" : "I.B"},
     { "C" : "I.C", "D" : "I.D"}
  ],
  "II" : {
    "A": "II.A",
    "B": "II.B"
  },
  "III": {
     "A": {
        "1" : {
          "a" : {
            "i": "III.A.1.a.i.",
            "ii": "III.A.1.a.ii"
          }
        }
     },
     "B": {
        "1" : {
          "a" : {
            "i": "III.B.1.a.i.",
            "ii": "III.B.1.a.ii"
          }
        }
     }
  }
}
"""

* text sub =
"""
{
  "I" : [
     { "A" : "I.A"}, {}
  ],
  "II" : {
    "B": "II.B"
  },
  "III": {
     "A": {
        "1" : {
          "a" : {
            "i": "III.A.1.a.i."
          }
        }
     }
  }
}
"""
* def assert = JsonUnit.assertJsonEqualsIgnoringExtra(sub, deepJson )

Scenario: validate date matches expected format

* text jsonWithDate =
"""
{
  "date": "2018-10-29T16:36:02Z"
}
"""

* text expected =
"""
{
  "date": "${json-unit.regex}[0-9]{4}-[0-9]{2}-[0-9]{2}.*"
}
"""

* def assert = JsonUnit.assertJsonEqualsIgnoringExtra(expected, jsonWithDate )

[ptrthomas]

?

@ericdriggs lol ok thanks, let's leave it to readers to choose. I really don't understand what's going on in the second example, but thanks anyway.

[ericdriggs]

Most of the use cases I have for json validation are validating about 5-10 fields at different levels of nesting from payloads that could be hundreds of lines long. One sparse matching meets that use case better than 5-10 one-line matchers.

[ptrthomas]

@ericdriggs IMHO I really don't think you have understood Karate's full capabilities. for example fuzzy and schema-like validation - there are multiple sections in the doc starting here: https://github.com/intuit/karate#fuzzy-matching

but all the best with the so-called "sparse matching" ;)

[ptrthomas]

@ericdriggs specifically - besides the fact that you can read JSON from a file (which includes fuzzy placeholders, see docs above) and perform a validation in one line - I think you have completely missed these 2 concepts:

a) you can un-pluck a sub-tree of a complex JSON into a variable to simplify validations b) you can use JsonPath to make the above step even more flexible

[ericdriggs]

Thanks for taking the time to reply and the great work you're doing on this tool. The json responses I'm dealing with are often over 100 lines long, and I usually only need to assert on about 5-10 values scattered throughout the payload in multiple subtrees, and each subtree has 5-10 sibling values which don't need to be asserted on. Maybe that doesn't match the use cases of most of your users.

I definitely see the value of adding schema validation for contract coverage, and I use karate fuzzy matching and jsonpath often when asserting on a few values. One of my favorite things about karate is the java interop. If you have a specialized need, it's easy to extend the functionality. Thanks again.

[ptrthomas]

I usually only need to assert on about 5-10 values scattered throughout the payload in multiple subtrees, and each subtree has 5-10 sibling values which don't need to be asserted on.

no worries. I tried. cheers.

[ptrthomas]

0.9.0 released