how to create JSON Assertion?

[mikeliucc]

Hi,

First, great project - THANK YOU!

I see that there's jsonExtractor() in JmeterDsl. Is there a JSON Assertion SamplerChild that we can use, something like this?

Thanks!

[rabelenda]

Hello, thank you for asking this and value the project!

Currently there is no jsonAssertion, but we should definitely add one.

What do you think of something like this:

jsonAssertion(jmesPath) // check for existence

jsonAssertion(jmesPath).matches(regex) // check for value matching regex

jsonAssertion(jsonPath).queryLanguage(JsonQueryLanguage.JSON_PATH) // use jsonpath instead of jmespath (the later being better specified, and using same defaults as jsonExtractor)

jsonAssertion(jmesPath).equalsTo(value) // can be used to check a value or null

jsonAssertion(jmesPath).not().equalsTo(value) // a way to invert checks

[mikeliucc]

Pretty good!

A few small additions, in addition to what you've suggested:

jsonAssertion(name, jmesPath) // check for existence

jsonAssertion(name, jmesPath).matches(regex) // check for value matching regex

jsonAssertion(name, jsonPath).queryLanguage(JsonQueryLanguage.JSON_PATH) // use jsonpath instead of jmespath

jsonAssertion(name, jmesPath).equalsTo(value) // can be used to check a value or null

jsonAssertion(name, jmesPath).not().equalsTo(value) // a way to invert checks

[rabelenda]

Yup, I missed those, thanks :).

[rabelenda]

We just released a new version which includes jsonAssertion.

Please try it and let us know what you think.

Regards

[mikeliucc]

Thanks for the quick turnaround!

I just tried it and for the most part it works. 2 comments, if you would entertain them...

1. asserting numeric value vs string value Here's how I would write the assertion for an expected numeric value:

   jsonAssertion("price", "$.price").queryLanguage(JSON_PATH).equals("${expected_price}")

   and for text assertion:

   jsonAssertion("name", "$.name").queryLanguage(JSON_PATH).equals("\"${expected_name}\"")

   While this is entirely correct, it would appear to be more idiomatic and more convenient to be able to do without the escaped double quotes, like:

   jsonAssertion("name", "$.name").queryLanguage(JSON_PATH).equals("${expected_name}")

   Your thoughts on this?

2. Null value or nonexistent path At times the same JSON path would return null value or simply not exists in the request payload - as per design of said API. Since we are using CSV to map our expected value as variable, how can we handle such scenario? Suppose ${expected_count} derived from an empty cell of my test data CSV,

   jsonAssertion("count", "$.count").queryLanguage(JSON_PATH).equals("${expected_count}")

   The above would throw an error with the message "Assertion failure message:No results for path:..." How could we assert that $.count either results in no path found or a null value? I can't use if/else in my Java code since I'm using CSV-derived variables. Is JSR223 the only way (hope not..)?

[mikeliucc]

... another typo (I think...)

sorry, don't mean to nitpick...

[rabelenda]

sorry, don't mean to nitpick...

Not a problem at all. In fact I value that you point this things 😄 .

You can also submit PR when you find fixes. They are quite easy to fix, and you would be contributing more to the tool development.

[rabelenda]

sorry, don't mean to nitpick...

Not a problem at all. In fact I value that you point this things 😄 .

... another typo (I think...)

Fixed, thanks :)

[mikeliucc]

Thanks for the quick turnaround!

I just tried it and for the most part it works. 2 comments, if you would entertain them...

1. asserting numeric value vs string value Here's how I would write the assertion for an expected numeric value:

   jsonAssertion("price", "$.price").queryLanguage(JSON_PATH).equals("${expected_price}")

   and for text assertion:

   jsonAssertion("name", "$.name").queryLanguage(JSON_PATH).equals("\"${expected_name}\"")

   While this is entirely correct, it would appear to be more idiomatic and more convenient to be able to do without the escaped double quotes, like:

   jsonAssertion("name", "$.name").queryLanguage(JSON_PATH).equals("${expected_name}")

   Your thoughts on this?

2. Null value or nonexistent path At times the same JSON path would return null value or simply not exists in the request payload - as per design of said API. Since we are using CSV to map our expected value as variable, how can we handle such scenario? Suppose ${expected_count} derived from an empty cell of my test data CSV,

   jsonAssertion("count", "$.count").queryLanguage(JSON_PATH).equals("${expected_count}")

   The above would throw an error with the message "Assertion failure message:No results for path:..." How could we assert that $.count either results in no path found or a null value? I can't use if/else in my Java code since I'm using CSV-derived variables. Is JSR223 the only way (hope not..)?

@rabelenda - any thoughts on the above?

[rabelenda]

1. asserting numeric value vs string value Here's how I would write the assertion for an expected numeric value:

   jsonAssertion("price", "$.price").queryLanguage(JSON_PATH).equals("${expected_price}")

   and for text assertion:

   jsonAssertion("name", "$.name").queryLanguage(JSON_PATH).equals("\"${expected_name}\"")

   While this is entirely correct, it would appear to be more idiomatic and more convenient to be able to do without the escaped double quotes, like:

   jsonAssertion("name", "$.name").queryLanguage(JSON_PATH).equals("${expected_name}")

   Your thoughts on this?

I don't know if understand your point. But I did some tests considering your question, and found out that JMESPath and JSONPath assertions are not consistent (due to JMeter elements difference in behavior) when handling String JSON properties.

For instance I see that JSONPath Assertion fails if your json contains a string property which contains a number (eg: {"prop":"1"}) and asserting with jsonAssertion("$.prop").queryLanguage(JSON_PATH).equalsTo("1"). JMESPath Assertion does not fail in such scenario.

If you use quotes in the assertion (eg: jsonAssertion("$.prop").queryLanguage(JSON_PATH).equalsTo("\"1\""), or even .equalsTo("\"${var}\""), the assertion doesn't fail. And in general, to make your assertion robust to any string you would need to quote them to avoid such potential issue. And I guess that's probably what you are referring to, and you mean that it would be nice to not have to add the additional quotes.

Did I get your point?

If so, I think we could implement some basic logic/fix. Accept in equalsTo an Object instead of String, and add a equalsToJson(String).

• equalsTo(Object) would, with this change, convert given value to JSON for the assertion element to use. You could even pass ints and bools without quoting, or even an array, list or map, or even a generic object (that is JacksonSerializable).
  • Take into consideration that if you want to assert an int or boolean property with a variable (eg: {"prop", 1} jsonAssertion("prop").equalsTo("${var}")), it will fail, since it will interpret by default the variable as String variable, and you would need to use equalsToJson("${var}") instead.
  • Sadly there is no simple way to solve this at DSL code since it only controls test plan construction (since contents of JSON are not necessarily known at test plan creation and element might as well be an int or a string), not test plan execution itself (which is controlled by JMeter, and is where element property value type info is available). One option could be to not use existing JSON Assertion JMeter elements and decide to implement a new one, or use JSON Extraction + Response Assertion as replacement. But I think that would be overkill for such border scenarios, requiring much logic and making test plans harder to understand for potential JMeter users and less portable to run them at scale.
• If you want to pass the actual expected json, without any transformations, then you can use equalsToJson(String), which just passes the string value to the JMeter test element as is. For instance you can use it for .equalsToJson("1") or .equalsToJson("${var}") for asserting a number,.equalsToJson("\"1\"") or .equalsToJson("\"${var}\"") for asserting a string, or use more complex scenarios like .equalsToJson("[1, 2]") or `.equalsToJson("{\"nestedProp\":1}").

What do you think?

[mikeliucc]

Thanks for your reply, your detailed analysis, and your suggested solution!

Certainly looks like equalsToJson(String) could resolve a variety of problems, at least for what I'm working with here. I would venture to think that these problems might not be as "border scenarios" for some folks. In our case, we generate the CSV test data from a sanitized production dataset.

Would you consider adding to your suggested equalsToJson(String) solution the following:

1. handling null data
   • for example, jsonAssertion("$.prop").queryLanguage(JSON_PATH).equalsToJson("${var1}") and ${var1} is null (derived from CSV data set). In some cases, the actual response payload from the invoked API might not even contain prop property since some server-side JSON serialization frameworks remove null property to optimize network transport -- would it be possible to handle such scenario gracefully?
2. numeric equivalency
   • for example, $.prop returns a 1.0 but the variable used for assertion ${var1} is int 1. Could we treat such comparison as the same (numerically equivalent)?

I apologize if I'm asking too much... :-( I'm reading/learning more about the source code so hopefully I can contribute more effectively in the future (and be less demanding)

Thanks for your consideration!

[rabelenda]

2. Null value or nonexistent path At times the same JSON path would return null value or simply not exists in the request payload - as per design of said API. Since we are using CSV to map our expected value as variable, how can we handle such scenario? Suppose ${expected_count} derived from an empty cell of my test data CSV,

    jsonAssertion("count", "$.count").queryLanguage(JSON_PATH).equals("${expected_count}")

The above would throw an error with the message `"Assertion failure message:No results for path:..."`
How could we assert that `$.count` either results in no path found or a null value? I can't use if/else in my Java code since I'm using CSV-derived variables. Is JSR223 the only way (hope not..)?

As far as I understand your API behaves in "non consistent way" (some times a property with null and sometimes without the property). In such cases, I think it doesn't make sense to make a general built in solution in JMeter DSL (like having a nullOrNonExisting in `jsonAssertion), since it an api design specific issue, and not something general for all JSON APIs.

One option is using jsonExtractor + responseAssertion.

Here is an example:

testPlan(
        vars().set("EXPECTED", "null"), // this is analogous to csvDataSet setting the variable
        threadGroup(1, 1,
            dummySampler("{\"prop\": null}"),
            dummySampler("{}"),
            jsonExtractor("VAL", "prop")
                .defaultValue("null"),
            responseAssertion()
                .scopeVariable("VAL")
                .equalsToStrings("${EXPECTED}")
        )
    ).run();

Another alternative could be just using responseAssertion with regex. Eg:

testPlan(
        vars().set("EXPECTED", "null"), // this is analogous to csvDataSet setting the variable
        threadGroup(1, 1,
            dummySampler("{\"prop\": null}"),
            dummySampler("{}"),
            responseAssertion()
                .containsRegexes("\"prop\": (?!null)")
                .invertCheck()
        )
    ).run();

What do you think? How would you use jsr223 in such scenario? Replacing jsonAssertion with jsr223PostProcessor and parsing JSON in jsr223 and changing sample result accordingly (status and potential addition of assertion)?

One thing we may add, and haven't yet, is adding support for jsr223Assertion, which would simplify the usage over using jsr223PostProcessor, but in this scenario I think is just simpler to use jsonExtractor + responseAssertion.

[rabelenda]

One option is using jsonExtractor + responseAssertion.

You can even abstract this logic to a method, eg:

private static int jsonPropIndex = 1;

  public MultiLevelTestElement[] assertJsonProperty(String assertionName, String propPath,
      String expectedValue) {
    String varName = "JSON_PROP_" + (jsonPropIndex++);
    return new MultiLevelTestElement[]{
        jsonExtractor(varName, propPath)
            .defaultValue("null"),
        responseAssertion(assertionName)
            .scopeVariable(varName)
            .equalsToStrings(expectedValue)
    };
  }

  @Test
  public void testWithExtractorAndAssertion() throws Exception {
    TestPlanStats stats = testPlan(
        vars().set("EXPECTED", "null"), // this is analogous to csvDataSet setting the variable
        threadGroup(1, 1,
            dummySampler("{\"prop\": null}")
                .children(
                    assertJsonProperty("prop Assertion", "prop", "${EXPECTED}")
                ),
            dummySampler("{}")
                .children(
                    assertJsonProperty("prop Assertion", "prop", "${EXPECTED}")
                )
        )
    ).run();
    assertThat(stats.overall().errorsCount()).isEqualTo(0);
  }

Or create your custom DslAssertion:

public static class MyJsonAssertion extends BaseTestElement implements DslAssertion {

    private static int jsonPropIndex = 1;
    private final String assertionName;
    private final String path;
    private final String value;

    public MyJsonAssertion(String assertionName, String path, String value) {
      super(null, null);
      this.assertionName = assertionName;
      this.path = path;
      this.value = value;
    }

    @Override
    public HashTree buildTreeUnder(HashTree parent, BuildTreeContext context) {
      String varName = "JSON_PROP_" + (jsonPropIndex++);
      jsonExtractor(varName, path)
          .defaultValue("null")
              .buildTreeUnder(parent, context);
      return responseAssertion(assertionName)
              .scopeVariable(varName)
              .equalsToStrings(value)
          .buildTreeUnder(parent, context);
    }

    @Override
    protected TestElement buildTestElement() {
      // Since we have already overwritten buildUnderTree this method will actually not be invoked
      return null;
    }

  }

  @Test
  public void testWithExtractorAndAssertion() throws Exception {
    TestPlanStats stats = testPlan(
        vars().set("EXPECTED", "null"), // this is analogous to csvDataSet setting the variable
        threadGroup(1, 1,
            dummySampler("{\"prop\": null}"),
            dummySampler("{}"),
            new MyJsonAssertion("prop Assertion", "prop", "${EXPECTED}")
        )
    ).run();
    assertThat(stats.overall().errorsCount()).isEqualTo(0);
  }

[mikeliucc]

ah... that's neat! Let me give that a try right now

[rabelenda]

I apologize if I'm asking too much... :-( I'm reading/learning more about the source code so hopefully I can contribute more effectively in the future (and be less demanding)

Don't apologize 😄 . Your questions and comments help review design decisions and develop a better solution for everyone. Thank you.

Would you consider adding to your suggested equalsToJson(String) solution the following:

I was considering providing equalsToJson(String) as a way to provide the "raw" expected value, so extracted property would need to match exactly the provided string. On the other hand equalsTo(Object) would ease comparing simple values like strings, numbers, booleans, and even list, arrays, maps and complex classes, by serializing provided json value and use Jmeter assertion element with such value. (eg: transforming "name" to "\"name\"", 1 to "1", false to "false", new int[]{1, 2, 3} to "[1, 2, 3]", Collections.singletonMap("prop",1) to "{\"prop\": 1}", and so on ).

1. handling null data

   • for example, jsonAssertion("$.prop").queryLanguage(JSON_PATH).equalsToJson("${var1}") and ${var1} is null (derived from CSV data set). In some cases, the actual response payload from the invoked API might not even contain prop property since some server-side JSON serialization frameworks remove null property to optimize network transport -- would it be possible to handle such scenario gracefully?

Is this the same scenario as you previously described and I proposed to use extractor + assertion and other alternatives? If so, as I previously discussed, since it sound like an API specific thing, because in most APIs I would expect the API to be consistent (either send the property with null value or not send it at all, but not some times send it, and some times not, depending on the particular property or endpoint), I think makes sense to not build a DSL specific method, and just use some workaround as the ones proposed for such scenarios.

2. numeric equivalency

   • for example, $.prop returns a 1.0 but the variable used for assertion ${var1} is int 1. Could we treat such comparison as the same (numerically equivalent)?

Changing this would require not using JMeter provided Json Assertion elements. But I think, that even though it might make the api simpler (don't have to remember putting a decimal when API returns decimals), specifying the actual expected value is better (.equalsTo(1.0) or .equalsToJson("1.0")) and simpler. If you need to normalize values for your specific API (maybe API for some props sometimes uses decimals and sometimes uses ints), then I would advice using a similar approach as the one provided for nulls or inexisting properties: use extractor + postProcessor (for value normalization) + responseAssertion, or extractor + responseAssertion with regex that supports both decimals and ints, or just responseAssertion with regex matching prop name and value with both decimals and ints

[mikeliucc]

ok, using your suggestion works for me. I ended writing my own DSL so I can learn more about this project (which I wanted to do anyways).

We standardized to use JsonPath, so here's what mine looks like:

public class JsonExtendedAssertion extends BaseTestElement implements DslAssertion {
    private final String path;
    private final String value;

    public JsonExtendedAssertion(String name, String jsonPath, String value) {
        super(name, null);
        this.path = jsonPath;
        this.value = value;
    }

    @Override
    public HashTree buildTreeUnder(HashTree parent, BuildTreeContext context) {
        String varName = name + " var_" + RandomUtils.nextInt(1000, 99999);
        jsonExtractor(varName, path).queryLanguage(JSON_PATH).defaultValue("null").buildTreeUnder(parent, context);
        return responseAssertion(name).scopeVariable(varName).equalsToStrings(value).buildTreeUnder(parent, context);
    }

    @Override
    protected TestElement buildTestElement() {
        // Since we have already overwritten buildUnderTree this method will actually not be invoked
        return null;
    }
}

Now that I'm a little more familiar with how things work, I think most of my previous inquiries regarding JSON assertion are probably irrelevant now. Well, except this one...

Numerical equivalency

In our case (and perhaps for others as well), we are testing against external API to which we don't have much control or say in the responses. It'd be great if the response is consistent in data type, but alas it's not (a story for another time, perhaps). So it looks like regex might be my only solution in this case. Unless something like the following exists or can be implemented...?

jsonAssertion(name, path)
    .defaultValue("null")
    .formatAsDecimal(true)    // magic here
    .equalsTo("${var_formatted_as_decimal}")

or

jsonAssertion(name, path)
    .defaultValue("null")
    .equalsTo("${var_as_int}", "${var_as_decimal}")    // match any

BTW, it's getting fun writing my perf. tests with this project! :-)

[rabelenda]

BTW, it's getting fun writing my perf. tests with this project! :-)

Awesome! that is a big part of the motivation of creating the library, to be simple & straightforward as to fill fun and efficient while writing and executing test plans. Don't have to worry too much on JMeter internals and try to aliviate also Java potential verbosity.

Numerical equivalency Since you have opted for the option of creating your own assertion (which I think is the most reusable one, since you might even generate your own jar, to share in different projects in your company, and build into it any other potential logic you might need), you might also add to it the decimals normalization process. Eg:

public class JsonExtendedAssertion extends BaseTestElement implements DslAssertion {
    private final String path;
    private final String value;

    public JsonExtendedAssertion(String name, String jsonPath, String value) {
        super(name, null);
        this.path = jsonPath;
        this.value = value;
    }

    @Override
    public HashTree buildTreeUnder(HashTree parent, BuildTreeContext context) {
        String varName = name + " var_" + RandomUtils.nextInt(1000, 99999);
        jsonExtractor(varName, path).queryLanguage(JSON_PATH).defaultValue("null").buildTreeUnder(parent, context);
        jsr223PostProcessor(s -> {
          String val = s.vars.get(varName);
          if (val.matches("^\\d+\\.0+$")) {
             s.vars.put(varName, val.substring(0, val.indexOf(".")));
          }
        }).buildTreeUnder(parent, context);
        return responseAssertion(name).scopeVariable(varName).equalsToStrings(value).buildTreeUnder(parent, context);
    }

    @Override
    protected TestElement buildTestElement() {
        // Since we have already overwritten buildUnderTree this method will actually not be invoked
        return null;
    }
}

One additional comment: I would use just a static auto-increment instead of Random value for variables names, to avoid any potential unintended conflicts, and also to make it easer to trace issues (is easier to predict with an auto increment the position of the element in a test plan, than with random). Final note: The length of the variable names might impact your test plan memory consumption. This might be relevant, or not, depending on the JVM you use and the amount of variables you create. There are some some alternatives which might help reducing memory footprint, but I think they only make sense if you reach a point where memory consumption is problematic.

[mikeliucc]

Got it! Thanks for the explanation. I updated my code, worked beautifully!

[rabelenda]

Hello @mikeliucc . We just released a new version which implements the changes in equalsTo and adds equalsToJson. If you are already using equalsTo you might need to change the test plan due to this change.

Ideally this type changes (since they might change behavior for existing users) would require a major version release (eg: 2.0), but I decided to release it just as a minor version since the feature is less than 1 week old and I guess only you and potentially few people (since the release) are using it.

Regards