Clarify behavior of jsonpath equality with objects?

[danielbprice]

I have two API services, call them "OLD" and "NEW". NEW is supposed to take over the duties of OLD, and must reproduce the API provided by OLD (but these are JSON apis, and the keys may appear in any order). I was hoping to write some hurl tests like this:

GET {{old_url}}/api/widgets

HTTP/* 200
[Captures]
old_resp: jsonpath "$"

GET {{new_url}}/api/widgets

HTTP/* 200
[Asserts]
jsonpath "$" == {{old_resp}}

In this case, both old and new return responses like: [{ ... }, { ... }, { ... }].

This seems to detect /some/ differences between old and new-- such as the number of items in the top-level collection. But I had hoped for a deeper comparison, to spot subtle differences between the APIs.

I think that at a minimum, the documentation should clarify the limitations of the equality test. But ideally, it would do what it says on the tin-- are these JSON objects the same?

For what it's worth, I am mostly using hurl to work with live data in our development environment, rather than as a unit testing tool. So I don't necessarily know in great detail what the API responses will look like, exactly, ahead of time. I realize this may be somewhat outside of the design center.

I did pull hurl from HEAD, instead of using v1.61, and I notice that the output around captures is a lot clearer, which helped me to understand much more clearly what was happening. Thanks for that!

[jcamiel]

Agreed, currently jsonpath assert does work well on json nodes, we need to improve it when comparing objects. Definitively on the todo list, we’ve other issues around improving assertion (#272 #347) and supporting « semantic » equality for json.

[fabricereix]

Thanks @danielbprice for your remarks. This is a very interesting and challenging area to work on. We need to collect a lot of examples.

In your use case, what type of JSON responses do you have? What type of output would you expect from Hurl?

[danielbprice]

@fabricereix The responses in this case are for a very typical REST API. Just to give one example from about 6 that I have: We're getting a list of information about groups of events which have happened inside of our system; we call these groups "sessions." Each session has an ID, as well as a variety of other attributes, some nested:

[
  {
    "id": "af89d5a6-61e0-4097-a1a2-c2d4b276df15",
    "origin": "36b9a4b3-bdf1-4240-a66a-0257808e7fd5",
    "started": "2022-08-24T00:09:33Z",
    "completed": "2022-08-24T03:22:25",
...
    "attachments": {
      "exceptions": 0,
      ...
    },
    "extended-attributes": {
      "code": 27,
      ...
    }
  }
]

[fabricereix]

We could display the expected object/list value on one line, and the first diff in the nested data structure on the line below.

For example for the expected list

[{"name": "item1"}, {"name": "item2"}, {"name": "item3"}, {"name":"item4"}, {"name":"item5"}, {"name": "item6"}, {"name":"item7"}]

if the item4 is missing in the actual list, the following error message could be displayed

   |
30 | jsonpath "$" == {{expected}}
   |   expected:  [{"name": "item1"}, {"name": "item2"}, {"name": "item3"}, {"name": "item4" ...
   |                                                                                 ^^^^^^^ actual is "item5"

With the default line-wrapping in the terminal, we should also see a first diff far in the JSON as follow. for example

   |
30 | jsonpath "$" == {{expected}}
   |   expected:  [{"name": "item1"}, {"name": "item2"}, {"name": "item3"}, {"name": "item4"}, {"name": "item5"}, {"nam
e": "item6"}, {"name": "item7"}, {"name": "item8"},  {"name": "item9"}, {"name": "item10"}, {"name": "item11"}, {"name"
: "item12"}, {"name": "item13" ...
   |                          
                       ^^^^^^^ actual is "item14"
   |

We could also indent the expected value

   |
30 | jsonpath "$" == {{expected}}
   |   expected:  [
   |     {"name": "item1"}, 
   |     {"name": "item2"}, 
   |     {"name": "item3"}, 
   |     {"name": "item4"}, 
   |     {"name": "item5"}, 
   |     {"name": "item6"}, 
   |     {"name": "item7"}, 
   |     {"name": "item8"},  
   |     {"name": "item9"}, 
   |     {"name": "item10"}, 
   |     {"name": "item11"}, 
   |     {"name": "item12"}, 
   |     {"name": "item13" ...
   |              ^^^^^^^ actual is "item14"  
   |