Declarations of objects in JSONPath

[spyzhov]

Hello!

I stuck with one decision: how to evaluate object declarations in the JSONPath?

So, as an example let's take JSONPath:

$..[?(@.foo == {"foo": @.bar})]

Here we can see object declaration: {"foo": @.bar}.

Following the rules, request @.bar gives a list of elements as the result.

If @.bar gives several items in response, there is no problem (except for the order).

The problem is: when request @.bar gives only one element in response or no results ever.

Questions that I stuck with:

1. If there are no responses, should we avoid this comparison, or use null as the result?

   {"foo": @.bar} -> {"foo": null}

2. If there is only value in the response, should we avoid cast to array in this case?

   {"foo": @.bar} -> {"foo": "bar"} or {"foo": ["bar"]} ?

Who is ready to discuss it?

[cabo]

We currently don't have a way to build literals for structured (non-primitive) values.

That doesn't take away from your point:

$..[?(@.foo == @.*.bar)]

... poses the same problem.

[danielaparker]

$..[?(@.foo == @.*.bar)]

... poses the same problem.

The grammar as per the draft

rel-path     = "@" *(S (dot-selector / index-selector))

doesn't allow expressions of the form @.*.bar.

Looking at prior experience, implementations tend to group into two categories.

(1) Implementations including original JavaScript Goessner that use JavaScript (or Python or PHP) for evaluating JSONPath filter expressions. In these scripts, @ is substituted with the "current value", and expressions consisting of the "current value" followed by a dot or a left square bracket are evaluated according to JavaScript (or Python or PHP) rules. This always result in a single value.

(2) JSONPath implementations that make up their own expression languages. The important ones (Java Jayway, JSON.Net), regard expressions starting with "@" as JSONPath expressions in their own right, and evaluate them as such. They specify rules for evaluating them as a single value (rather than a list of values) so that evaluations and JSON literals can be meaningfully compared.

The grammar as per the draft, and the fact that the committee rejected a proposal by Bettio to allow JSONPath expressions beginning with "@", both suggest that the committee's thinking is motivated by (1) rather than (2).

[spyzhov]

The grammar as per the draft

rel-path     = "@" *(S (dot-selector / index-selector))

doesn't allow expressions of the form @.*.bar.

Good point, I missed this part, but what about this?

$..[?(@.foo == {"foo": $..bar})]

[spyzhov]

The grammar as per the draft

rel-path     = "@" *(S (dot-selector / index-selector))

doesn't allow expressions of the form @.*.bar.

Also, I have a question about this part. I found this comment https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-base/issues/59#issuecomment-915140548 about adding filter-selector also, but I'm not sure why do we have such restrictions?

From my point of view, technically filtering with the rel-path should be like this:

Payload:

{
  "foo": "bar",
  "baz": {
    "bar": {"biz": "bar"},
    "biz": [null, "foo", {"foo": "bar"}]
  },
}

JSONPath: $..[?(@.foo == @.baz..)]

Let's use the full path to the child to show comparisons. The filter should go through this list of comparisons:

path(@) = $
So @.foo === $.foo
Filter is TRUE if:
any([
  $.foo == $.baz,
  $.foo == $.baz.bar,          // true
  $.foo == $.baz.bar.biz,
  $.foo == $.baz.biz,
  $.foo == $.baz.biz[0],
  $.foo == $.baz.biz[1],
  $.foo == $.baz.biz[2],
  $.foo == $.baz.biz[2].foo, // true
])

That was my thought. Based on that, I don't see any problems using any other selection, such as dot-wild-selector or any other.

Am I right? Or there are any restrictions that I don't know about?

[gregsdennis]

If we were to consider something, it would certainly come in a later version of the spec. I don't think this sort of syntax needs to be considered as part of the initial release.

Feel free to continue discussing it, though.

[danielaparker]

what about this?

$..[?(@.foo == {"foo": $..bar})]

As for existing implementations, I don't think there are any that allow path expressions inside JSON literals, none that allow {"foo": $..bar} as an operand.

[danielaparker]

Also, I have a question about this part. I found this comment #59 (comment) about adding filter-selector also, but I'm not sure why do we have such restrictions?

It's a good question. In terms of prior experience, the distinction is largely between (1) implementations that followed the original JavaScript Goessner, that use JavaScript (or Python or PHP) for evaluating JSONPath filter expressions, where expressions beginning with "@" are not JSONPath expressions, and (2) implementations such as Jayway that made up their own expression language, where expressions beginning with "@" are JSONPath expressions.

In the draft, there is a made up expression language for filters, but expressions beginning with "@" are not JSONPath expressions.

[spyzhov]

@gregsdennis, sure! But I suppose we have to consider it. The main question here is not just "how to use rel-path in request", but much wide: specification of the internal script language.

We are already able to use requests like $[!(@.length - 1)] to get the last element from the list, but what is the @.length - 1 for sure? It is the modification of the given variables. So it could be @.foo - 1 as an example. We already "know", what is the @.foo here, we have at least part of the specification here, but the second question: what is 1 in this request? And it is just literal. So we met the third and main question here: what types of data and literal particularly do we have? Data types are: null, numeric, bool, string, array and object. It's logical to assume, that literals will have the same list of types.

And now we face the problem: we know how to create and modify "simple" types, like string or numeric, but what about complex? Should we be able to compare arrays? Like this: ?(@.array == [1, 2, 3]). Probably yes. That means that we should be able to create literal with the array type.

Should we be able to create a variable numeric type? Sure! Request @.length - 1 is the variable numeric value.

And what about variable array or object type? What about request: ?(@.array == [@.foo, @.bar, @.baz])?

Most of those questions are not about JSONPath itself, but script mostly. And yes, I suppose in the initial release it can be avoided.

@danielaparker, Ok, I see... Currently, I should deeper investigate the specification and understand why is @ is not the JSONPath expression. From my point of view, it should be the same, so I have to learn about it more.

Regarding this:

As for existing implementations, I don't think there are any that allow path expressions inside JSON literals, none that allow {"foo": $..bar} as an operand.

Yes. And that is the biggest problem for me.

Creating ajson library, I thought about the possibility to get atomic values from the whole payload and modify only them. But today I'd like to wide this possibility to be able to create values through request. To be able to use a scripting language from the JSONPath to create new payloads, like this:

ajson.JSONPath(`!({
  "foo": $..foo,
  "bar": {$..bar: $..baz}
})`, data)

So, if script request !() could be used outside of the filter-request it can push JSONPath to a new level.

But! That is only my point of interest, so I will not push spec to follow this way.

[danielaparker]

Creating ajson library, I thought about the possibility to get atomic values from the whole payload and modify only them. But today I'd like to wide this possibility to be able to create values through request. To be able to use a scripting language from the JSONPath to create new payloads, like this:

ajson.JSONPath(`!({
  "foo": $..foo,
  "bar": {$..bar: $..baz}
})`, data)

It's hard to see that capability coming to JSONPath, as JSONPath is generally seen as a language for selecting values from a JSON document, and not for transforming JSON.

There are other query languages that support transformation. You can do this in JMESPath, a query language with support in both the AWS and Azure CLI, and libraries available in a number of languages. In JMESPath, the query

{"foo": baz.biz[2].foo, "bar": foo}

applied to the document

{
  "foo": "bar",
  "baz": {
    "bar": {"biz": "bar"},
    "biz": [null, "foo", {"foo": "bar"}]
  }
}

gives the result

{
  "foo": "bar",
  "bar": "bar"
}

You can also do it in JSONiq, an ISO/IEC approved, OASIS standard. JSONiq was largely incorporated into XQuery 3.1, in a way that integrates the XML and JSON data models. The big vendors have mostly moved to XQuery for querying and transforming data, whether XML or JSON, IBM WebSphere still uses an older version of JSONiq. It's difficult to see a role for JSONPath here.

[timbray]

Useful discussion, thanks.