IETF Fwd: Some Comments ...

Hello List,

It has been important to go through this list threads carefully. In fact I should have done that at first. Now I can understand the current draft and appreciate the work already done much better.

I collected some citations (important from my point of view) with comments already in Markdown.

Title of the specification

JSONPath: A query language for JSON data. (Carsten Bormann)

I think I’d slightly prefer the term “syntax” to “language” because “query language” has a smell of various things that end with the letters “Q” and “L”. But not passionate about that. (Tim Bray)

JSONPath: A query syntax for JSON. Another wild-card idea: JSONPath: Query expressions for jSON (Tim Bray)

The beauty of this is that the plural form “query expressions” implies a set of expressions, so it implies “language”. It’s indeed more than the grammar/syntax of those, so why not talk about the expressions as a whole. This also makes it possible to just use “for JSON”, without going into detail what these query expressions operate on. (Carsten Bormann)

There seems to be an agreement for "JSONPath: Query expressions for JSON". I like that also.

Terminology

My own view is that the terminology should stay consistent with RFC 8259, and that the word "object" should not be used for items that are not JSON objects in the sense of RFC 8259. (Daniel P)

To Carsten's point about what we call things, the number of distinguished terms per RFC8259 is pretty small: JSON text, value, object, array, number, string. Having spent quite a bit of time specifying JSON DSLs, I find that using just those terms doesn't seem to get in the way or cause problems, so I'd argue that we should stick to them (and build up to higher-level constructs as required for JSONPath).

… oh, and I forgot the very useful "member". (Tim Bray)

… and “element” (the things in arrays). (Carsten Bormann)

The problem with JSON value is that it also can be quite confusing due to the usual use of that term. Pointing to a tree and saying “the values inside that tree” is not going to be felt as equivalent to “the set of all subtrees of that tree, including the tree itself”. But if JSON value is the only term we have, it has to be. Hence my preference to talk about data items when I mean the items themselves and not their “value”. (Carsten Bormann)

I think the key difficulty is whether each (key, value) pair in an object is "a thing" that can be identified and manipulated and potentially returned. (If we're talking analogies, then it's analogous to an attribute node in the XDM model). (Michael Kay)

ECMA-404 uses "name/value pair", which is what I understand the term "member" to mean (Douglas Crockford uses "member"). (Daniel P)

I think the term “union” is poor. If we think of it as concatenation of results, then the result is as expected. (Glyn Normington)

I understand, that within RFC8259 we have JSON values of different types. They are structured somehow, which is not so much of interest here.

But while querying that structure with JSONPath it is vitally important to identify that hierarchical structure as a tree. So in fact we build up a higher-level construct here. We also need to call "the things" in the tree somehow. I was able to identify

"node" or "item" of a tree
"member" of an object
"name/value" or "key/value" pair alias "member"
"element" of an array

but could not see an agreement here.

I agree to Glyn calling the term "union" poor (s. below).

Differentiation from JSON Pointer (JSONPath draft charter)

I anticipate being asked "Why is JSON Pointer not sufficient?" Indeed its abstract says:

JSON Pointer defines a string syntax for identifying a specific value within a JavaScript Object Notation (JSON) document.

... which sounds awfully similar. If we could include a sentence about that, or a link to an answer, that might be helpful. (Murray S. Kucherawy)

No - it's not similar in concept, they're separate things. If you really wanted to mention JSON Pointer, you could say something like "Note that while JSON Pointer (RFC xxxx) is already standardised, it is designed to provide a reference to a single, specific part of a JSON document, whereas JSONPath provides the ability to query a document and potentially return multiple values." (Mark Nottingham)

The short answer is that JSON pointer is good if you already know the structure of the JSON data item you want to point into, and you want to point to exactly one position in there. If you need to do something that is closer to a “search” (which might also result in multiple positions), JSONPath gives you more rope. (Carsten Bormann)

References to XPath

I wonder if the analogies between XPath and JSONPath are going to be helpful, or whether they're actually dangerous by implying equivalences between constructs that are in fact somewhat different? (Michael Kay)

I tend to agree. Although JSONPath was inspired by XPath, I wouldn't want to confuse the JSONPath spec by going into detailed comparisons at the risk of contradicting the normative text. (Glyn Normington)

Someone on StackOverflow today asked a question about JSONPath; they called it (and tagged it) XPath, we really don't want that kind of confusion.

In addition, the reference to the XPath specification in 6.2 is out of date, and the comparison with XPath in Table 2 is very approximate and the terminology inaccurate: for example there is a mention of "node sets", which exist in XPath 1.0 but not in XPath 2.0, yet the citation is to XPath 2.0. For someone who knows the semantics of XPath the comparison raises all sorts of questions about sorting of results into document order, elimination of duplicates etc, which are complications this spec can well do without. (Though some answers are needed, for example if ..store..price matches the same price in more than one way, do you get more than one result? And if not, what does "the same price" actually mean?) (Michael Kay)

It seemed to be important in 2007, while argumenting to have something like XPath for JSON. If nowadays the terminology used has changed significantly with XPath 2.0 and 3.0, we better leave that comparison table 2 out. I am quite passionless here.

Array Slice Operator

Thanks! The ABNF for an array slice in that reference
integer = [%x2D] (%x30 / (%x31-39 *%x30-39))

array-slice = [ integer ] ws %x3A ws [ integer ]
                    [ ws %x3A ws [ integer ] ]
                             ; start:end or start:end:step
is consistent with JMESPath, Python, and my understanding of ECMASCRIPT 4. (Daniel P)

Did anyone else have an opinion on the behaviour of slices such as [::0]? The current draft allows this and says it returns an empty array, but there is good reason to say it should error so that the slice operation is then consistent with Python slicing. See below for more context. (Glyn Normington)

It's good having read this thread and thus understand the current draft much better. I like the decision to be consistent with Python and also getting an empty selection set with step=0.

FYI: there is a recent proposal for adding slice notation syntax to JavaScript, currently at stage 1 of the TC39 process.

https://github.com/tc39/proposal-slice-notation

Interestingly it won't have a step argument ...

https://github.com/tc39/proposal-slice-notation#why-doesnt-this-include-a-step-argument-like-python-does

... because of syntax collision with the new this-binding syntax proposal ::

https://github.com/tc39/proposal-bind-operator

However, we should not let us influence by this.

Unions

I don't think any implementation would remove duplicates from a path such as "$.store.book". I believe this is only somewhat controversial in the context of unions [,]. The name "union" suggests that distinct values be returned, compare with SQL unions. But Stefan Goessner's implementation doesn't do that, it concatenates all results that meet each criteria. There are a few JSONPath implementations that produce real unions with no duplicates instead of concatenated results, but I don't think that's the consensus. (Daniel P)

I think the term “union” is poor. If we think of it as concatenation of results, then the result is as expected. (Glyn Normington)

I agree with that comment, but it's partly because I'm used to SQL UNION, which is different. I prefer the JMESPath term for an analogous construct, MultiSelect List, https://jmespath.org/specification.html#multiselect-list. (Daniel P)

Introducing the union operator [,] simply was meant an analogon to XPath's operator '|'. I cannot tell, if it was a simple combination of node sets in Xpath 1.0 or a true union without duplicates. I obviously was not aware of that subtle (essential ?) union characteristic.

So I fully agree to Glyn Normington's '... the term “union” is poor' statement. Are there some better alternative terms, perhaps 'multi-index operator', 'index list', 'subscript list', etc.?

Duplicates and Ordering

It was my impression that we were talking about duplicated nodes not duplicated values:

Given th array [10,20,30]

$..[0,1,0]

Would yield only two results [10, 20]

(Not that I'm advocating for removing duplicates, personally I think we shouldn't) (Marko Mikulicic)

You’re framing this as “removing duplicates”. Another view is that [10, 20, 10] would be “adding duplicates” (copies of the same node). Related are ordering issues:

$..[1,0] ➔ [20, 10] Or [10, 20]

I would expect the spec will leaves implementations some leeway here, but that should be based on an examination of existing implementations. (Carsten Bormann)

The mental model that leads to omitting duplicate nodes in the output is "selection": if you take an input array and select nodes with index 0,1 or 0, you get only 2 results (since selecting an index twice has no effect).

OTOH, if you opt for a "collect" model, whenever you encounter a node that matches that query you add it to the result stream, thus the same nodes can be present multiple times in the result.

I have a slight preference for the "collect" model, because the general case in jsonpath is to collect things that appear at various points in the json tree. For example:

{"a": {"b": 1, "c": 2}, "d": 3}, $.a.b yields [1] and not {"a":{"b":1}}

(i.e. jsonpath is not a filter and view operation but a pick and gather operation) (Marko Mikulicic)

In implementations that support paths (the majority don't), the query function takes a parameter that indicates values or paths. In both cases the query returns a JSON array of JSON values, in the latter case, a JSON array of normalized paths. (Daniel P)

I must confess to never having thought about duplicates, let alone wanting to eliminate them. So I do like Marko's comparison of 'selection-model' vs. 'collection-model' a lot. I would opt for the latter. In this sense the result of a 'JSONPath query expression' should be termed a 'collection'.

Regarding ordering I see something like a 'natural ordering', according to which

$..[0,1] ➔ [10, 20] $..[1,0] ➔ [20, 10]

would result with the example above.

I do understand the use cases for reordering, duplicates removal, filtering, etc.. But this can always be seen as a postprocessing step on the resulting collection by handing it over to accompanying tools (think of pipe operator).

Of course this cannot work on the result collection of values alone (s. duplicate nodes vs. duplicate values above), it rather requires a collection of (normalized) pathes. In this sense, I like this view:

In my opinion the right balance between powerfulness and enabling simple implementations has been so far one of the key factors that made JSONPath popular over other alternatives, even if it lacks support for aggregation functions. (Davide Bettio)

Filter Expressions

Related to that, it would be helpful to determine if JSONPath filters apply to both JSON objects and arrays, or only to JSON arrays. (Daniel P)

I would support restricting filters to arrays, if others agree. (Glyn Normington)

I tend to let implementations and their "normative force of the factual" decide here or in doubt agree to Glyn's restriction to arrays.

I am very unhappy with confusing $..book[(@.length-1)], where '@' addresses the array itself and implies that array has a length property. In filter expression examples '@' more consistently addresses the current array element.

The invocation of 'the underlying scripting engine' wasn't meant a serious normative aspect, but rather a quick and dirty solution for JavaScript and PHP implementations at that time.

Corner Case

Consider this perfectly legal JSON object

{ "ab": 0, "'a.b": 1, "a-b": 2, "a": { "b": 3 } }

So $.ab is 0, $.a.b is 3, $['a.b'] is 1, $['a-b'] is 2. You'd like to say $.a-b but lots of libraries will refuse it because "a-b" is not a legal JavaScript "name" construct, that's why you have to say $['a-b'].

But suppose your library would accept $.a-b. Then $.a-b and $['a-b'] would be synonyms, but $.a.b and $['a.b'] wouldn't. (Tim Bray)

Hmm ... this seems to be a hint to better exclude '-' from dot-child-selector syntax. I think I have read more discussion about that, currently don't know where.

Respect Implementations

As I mentioned in the session, I think there's a non-trivial amount of risk here that some implementations won't be willing or able to move away from their current behaviours, even if interoperability would improve if they did so. However, there are ways to mitigate that (e.g., a separate 'rfcxxxx compliant' mode). Even so, it will be important to get good participation from as many current implementers as possible. (Mark Nottingham)

The WG will develop a standards-track JSONPath specification that is technically sound and complete, based on the common semantics and other aspects of existing implementations. Where there are differences, the working group will analyze those differences and make choices that rough consensus considers technically best, with an aim toward minimizing disruption among the different JSONPath implementations. (Barry Leiba)

I'm OK with this, but for context: I've been a pretty intense JSONPath user in recent years, and AFAIK the spec, and the implementations, are mostly OK, so the choice between "make JSONPath good" and "don't invalidate implementations" is unlikely to come up. If it did, my predisposition would be to err on the side of not breaking implementations, but I don't think that's inconsistent with Barry's text. (Tim Bray)

+1 to all.

Error Handling

My mental model at the moment is that a JSONPath expression can be valid or erroneous; application of a valid expression yields a result (which may be empty), but does not raise errors. That may not be the right model for all applications. (Carsten Bormann)

The general approach that I've seen several times (including my Elixir implementation) is that an error is raised when there is a syntax error, therefore an invalid expression (e.g. $.foo[[5]) raises an error. Conversely a valid expression applied to a bogus input never raises an error (e.g. $.foo.bar on "test" evals as []). (Davide Bettio)

On the whole I think JSONPath is designed to be "forgiving", i.e. such things aren't errors, e.g. I think I read in the spec that filtering a non-array isn't an error, it's some kind of no-op. That approach isn't always best for everyone, but it's important to be consistent. (Michael Kay)

I would expect one component of this policy to be:

Whether a JSONPath query is valid or not does not depend on the arguments it is applied to.

I.e., you can look at the query and find out independently, without knowing any data, whether it is valid or not. (Carsten Bormann)

I like and totally agree with the forgiving mental model, so having only syntax errors, which do not dependent on data.

Thanks

-- sg

ietf-wg-jsonpath / draft-ietf-jsonpath-base