This JEP proposes a new function let() (originally proposed by Michael
Dowling) that allows for evaluating an expression with an explicitly defined
lexical scope. This will require some changes to the lookup semantics in
JMESPath to introduce scoping, but provides useful functionality such as being
able to refer to elements defined outside of the current scope used to evaluate
an expression.
Motivation
As a JMESPath expression is being evaluated, the current element, which can be
explicitly referred to via the @ token, changes as expressions are
evaluated. Given a simple sub expression such as foo.bar, first the
foo expression is evaluted with the starting input JSON document, and the
result of that expression is then used as the current element when the bar
element is evaluted. Conceptually we’re taking some object, and narrowing down
its current element as the expression is evaluated.
Once we’ve drilled down to a specific current element, there is no way, in the
context of the currently evaluated expression, to refer to any elements outside
of that element. One scenario where this is problematic is being able to refer
to a parent element.
Let’s say we wanted to get the list of cities of the state corresponding to our
first_choice key. We’ll make the assumption that the state names are
unique in the states list. This is currently not possible with JMESPath.
In this example we can hard code the state WA:
states[?name==`WA`].cities[]
but it is not possible to base this on a value of first_choice, which
comes from the parent element. This JEP proposes a solution that makes
this possible in JMESPath.
Specification
There are two components to this JEP, a new function, let(), and a change
to the way that identifiers are resolved.
The let() Function
The let() function is heavily inspired from the let function commonly
seen in the Lisp family of languages:
let is a function that takes two arguments. The first argument is a JSON
object. This hash defines the names and their corresponding values that will
be accessible to the expression specified in the second argument. The second
argument is an expression reference that will be evaluated.
Resolving Identifiers
Prior to this JEP, identifiers are resolved by consulting the current context
in which the expression is evaluted. For example, using the same
search function as defined in the JMESPath specification, the
evaluation of:
search(foo, {"foo": "a", "bar": "b"}) -> "a"
will result in the foo identifier being resolved in the context of
the input object {"foo": "a", "bar": "b"}. The context object defines
foo as a, which results in the identifier foo being resolved as
a.
In the case of a sub expression, where the current evaluation context
changes once the left hand side of the sub expression is evaluted:
search(a.b, {"a": {"b": "y"}) -> "y"
The identifier b is resolved with a current context of
{"b": "y"}, which results in a value of y.
This JEP adds an additional step to resolving identifiers. In addition
to the implicit evaluation context that changes based on the result
of continually evaluating expressions, the let() command allows
for additional contexts to be specified, which we refer to by the common
name scope. The steps for resolving an identifier are:
Attempt to lookup the identifier in the current evaluation context.
If this identifier is not resolved, look up the value in the current
scope provided by the user.
If the idenfitier is not resolved and there is a parent scope, attempt
to resolve the identifier in the parent scope. Continue doing this until
there is no parent scope, in which case, if the identifier has not been
resolved, the identifier is resolved as null.
Parent scopes are created by nested let() calls.
Below are a few examples to make this more clear. First, let’s
examine the case where the identifier can be resolved from the
current evaluation context:
search(let({a: `x`}, &b), {"b": "y"}) -> "y"
In this scenario, we are evaluating the expression b, with the
context object of {"b": "y"}. Here b has a value of y,
so the result of this function is y.
Now let’s look at an example where an identifier is resolved from
a scope object provided via let():
search(let({a: `x`}, &a), {"b": "y"}) -> "x"
Here, we’re trying to resolve the a identifier. The current
evaluation context, {"b": "y"}, does not define a. Normally,
this would result in the identifier being resolved as null:
search(a, {"b": "y"}) -> null
However, we now fall back to looking in the provided scope object {"a": "x"}, which was provided as the first argument to let. Note here that
the value of a has a value of "x", so the identifier is resolved as
"x", and the return value of the let() function is "x".
Finally, let’s look at an example of parent scopes. Consider the
following expression:
Here we have nested let calls, and the expression we are trying to
evaluate is the multiselect hash {a: a, b: b, c: c}. The
c identifier comes from the evaluation context {"c": "z"}.
The b identifier comes from the scope object in the second let
call: {b: \y`}. And finally, here’s the lookup process for the a` identifier:
Is a defined in the current evaluation context? No.
Is a defined in the scope provided by the user? No.
Is there a parent scope? Yes
Does the parent scope, {a: \x`}, definea? Yes,ahas the value of"x", soais resolved as the string"x"`.
Current Node Evaluation
While the JMESPath specification defines how the current node is determined,
it is worth explicitly calling out how this works with the let() function
and expression references. Consider the following expression:
a.let({x: `x`}, &b.let({y: `y`}, &c))
Given the input data:
{"a": {"b": {"c": "foo"}}}
When the expression c is evaluated, the current evaluation context is
{"c": "foo"}. This is because this expression isn’t evaluated until
the second let() call evaluates the expression, which does not
occur until the first let() function evaluates the expression.
Motivating Example
With these changes defined, the expression in the “Motivation” section can be
be written as:
Which evalutes to ["Seattle", "Bellevue", "Olympia"].
Rationale
If we just consider the feature of being able to refer to a parent element,
this approach is not the only way to accomplish this. We could also allow
for explicit references using a specific token, say $.
The original example in the “Motivation” section would be:
states[?name==$.first_choice].cities[]
While this could work, this has a number of downsides, the biggest one being
that you’ll need to always keep track of the parent element. You don’t know
ahead of time if you’re going to need the parent element, so you’ll always need
to track this value. It also doesn’t handle nested lexical scopes. What if
you wanted to access a value in the grand parent element? Requiring an
explicit binding approach via let() handles both these cases, and doesn’t
require having to track parent elements. You only need to track additional
scope when let() is used.
To this end, the project authors had to introduce a new abstraction to the AST object that implements function calls.
/// <summary>
/// The <see cref="IScopeParticipant" /> interface lets
/// implementations participate in a stack of contexts
/// to assist evaluating expressions.
///
/// This supports the <see cref="LetFunction" />.
/// </summary>
public interface IScopeParticipant
{
void PushScope(JToken scope);
void PopScope();
}
This abstraction is actually only used for the implementation of the let() function itself.
public class LetFunction : JmesPathFunction
{
public LetFunction(IScopeParticipant scopes)
: base("let", 2, scopes)
{
}
public override void Validate(params JmesPathFunctionArgument[] args)
{
System.Diagnostics.Debug.Assert(base.Scopes != null);
EnsureObject(args[0]);
EnsureExpressionType(args[1]);
}
public override JToken Execute(params JmesPathFunctionArgument[] args)
{
scopes_?.PushScope(args[0].Token);
try
{
var expression = args[1].Expression;
var result = expression.Transform(Context);
return result.AsJToken();
}
finally
{
scopes_?.PopScope();
}
}
}
This allows to register the function evaluation context to the corresponding lexical scope.
Additionally, the following abstraction was introduced:
```c#
public interface IContextEvaluator
{
JToken Evaluate(string identifier);
}
The IContextEvaluator abstraction encapsulates context evaluation logic required to extract the proper value
from the stack of lexical scopes. The implementation follows the specification requirements:
public sealed class LexicalScopes : IScopeParticipant, IContextEvaluator
{
private readonly Stack<JToken> scopes_
= new Stack<JToken>()
;
public JToken Evaluate(string identifier)
{
if (scopes_.Count == 0)
return JTokens.Null;
foreach (var scope in scopes_)
{
if (scope[identifier] != null)
return scope[identifier];
}
return JTokens.Null;
}
public void PushScope(JToken token)
{
scopes_.Push(token);
}
public void PopScope()
{
scopes_.Pop();
}
}
The lexical scope stack contains a series of JSON objects referred to by the JToken type in C#.
When evaluating a JMESPath expression, identifier expressions are evaluated. That’s where scope
evaluation must take place.
public class JmesPathIdentifier : JmesPathExpression
{
private readonly string name_;
internal IContextEvaluator evaluator_;
public JmesPathIdentifier(string name)
{
name_ = name;
}
public string Name => name_;
protected override JmesPathArgument Transform(JToken json)
{
var jsonObject = json as JObject;
return jsonObject?[name_] ?? Evaluate(name_);
}
public override string ToString()
{
return $"JmesPathIdentifier: {name_}";
}
public JToken Evaluate(string identifier)
=> evaluator_?.Evaluate(identifier) ?? JTokens.Null;
}
When evaluating an identifier against the current JSON object, the implementation first uses the current context
which is specified as an argument of the corresponding expression. If the identifier does not refer to an existing
value, the identifier switches to using the IContextEvaluator abstraction referred to above to find the required
value out of the stack of lexical scopes.
No dependency were required to implement this JEP.
Python
A Python implementation also currently exists.
It features the same two main pieces of functionality.
Given that most object-oriented languages support the concept of abstractions via interfaces (or prototypes) and that
an expected implementation would map grammar constructs to some form of AST, it seems reasonable to believe that a
similar implementation as the one shown here for C# could be achieved with the following languages:
C++
Java
JavaScript
TypeScript
Go
Rust
Although I have no experience on other languages, there is no reason to believe it would be any different or even harder
than the simple implementation shown here.
Lexical Scoping
Abstract
This JEP proposes a new function
let()
(originally proposed by Michael Dowling) that allows for evaluating an expression with an explicitly defined lexical scope. This will require some changes to the lookup semantics in JMESPath to introduce scoping, but provides useful functionality such as being able to refer to elements defined outside of the current scope used to evaluate an expression.Motivation
As a JMESPath expression is being evaluated, the current element, which can be explicitly referred to via the
@
token, changes as expressions are evaluated. Given a simple sub expression such asfoo.bar
, first thefoo
expression is evaluted with the starting input JSON document, and the result of that expression is then used as the current element when thebar
element is evaluted. Conceptually we’re taking some object, and narrowing down its current element as the expression is evaluated.Once we’ve drilled down to a specific current element, there is no way, in the context of the currently evaluated expression, to refer to any elements outside of that element. One scenario where this is problematic is being able to refer to a parent element.
For example, suppose we had this data:
Let’s say we wanted to get the list of cities of the state corresponding to our
first_choice
key. We’ll make the assumption that the state names are unique in thestates
list. This is currently not possible with JMESPath. In this example we can hard code the stateWA
:but it is not possible to base this on a value of
first_choice
, which comes from the parent element. This JEP proposes a solution that makes this possible in JMESPath.Specification
There are two components to this JEP, a new function,
let()
, and a change to the way that identifiers are resolved.The let() Function
The
let()
function is heavily inspired from thelet
function commonly seen in the Lisp family of languages:https://clojuredocs.org/clojure.core/let
http://docs.racket-lang.org/guide/let.html
The let function is defined as follows:
let
is a function that takes two arguments. The first argument is a JSON object. This hash defines the names and their corresponding values that will be accessible to the expression specified in the second argument. The second argument is an expression reference that will be evaluated.Resolving Identifiers
Prior to this JEP, identifiers are resolved by consulting the current context in which the expression is evaluted. For example, using the same
search
function as defined in the JMESPath specification, the evaluation of:will result in the
foo
identifier being resolved in the context of the input object{"foo": "a", "bar": "b"}
. The context object definesfoo
asa
, which results in the identifierfoo
being resolved asa
.In the case of a sub expression, where the current evaluation context changes once the left hand side of the sub expression is evaluted:
The identifier
b
is resolved with a current context of{"b": "y"}
, which results in a value ofy
.This JEP adds an additional step to resolving identifiers. In addition to the implicit evaluation context that changes based on the result of continually evaluating expressions, the
let()
command allows for additional contexts to be specified, which we refer to by the common name scope. The steps for resolving an identifier are:Attempt to lookup the identifier in the current evaluation context.
If this identifier is not resolved, look up the value in the current scope provided by the user.
If the idenfitier is not resolved and there is a parent scope, attempt to resolve the identifier in the parent scope. Continue doing this until there is no parent scope, in which case, if the identifier has not been resolved, the identifier is resolved as
null
.Parent scopes are created by nested
let()
calls.Below are a few examples to make this more clear. First, let’s examine the case where the identifier can be resolved from the current evaluation context:
In this scenario, we are evaluating the expression
b
, with the context object of{"b": "y"}
. Hereb
has a value ofy
, so the result of this function isy
.Now let’s look at an example where an identifier is resolved from a scope object provided via
let()
:Here, we’re trying to resolve the
a
identifier. The current evaluation context,{"b": "y"}
, does not definea
. Normally, this would result in the identifier being resolved asnull
:However, we now fall back to looking in the provided scope object
{"a": "x"}
, which was provided as the first argument tolet
. Note here that the value ofa
has a value of"x"
, so the identifier is resolved as"x"
, and the return value of thelet()
function is"x"
.Finally, let’s look at an example of parent scopes. Consider the following expression:
Here we have nested let calls, and the expression we are trying to evaluate is the multiselect hash
{a: a, b: b, c: c}
. Thec
identifier comes from the evaluation context{"c": "z"}
. Theb
identifier comes from the scope object in the secondlet
call:{b: \
y`}. And finally, here’s the lookup process for the
a` identifier:Is
a
defined in the current evaluation context? No.Is
a
defined in the scope provided by the user? No.Is there a parent scope? Yes
Does the parent scope,
{a: \
x`}, define
a? Yes,
ahas the value of
"x", so
ais resolved as the string
"x"`.Current Node Evaluation
While the JMESPath specification defines how the current node is determined, it is worth explicitly calling out how this works with the
let()
function and expression references. Consider the following expression:Given the input data:
When the expression
c
is evaluated, the current evaluation context is{"c": "foo"}
. This is because this expression isn’t evaluated until the secondlet()
call evaluates the expression, which does not occur until the firstlet()
function evaluates the expression.Motivating Example
With these changes defined, the expression in the “Motivation” section can be be written as:
Which evalutes to
["Seattle", "Bellevue", "Olympia"]
.Rationale
If we just consider the feature of being able to refer to a parent element, this approach is not the only way to accomplish this. We could also allow for explicit references using a specific token, say
$
. The original example in the “Motivation” section would be:While this could work, this has a number of downsides, the biggest one being that you’ll need to always keep track of the parent element. You don’t know ahead of time if you’re going to need the parent element, so you’ll always need to track this value. It also doesn’t handle nested lexical scopes. What if you wanted to access a value in the grand parent element? Requiring an explicit binding approach via
let()
handles both these cases, and doesn’t require having to track parent elements. You only need to track additional scope whenlet()
is used.Implementation Survey
C
JMESPath.NET implements this proposal.
To this end, the project authors had to introduce a new abstraction to the AST object that implements function calls.
This abstraction is actually only used for the implementation of the
let()
function itself.The
IContextEvaluator
abstraction encapsulates context evaluation logic required to extract the proper value from the stack of lexical scopes. The implementation follows the specification requirements:The lexical scope stack contains a series of JSON objects referred to by the
JToken
type in C#. When evaluating a JMESPath expression,identifier
expressions are evaluated. That’s where scope evaluation must take place.When evaluating an
identifier
against the current JSON object, the implementation first uses the current context which is specified as an argument of the corresponding expression. If theidentifier
does not refer to an existing value, theidentifier
switches to using theIContextEvaluator
abstraction referred to above to find the required value out of the stack of lexical scopes.No dependency were required to implement this JEP.
Python
A Python implementation also currently exists. It features the same two main pieces of functionality.
When the function
let()
is evaluated, it creates a child scope. https://github.com/brinkmanlab/BioPython-Convert/blob/73473ea6a7cd2ac5006d9fbfb131b5d937bb400c/biopython_convert/JMESPathGen.py#L65-L73Identifier evaluation goes to the scope hierarchy. https://github.com/brinkmanlab/BioPython-Convert/blob/73473ea6a7cd2ac5006d9fbfb131b5d937bb400c/biopython_convert/JMESPathGen.py#L133-L135
Other languages
Given that most object-oriented languages support the concept of abstractions via interfaces (or prototypes) and that an expected implementation would map grammar constructs to some form of AST, it seems reasonable to believe that a similar implementation as the one shown here for C# could be achieved with the following languages:
Although I have no experience on other languages, there is no reason to believe it would be any different or even harder than the simple implementation shown here.