JEP-15 Third-Party Functions

[springcomp]

Third-Party Functions

JEP	15
Author	Nolan Wood, Maxime Labelle
Status	draft
Created	28-Feb-2022

Abstract

This JEP proposes a new distinction between reserved function names and _third-party function names. This allows the spec to evolve by introducing new functions without the fear of clashing with existing functions from popular third-party implementations.

Motivation

Virtually all the current language implementations of JMESPath support registering additional "custom" functions that add features to JMESPath.

There is currently no restrictions as to what those function names may be except that is must match the unquoted-string production.

This proposal specifically acknowledges that third-party functions may be used to extend JMESPath and mandates tha all third-party function names should start with a leading underscore character.

Functions whose names starts with a letter – known or future – are effectively reserved exclusively for use by JMESPath.

Specification

This JEP proposes adding the following text to the specification in the Function Expressions section:

Function Expressions

Functions allow users to easily transform and filter data in JMEPath expressions. Built-in functions are defined by JMESPath. However, a compliant implementation MAY support registration of new functions to extend JMESPath.

Functions whose name starts with an alphabetic character (i.e A-Za-z) are reserved for JMESPath. Conversely, third-party function names SHOULD start with a leading underscore character (i.e _ ).

Registration of conflicting function names MAY raise an error. However, if no error is raised, built-in functions MUST take precedence over third-party functions. The resolution of conflicting function names from competing third-party libraries is undefined.

Rationale

This JEP does not introduce any grammar change.

Although there has been discussions about having a different semantic constructs for both reserved functions and third-party functions, it has been considered as an implementation detail. Besides, this would unnecessarily make it more difficult for library authors to maintain two different logics for evaluating functions.

[innovate-invent]

Gitter conversation discussing merit of this JEP and selection of proposed prefix: https://gitter.im/jmespath/chat?at=6219441ad1b64840db3cfb1c

[innovate-invent]

Thanks so much for writing this up. I have some notes about the final wording.

Conversely, third-party function names SHOULD start with a leading underscore character (i.e _ ).

I think SHOULD needs to be changed to MUST for this to be anything more than a recommendation.

Registration of conflicting function names MAY raise an error. However, if no error is raised, built-in functions MUST take precedence over third-party functions.

This is an implementation detail that doesn't really need to be in language spec. It may be desirable for registration of conflicting functions to silently overwrite the conflict.

The resolution of conflicting function names from competing third-party libraries is undefined.

This is also an implementation detail.

[springcomp]

In theory 3rd party names MUST start with underscore. However, I feel this renders all existing implementations non-compliant. That's why I changed it to SHOULD to try and preserve backwards compatibility.

From then on, it followed that there must be some priority for built-in functions. Even for future ones. Another way of saying it is ; even though this change would not render your custom implementation non-compliant, you have no guarantee if ever a built-in function appears.

What do you think?

[springcomp]

The resolution of conflicting function names from competing third-party libraries is undefined.

This is also an implementation detail.

Agreed. But wanted to leave no room for ambiguity. Happy to remove if needed.

[springcomp]

Registration of conflicting function names MAY raise an error. However, if no error is raised, built-in functions MUST take precedence over third-party functions.

This is an implementation detail that doesn't really need to be in language spec. It may be desirable for registration of conflicting functions to silently overwrite the conflict.

I think we are saying the same thing. I'm trying to say that it can be silently ignored. But there should no circumstances for a built-in function to be overwritten by a custom implementation. Or are you specifically saying that the spec should not forbid that explicitly?

[innovate-invent]

In theory 3rd party names MUST start with underscore. However, I feel this renders all existing implementations non-compliant. That's why I changed it to SHOULD to try and preserve backwards compatibility.

My idea was to render all existing implementations non-complaint, with the follow up that most of the functions they currently provide would be added to the spec bringing them back into compliance.

From then on, it followed that there must be some priority for built-in functions. Even for future ones. Another way of saying it is ; even though this change would not render your custom implementation non-compliant, you have no guarantee if ever a built-in function appears.

I think that would be detrimental to backwards compatibility.

[innovate-invent]

I think we are saying the same thing. I'm trying to say that it can be silently ignored. But there should no circumstances for a built-in function to be overwritten by a custom implementation. Or are you specifically saying that the spec should not forbid that explicitly?

I think that the language spec shouldn't speculate on implementation at all, otherwise it could cause implementer infighting on the various ways of interpreting the speculation. It is better to be ambiguous now and clarify later than to say something now and have to change it later.

[springcomp]

My idea was to render all existing implementations non-compliant, with the follow up that most of the functions they currently provide would be added to the spec bringing them back into compliance.

I see.

Maybe we need to coordinate and list which functions we are talking about here. I assume those would be:

• items, from_items (previously known as to_object) and zip functions.
• GorillaStack-style split string manipulation function, but maybe other expected functions as well ?
• Particularly the lower and upper functions from jmespath.py.

Do you see others?

[jordemort]

I'm currently working on an app that is going to embed JMESPath with a bunch of custom functions so this is relevant to my interests. I was wondering how I could avoid my custom functions getting clobbered by a future version of the spec, which is how I ended up here :)

Would you consider x_ or X_ as a prefix instead of just _? IMO, functions named like x_foo, x_bar etc would be more aesthetic and more readable in complex expressions than just _foo and _bar.

I think what would be really ideal is to have some sort of namespace for functions, so I could have something like myapp:foo and myapp:bar but I don't have a strong enough understanding of the grammar to know if something like that would be possible without breaking a bunch of things. Namespaces would make it more obvious that a function is specific to a particular app, though.

[springcomp]

@jordemort I'm secretly a fan of namespaces 😏 but we're not sure it's the best way to achieve this. Besides, we plan to include functions in the specification at a much faster pace than previously.

Please head up to the discussions area to be involved. Also, you might want to read the chat we had on Glitter on 25 February (Cannot seem to be able to link to the actual conversation from my mobile device)

[springcomp]

Closed in favor of corresponding discussion.