Dynamic schemas using templated "$ref"

[jerstlouis]

As discussed in https://github.com/json-schema-org/json-schema-vocabularies/issues/35#issuecomment-767624829 , in a context of using JSON Schema in an OpenAPI definition, I suggested the possibility to use templated path parameters in "$ref", to allow validation of later path parameters based on dynamic schemas that could be retrieved by substituting the templates by the selected value for earlier path parameters.

The resolved "$ref" would become regular "$ref", but in OpenAPI definitions one could write e.g.

"$ref" : "./api/collections-styles/{collectionId}"

and before the final validation step, the {collectionId} template would be replaced by a selected value like collection1. (e.g. in this example, this "$ref" is used to define the valid values for {styleId} in a path /collections/{collectionId}/styles/{styleId}).

The {collectionId} would have previously been validated against another potentially dynamic list like "$ref" : "./api/collections".

./api/collections would return something like:

{
   "type" : "string",
   "enum" : [
      "collection1",
      "collection2",
      "collection3"
   ]
}

and ./api/collection-styles/collection1 would return something like

{
   "type" : "string",
   "enum" : [
      "style234",
      "style235",
      "style236"
   ]
}

while ./api/collection-styles/collection2 would return something like

{
   "type" : "string",
   "enum" : [
      "style454",
      "style455",
      "style456"
   ]
}

The context being that available/compatibles styles depend on the selected collection.

One simple way to implement this would be to have a validating function for such schemas which accepts a dictionary of template parameters / values, allowing to resolve all the templated $ref.

[Relequestual]

Venders who want to generate API libraries won't like that much =D

[jerstlouis]

@Relequestual I believe it would technically be possible to generate API libraries for this, treating those templated parameters generitcally, and the library would need to fetch those schemas using references at run-time.

This allows for much more powerful/flexible APIs which work on multiple instances/deployments using the same templated API definition, and thus greatly increases interoperability.

[Relequestual]

Maybe. My response was a little throw-away. This repo is really a holding ground till groups of interested parties (including key vendors) can form a group to create vocabularies with agreed semantic meaning. Hopefully one day you'll be drawn back here with someone wanting to make this happen =]

[jdesrosiers]

@jerstlouis Thanks for writing up your thoughts.

We can't make changes to the $ref keyword with a vocabulary because it's a core keyword. However, if you give it a different name such as templatedRef, it can be defined in a vocabulary and implemented. If the vocabulary becomes popular, we might consider merging the templatedRef functionality into $ref.

Generally, implementing a new keyword should be easy, but this one probably won't be because it violates a core assumption in JSON Schema that there are only two kinds of things: schemas and instances. You have added a new kind of thing: template variables. Rather than just writing a plugin, you'll likely have to fork your favorite implementation and modify it (or write something from scratch).

But, first things first, your proposal needs more detail. Here are some things you will need to think about and address.

1. How does template expansion work? Is there a standard you'd like to use or do you want to define your own process? (hint: choose this one, https://tools.ietf.org/html/rfc6570)
2. Your example only has a $ref at the top level of the schema. Are templatedRefs only allowed at the top level of the schema? Probably not.
3. Are all templatedRefs in a schema resolved against the same template variables?
4. Do templatedRefs in referenced schemas resolve against the template variables as well?
5. What if a templatedRef in a referenced schema has a template name collision with a templatedRef in the original schema? Is there a way to disambiguate?
6. What happens if a templatedRef is encountered and the necessary template variables aren't given? Is it an error? Does it do a best effort and continue?

[jerstlouis]

@jdesrosiers Thank you very much for proposing a way to move this forward.

1. Template expansion should follow RFC 6570, although it could potentially be restricted to a subset of what is allowed by the RFC (e.g. only simple {identifier}).
2. $templatedRef could be allowed at any level, however I think even having it only at the top-level would allow to implement the use cases we are thinking of, which is mainly to specify a unique schema for OpenAPI path parameters, which depends on earlier path parameters. That is, being able to write: "schema" : { "$templatedRef" : "./api/collections-styles/{collectionId}" } instead of "schema" : { "type" : "string" } directly in the properties of a definition for an OpenAPI components/parameters.
3. Yes templatedRefs in a schema would be resolved against the same template variables.
4. Yes templatedRef's in referenced schemas resolve against the same template variables -- all template variables required for a schema being referenced, including any nested use of template variables, would be provided. But for our use cases, just being able to use a $templatedRef to reference one schema which would not need to support $templatedRef would be enough. My initial understanding (which might be wrong, see below) was that JSON Schema needs to support this because the "schema" property itself is defined as being of a JSON Schema type, therefore the "$ref"/$"templatedRef logic needs to be supported by JSON Schema. Although looking at https://swagger.io/specification/#schema-object , the Schema Object is being defined as an "extended subset", and the Reference Object, is something else entirely, so potentially one OpenAPI extension could be the support for $templatedRef?
5. I don't think template name collisions occur if all variables are defined before referencing the top schema, as mentioned in 3 & 4.
6. If a templateRef is encountered and the necessary template variables were not defined, that's an error. It could be handled as gracefully as possible, but it's still an error and should not happen.

I am realizing that really what needs to be extended is probably the OpenAPI Reference Object or the JSON Reference, rather than JSON Schema, from https://swagger.io/specification/ :

The Reference Object is defined by JSON Reference and follows the same structure, behavior and rules.
For this specification, reference resolution is accomplished as defined by the JSON Reference specification
and not by the JSON Schema specification.

Thank you!