The Big Plan for external references, non-`$def` references, 2020-12, and much more...

[ahl]

There are a number of issues in typify and in progenitor that pertain to the handling of references. In particular, we assume that schema references will be to JSON paths of the form #/$defs/<Target> (or #/components/schemas/<Target> in OpenAPI for progenitor). This is often true! But it is by no means correct, and many OpenAPI documents and schemas contain references to other points within the file. In addition, both JSON schema and OpenAPI allow for references that target other files--also not properly handled today.

There are still more reference types! A schema may contain an $id tag that can be referenced elsewhere in the schema. This is particularly challenging because we need to take a pass through all valid schemas to discover the location of lurking reference targets. If that seems complicated, JSON Schema 2019-09 added $recursiveAnchor and $recursiveRef... which was removed and replaced in the next JSON Schema release with $dynamicAnchor and $dynamicRef. Both of these (specifically, and generically respectively) are intended for complex, multi-document, self-referential schemas. Such as those used to define the JSON schema spec itself. These last reference types ONLY seem to be used (or understood!) by the JSON Schema authors themselves, writing the JSON Schema for JSON Schema.

Much of this seems like a descent into easily avoided complexity by a standard whose guiding principles differ from my expectations... but it's what we have so let's figure it out!

In addition to these many kinds of references we don't handle today, I plan on including a couple more design goals into the proposed solution:

Multi-schema draft handling. What flavor of JSON schema does typify handle (in the most generous sense of the term)? Typify uses schemars for the Schema structure and that supports something in the vicinity of Draft 7 and 2019-09 (successive version across a naming convention change). But not really. It's missing a number of keywords from both such as contentEncoding and dependencies from Draft 7 and dependentSchemas and unevaluatedProperties from 2019-09. So, typify doesn't really handle any JSON Schema draft all that precisely or completely. Which is fine in that it works pretty well for many schemas, but it would be nice to be a little fussier when it comes to answering the question of supported schema specs.

Error reporting is lousy. Even when typify tells you what went wrong, it rarely tells you where it went wrong. If we're changing how we interact with files generally, it would be nice to keep track of where we are in files so that we can point users to the places we encounter problematic constructions.

To summarize:

• General references (i.e. not to $defs)
• External references / multi-file "bundles"
• $dynamicRef / $recursiveRef
• Strict spec conformance
• Multiple JSON Schema draft support (e.g. determine processing based on the value of $schema)
• JSON paths from errors

---

Here are some general attributes of how we intend to address this:

• Expect "bundles" of related JSON-formatted files
• Treat references more generically i.e. so that we can handle references to anywhere within the bundle
• Maintain "location" path information as we navigate schemas so that we resolve relative references (i.e. to $ids) and produce better error messages
• Serialize into version-specific structures; process into a more generic internal representation

[jrudolph]

Would be great to support split up schemas (and also directly run against a schema at a URL) because... once you get into the business of resolving non-local schemas, you get into the fun of having to resolve URLs correctly. These can be fully qualified references, absolute paths, and relative paths. And potentially referencing anywhere into those documents using fragments. Especially path resolution is something that several tools get wrong (a relative path URL reference is relative to what?).