Consistent JSON schema version for core and extensions

[javagl]

The JSON Schema version of the core specification has been updated from draft-04 to draft/2020-12 (via https://github.com/KhronosGroup/glTF/pull/2031 and https://github.com/KhronosGroup/glTF/pull/2034).

This raises some questions about the consistency of the specification. The core specification now uses draft/2020-12, whereas the extensions usually use draft-04. Without any guidelines or further actions, this could cause considerable fragmentation and inconsistencies.

Will the existing extension specifications be updated to also use draft/2020-12? (Who will do that, and how?)

Is there any guideline for the schema versions that extension specifications should use? (The section at https://github.com/KhronosGroup/glTF/tree/main/extensions#creating-extensions does not say anything).

[lexaknyazev]

Yes, the extension specs need to be upgraded:

• to the new JSON schema version;
• to use BCP 14 normative language;
• to use AsciiDoc tooling (long-term).

[javagl]

I'm trying to get a bit up to date with the latest schema, and I hope that the following is closely enough related to this topic that it is OK to bring that up here:

---

Section https://json-schema.org/understanding-json-schema/structuring.html#id says

However, using a relative reference when setting a base URI can be problematic. For example, we couldn’t use this schema as an anonymous schema because there would be no Retrieval URI and you can’t resolve a relative reference against nothing. For this and other reasons, it’s recommended that you always use an absolute URI when declaring a base URI with $id.

Right now, the $id are relative. But actually inserting the whole https://raw.githubusercontent.com/KhronosGroup/glTF/main/specification/2.0/schema/ prefix there looks strange at the first glance...

---

Similarly, section https://json-schema.org/understanding-json-schema/structuring.html#ref points out

When using $ref in an anonymous schema, relative references may not be resolvable.

This can be a problem for the extensions (even though they are not "anonymous" - they only refer to a different context than the one that is implied by their base URL). Specifically, I wondered whether extensions that currently contain relative references like

"allOf": [ { "$ref": "glTFProperty.schema.json" } ],

should actually contain absolute references. Again, imagining the full path there ...

"allOf": [ { "$ref": "https://raw.githubusercontent.com/KhronosGroup/glTF/main/specification/2.0/schema/glTFProperty.schema.json" } ],

looks a bit strange, but could be justified: If someone placed his own glTFProperty.schema.json into the same directory, then this would be resolved, instead of the right one...

[lexaknyazev]

For now, I'd say that extension schema files are "located" in the same folder as the core schema files for the purposes of reference resolution.

Long-term, we should plan for exposing all of them with absolute URIs on the https://schema.khronos.org/ website.

[emackey]

For what it's worth, VSCode goes the extra mile and links the core schema out to the extension schemas in the appropriate places. For example the core "material" schema is shown here referencing material extensions, which allows VSCode's auto-complete, hover tooltips, and schema validation to find these extended schemas even while validating starting from the root.

These modified schemas are auto-generated by a tool that resolves all the paths while placing the extensions in subfolders. There is a map of which extensions plug in where. I did a quick test to ensure that this tool continues to work with 2020-12, and it does, although I have not yet pushed a new version of its output to GitHub.

[emackey]

Also for what it's worth, I have no plans to convert VSCode to use absolute references back to GitHub. For my users' purposes VSCode needs to be able to function disconnected from the Internet, validating schemas against local copies only.

[javagl]

I also don't like the absolute paths, but when they can just be the schema.khronos.org ones, that could be acceptable. It still raises some questions for offline validation and such. In the context of tools like the VSCode plugin, one could go the extra mile and so something like the import that you linked to, and additionally upgrade all $refs to use the local file system. From a quick (!) glance at the code, there also seems to be a check whether a certain file exists, and some "guess" of how to resolve it, at https://github.com/AnalyticalGraphicsInc/gltf-vscode/blob/master/util/importSchema.js#L158 ... But it's difficult to find a silver bullet here...

[lexaknyazev]

The JSON-Schema $id property is semantically close to XML namespaces and JSON-LD contexts. It does not require any network operations on its own.

[lexaknyazev]

@emackey Direct quotes from the JSON-Schema spec:

The "$id" keyword identifies a schema resource with its canonical URI.

Note that this URI is an identifier and not necessarily a network locator. In the case of a network-addressable URL, a schema need not be downloadable from its canonical URI.

As the values of "$ref" and "$dynamicRef" are URI References, this allows the possibility to externalise or divide a schema across multiple files, and provides the ability to validate recursive structures through self-reference.

The resolved URI produced by these keywords is not necessarily a network locator, only an identifier. A schema need not be downloadable from the address if it is a network-addressable URL, and implementations SHOULD NOT assume they should perform a network operation when they encounter a network-addressable URI.

[emackey]

Sounds good. Do whatever is right for glTF spec in general here, of course. Later I'll check how the VSCode implementation handles this, and if it doesn't work I'll adapt it using the schema importer script that's already adapting the local path references.

[javagl]

As a 'bump' due to several related issues and discussions:

Who will do that, and how? Is there any guideline for the schema versions that extension specifications should use?

And derived from that: Will there be guidelines for how the schema should be structured, in terms of 'best practices'?

Regarding the update of the extensions to 2020-12: I could give this a stab, but would have to read the change log for these versions more carefully. Also, this would probably be a series of PRs, one for each extension, starting with the most simple ones (there may be several extensions where indeed only the version string has to be updated).

[shrinktofit]

For now, I'd say that extension schema files are "located" in the same folder as the core schema files for the purposes of reference resolution.

Long-term, we should plan for exposing all of them with absolute URIs on the https://schema.khronos.org/ website.

2 yeas past. Still expecting this change. I want to generate extensions' typescript declaration files from schemas. Currently the relative $refs make the generator unable to locate the schemas under spec folder.

[javagl]

There are different workarounds for that.

• The overly pragmatic "get-it-done-no-matter-how" one: Open a text editor. Replace all $ref... file.json with $ref ... c:/data/file.json ...
• The "clean" one: Pass the right $refOptions to the json-schema-to-typescript call, so that it can properly resolve the refs. (Discouraging spoiler: I tried this once, but IIRC didn't manage to actually get this working. And even if it worked, there would be caveats - e.g. whether the output directory structure should match the input directory structure - related to the import statements that are supposed to be generated)
• There is a tool called wetzel that is used for generating the Markdown/AsciiDoc documentation from the schema. This tool is carefully tweaked to ~"usually work with" the glTF schema, meaning that it at least can consume the schema and resolve the $refs accordingly. And ... I once plugged in a quick (and pragmatic) TypeScriptGenerator there. This is NOT an official functionality of that tool. It is NOT maintained in any way. It was ONLY intended as a one-shot approach to generate a bunch of TypeScript classes that I needed. But there is a PR at https://github.com/CesiumGS/wetzel/pull/87

But... of course, having the schemas published in https://schema.khronos.org/ would be preferable...

[emackey]

@shrinktofit This isn't official, but check out my own copy of the glTF schemas that I use with VSCode here:

https://github.com/AnalyticalGraphicsInc/gltf-vscode/tree/master/schemas/gltf-2.0

The above schemas are auto-generated from the official ones with some transformations applied by a NodeJS script, causing the extensions to be fully wired into the main schema. The process is described in more detail here:

https://github.com/AnalyticalGraphicsInc/gltf-vscode/tree/master/util

I think you could generate Typescript defs from there, and they would be compatible.

[aaronfranke]

Are there plans to move forward with updating the schemas? There are two PRs waiting: #2379 and #2380.