Create OAS 3.0 friendly schema versions

[rob-metalinkage]

OAS cannot handle relative references in included schemas, nor the $schema keyword, and will require back-compiling $dynamicRefs and other modern keywords.

MVP is create version without $schema and absolute URIs.

[rob-metalinkage]

blocker for #1

[avillar]

Ideas so far:

• Convert all relative $refs to full URLs.
• Wrap anything at the same level of a $ref into an allOf (because the $ref can override the full contents of the object it's in).

Issues so far:

• [ ] We can't rewrite $refs to third-party resources (e.g., GeoJSON, JSON-FG...). What do we do with them?
• [ ] What happens to $refs to schemas that are inside the bblocks repo, but are not annotated schemas? Should we simply not allow that?
• [ ] Properly rewriting $refs to bblock schemas is not that simple (we need to detect whether the target is an annotated schema or not, so that we know whether the file name has to be renamed as well). Currently grappling with this now.
• [ ] Mixing schema versions is not supported by some (most?) of the validation tools.

Tagging @pzaborowski here as well.

[avillar]

Obviously our lives would be easier if we could simply forbid external $refs unless they're to other bblocks... :upside_down_face:

[rob-metalinkage]

So what seems like an antimatter to copy everything is actually forced by the tools...

I think we should design with ref's and cache and annotate provenance in a way we can retrofit to legacy copies..

[avillar]

I'm adding a new item to "issues so far": mixing schema versions is not supported by some (most?) of the validation tools.

For example, AJV can validate 2020-12, but once it starts resolving $refs, it breaks if it comes across something that's not 2020-12. Which makes sense, because what should it do if it encounters a deprecated constraint, or one whose behaviour is now different? Backward compatibility also seems to be a hot topic in the JSON Schema community.

The JSON-FG schemas we reference from the bblocks, for example, are written in 2019-09. I think we should maybe 1) stick to a specific version for all the bblocks (i.e., force 2020-12), and if we need to reuse a schema, we apply something like alterschema and copy the result to the bblocks; and 2) use a single schema.json/schema.yaml for each bblock that only depends on other bblock schemas and not on external resources. This would solve all our problems, but perhaps at the cost of being too constrictive?

[rob-metalinkage]

yeah - lets work on the basis of design with what is available - but compile and convert local copies as required (mirroring manual processes), but inject metadata about the provenance of each copy. We need to establish what "interoperability means" for systems that use heterogeneous schema versions - but cross-testing test suites between systems might be a way forward.

[avillar]

I've implemented a PoC for (simple) OAS 3.0-compatible schemas:

• All $refs are recursively fetched, assigned a unique identifier and added into $defs. Their value is then replaced with the full URL to the root schema, plus the #/$defs/... path.
• If an object contains a $ref and other properties, the $ref is split and everything gets put into an allOf.
• type arrays are transformed into oneOf.
• type: null is converted into nullable: true.
• The root schema is also added to $defs and a $ref to it is wrapped into a allOf.

Example: https://opengeospatial.github.io/bblocks/annotated-schemas/unstable/sosa/features/observationCollection/schema-oas3.0.json

SwaggerHub seems to be able to parse this, and can even generate a client (albeit not very usable, due to the $def identifier generation strategy) from this:

openapi: 3.0.0
info:
  version: '0.1'
  title: BBlocks Test
  description: ''
paths:
  /:
    get:
      responses:
        '200':
          description: Standard response
          content:
            application/json:
              schema:
                $ref: https://opengeospatial.github.io/bblocks/annotated-schemas/unstable/sosa/features/observationCollection/schema-oas3.0.json

Obviously, this is a barely scratching the surface of the issue, but I think it's a start.

[handrews]

All of the schema transformations sound good!

However, OAS 3.0 does not allow $defs, it only understands the Components Object (e.g. #/components/schemas) in the OpenAPI Object (not the Schema Object). You'd have to make an x-$defs or x-definitions keyword and hope there's a way to get tools to support it. Some might have already defined an extension.

[avillar]

Ah, right! I'll update the property name and cross my fingers. Thanks!

[handrews]

@avillar if $defs is working with the tools your using, then maybe you're fine. I just wanted to point out that it's not part of the OAS 3.0 spec.

$defs (and its predecessor, definitions) are odd keywords in that for some uses tools don't really have to "do" anything with them, so sometimes they work even if they're technically not supported. Other tools only recognize schemas under specific keywords, so they will only work if those tools recognize those keywords.

[rob-metalinkage]

Probably need to wrap this up as complete, but generate a new issue to determine testing and compatibility requirements for OGC APIs.

[avillar]

I saw something weird going on with the oas3.0 versions yesterday, so I may have to double check that this is in fact working (thanks for the reminder!)