Minimalist vocabulary file for expressing keyword dependencies.

handrews commented 4 years ago

NOTE: It is 2020. The world has gone to shit, and I'm barely functioning as a human being. But we have a draft to get out and a vague deadline in the form of OAS 3.1. So I'm trying to get this done. HOWEVER I do not have the capacity to debate things in detail. The only things I'm looking for here are:

Will it work?
Will it make our lives difficult in a significant way in the future?

If you think you have a substantially better idea, please file it as its own issue and if it is better we will do that. Feel free to cut-and-paste this issue and just change whatever you want to change if it's not completely different.

I am specifically not up for a debate on naming. If a group of people want to go off on slack and reach a consensus that includes at minimum @Relequestual and @gregsdennis on a single proposal of alternative names and bring that back, we'll switch to those. Just don't debate it here.

I am also not interested in what else we could do with a vocabulary file. I have a much more complex idea in my head that would even allow a generic implementation to automatically implement certain classes of keyword straight from the file. But now is not the time for that.

And I am very much not at all interested in revisiting the question of per-vocabulary meta-schemas. I understand that there are redundancies. I understand that not everyone "gets" why I set them up that way. But that discussion is out of scope for this issue, and really for this whole draft.

Yes, I'm being unreasonably dictatorial. None of us are getting what we want this year, are we?

This is an alternative to #995 about a plugin architecture appendix. This approach would support the plugin architecture sketched out there but is more precise.

The format (as I have long said would be the case) is not a schema. There are two top-level keywords:

vocabulary: this MUST match the URI used in the meta-schema's $vocabulary keyword
keywords: the object of keywords defined by this vocabulary.

Within the keywords object, for each schema keyword there is an object with the following vocabulary file keywords:

inPlaceApplicator: if true, this keyword is an in-place applicator, and MUST run before any other keyword that depends on annotations collected through adjacent in-place applicators defaults to false
dependsOn: list of annotation names (which are also keyword names) on which this keyword depends; there is no way to distinguish which vocabulary provides the annotation, and as always (yes this is in the spec) combining vocabularies with conflicting semantics produces undefined behavior so just don't defaults to an empty list []
throughInPlaceApplicators: if true, the annotations in dependsOn need to be examined both as adjacent keywords, and as annotation propagating up through adjacent in-place applicators; if false only adjacent keywords noted in dependsOn are to be considered. defaults to false
dependsOnValidity: object of keywords (although really it should only be one but that's should and not SHOULD) that MUST be present and MUST produce the given boolean value assertion result in order for this keyword to be processed

All the usual stuff about $vocabulary key names / vocabulary values being URIs and not URLs etc. etc. is the same as for all of the other stuff like this. Furthermore, implementations are welcome to hardcode behavior for well-known vocabularies (e.g. the ones in our spec documents).

This is intended to manage keyword-level dependencies. I am NOT OPEN to vocabulary-level dependencies. Vocabularies are semantic units of convenience that facilitate a plugin architecture. Vocabularies are "aware" of keywords, keywords are not aware of vocabularies. Because reasons.

Since we're documenting dependencies, I wanted to handle if/then/else, and realized I want annotation dependencies and validation dependencies to be handled separately. There's no concept of validation results interacting in a special way with in-place applicators, they just work at the immediately adjacent level. You could in theory depend on both validity and annotations but I wouldn't.

I am open to ONE naming question and only this one: if folks would rather put a $ in front of all of the vocabulary file keywords I'd be OK with that. But this format is not intended to be extensible so idk it just seemed simpler to not bother.

I'm not 100% sure I got all of the keywords here but you get the idea and it shows the dependency stuff.

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/core",
  "keywords": {
    "$schema": {},
    "$vocabulary": {},
    "$id": {},
    "$anchor": {},
    "$ref": {
      "inPlaceApplicator": true
    },
    "$dynamicAnchor": {},
    "$dynamicRef": {
      "inPlaceApplicator": true
    },
    "$defs": {},
    "$comment": {}
  }
}

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/applicator",
  "keywords": {
    "allOf": {
      "inPlaceApplicator": true
    },
    "anyOf": {
      "inPlaceApplicator": true
    },
    "oneOf": {
      "inPlaceApplicator": true
    },
    "not": {
      "inPlaceApplicator": true
    },
    "if": {
      "inPlaceApplicator": true
    },
    "then": {
      "inPlaceApplicator": true,
      "dependsOnValidity": {"if": true}
    },
    "else": {
      "inPlaceApplicator": true,
      "dependsOnValidity": {"if": false}
    },
    "dependentSchemas": {
      "inPlaceApplicator": true
    }, 
   "prefixItems": {},
    "items": {
      "dependsOn": ["prefixItems"],
      "throughInPlaceApplicators": false
    },
    "contains": {},
    "properties": {},
    "patternProperties": {},
    "additionalProperties": {
      "dependsOn": ["properties", "patternProperties"],
      "throughInPlaceApplicators": false
    },
    "propertyNames": {}
  }
}

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/unevaluated",
  "keywords": {
    "unevaluatedProperties": {
      "dependsOn": ["properties", "patternProperties", "additionalProperties", "unevaluatedProperties"],
      "throughInPlaceApplicators": true
    },
    "unevaluatedItems": {
      "dependsOn": ["prefixItems", "items", "contains", "unevaluatedItems"],
      "throughInPlaceApplicators": true
    }
  }
}

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/validation",
  "keywords": {
    "type": {},
    "enum": {},
    "const": {},
    "minimum": {},
    "maximum": {},
    "exclusiveMinimum": {},
    "exclusiveMaximum": {},
    "multipleOf": {},
    "minLength": {},
    "maxLength": {},
    "pattern": {},
    "minItems": {},
    "maxItems": {},
    "minContains": {
      "dependsOn": ["contains"]
    },
    "maxContains": {
      "dependsOn": ["contains"]
    },
    "required": {},
    "dependentRequired": {},
    "minProperties": {},
    "maxProperties": {}
  }
}

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/meta-data",
  "keywords": {
    "title": {},
    "description": {},
    "readOnly": {},
    "writeOnly": {},
    "deprecated": {},
    "examples": {},
    "default": {}
  }
}

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/format",
  "keywords": {
    "format": {}
  }
}

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/content",
  "keywords": {
    "contentMediaType": {},
    "contentEncoding": {},
    "contentSchema": {}
  }
}

{
  "vocabulary": "https://json-schema.org/2019-09/vocab/hyper-schema",
  "keywords": {
    "base": {},
    "links": {}
  }
}

gregsdennis commented 4 years ago

This looks fine at a glance. Let me look at what I did, I'll post a comparison.

handrews commented 4 years ago

@gregsdennis cool- yeah if you have something that's already functioning comparing would be great.

gregsdennis commented 4 years ago

Okay... just had a look, and it just exposes the keywords that are defined by the specified keyword.

However, I would like to give mention to #911. There was quite a bit of discussion on this topic. I'd have to review that again, but as I recall, we had a pretty good system that people seemed to like. I'd like to see some of that incorporated here (then close that one) if it hasn't been already.

handrews commented 4 years ago

@gregsdennis regarding #911: if you build a consensus on that (that should include all of the core team other than me, and at least some level of OK from OpenAPI- possibly just "we don't care what you do with meta-schemas" which is pretty likely TBH) and can get it done before OAS 3.1, cool. But last I heard that wasn't going in the next draft.

This issue is not meant to be "this is what vocabulary files should be forever and ever" it's "there's stuff that needs to be solved for the OAS 3.1-matching draft in the next couple weeks and this gets it done." So lack of movement on #911 in time, with broad enough consensus, would mean that it's out. It's not a merits competition here, it's a schedule thing and this is the minimal option that mostly leaves the status quo in place.

handrews commented 4 years ago

@gregsdennis if #911 doesn't look likely to go out with OAS 3.1, but you think it (or something more like it) is a pretty good bet for the next draft, I'm happy to put something in this draft about how it has a very tentative minimal proposal, and may change substantially.

gregsdennis commented 4 years ago

I'm happy to put something in this draft about how it has a very tentative minimal proposal, and may change substantially.

I think this is definitely a thing we should do. My main concern is rushing into defining the file format, then trying to work around it in future drafts to maintain compatibility.

handrews commented 4 years ago

@gregsdennis totally reasonable. Yeah, being clear that this is a tentative thing to make sure implementors have guidance on the vocabulary contract, but that it ultimately might be a substantially different format. I do think that this contract in terms of exposing keyword names and marking dependencies and in-place applicators is a solid proposal. More might be added but I don't think we'd drop any of these requirements.

jdesrosiers commented 4 years ago

Listing which keywords the vocabulary defines is nice, but if we can also include a way to declare an identifier for each keyword, I can do a lot more to support custom dialects without having to write code. I'm not sure I have a use for the meta-data stuff, but I'm happy to let it just be documentation if I don't need it.

I second (third?) concerns about timeline. It might be best to wait for the next draft for this one.

handrews commented 4 years ago

@jdesrosiers

I am also not interested in what else we could do with a vocabulary file. I have a much more complex idea in my head that would even allow a generic implementation to automatically implement certain classes of keyword straight from the file. But now is not the time for that.

I am also not interested in what else we could do with a vocabulary file. I have a much more complex idea in my head that would even allow a generic implementation to automatically implement certain classes of keyword straight from the file. But now is not the time for that.

I am also not interested in what else we could do with a vocabulary file. I have a much more complex idea in my head that would even allow a generic implementation to automatically implement certain classes of keyword straight from the file. But now is not the time for that.

I am also not interested in what else we could do with a vocabulary file. I have a much more complex idea in my head that would even allow a generic implementation to automatically implement certain classes of keyword straight from the file. But now is not the time for that.

handrews commented 4 years ago

PLEASE RESPECT THE SCOPE ON THIS ISSUE.

jdesrosiers commented 4 years ago

@handrews I'm sorry. It was such a small thing that I didn't notice I stepped out of scope.

But, seriously, that response is uncalled for. I know you're having a hard time and I'm sorry for my part in raising your stress level. Please, take a break if you need it. We'll be here when you're feeling better.

Relequestual commented 4 years ago

This looks interesting but I feel it's WAY outside of scope for 2020-NN. Let's discuss at a later point.

karenetheridge commented 4 years ago

Listing which keywords the vocabulary defines is nice, but if we can also include a way to declare an identifier for each keyword, I can do a lot more to support custom dialects without having to write code.

I'm wondering how the need to write code would be removed from having this data available (or what additional data you think could be provided for this purpose). Except for the most trivial cases (such as keywords that purely produce annotations, like the meta-data vocabulary), surely some code needs to be written, either to contain logic for what constitutes a valid result, or to indicate how the results of subschemas will be collected and relayed upwards?

I guess we'd need to try this out on a new vocabulary, if someone is inclined to push forward on that front.

This looks interesting but I feel it's WAY outside of scope for 2020-NN.

Agreed!

gregsdennis commented 4 years ago

I guess we'd need to try this out on a new vocabulary, if someone is inclined to push forward on that front.

I have a vocab that I use for my tests: https://github.com/gregsdennis/json-everything/blob/master/JsonSchema.Tests/VocabularyTests.cs. It's pretty basic, defining minDate and maxDate, but I definitely need additional code to support them. Their operation can't just be defined in some file that gets loaded dynamically.

jdesrosiers commented 4 years ago

I'm wondering how the need to write code would be removed from having this data available

It doesn't remove the need to write all code, just the boilerplate of wiring keyword names to keyword implementations. This proposal declares the keyword names, but that's not enough to determine an implementation because keyword names are not unique. For example, exclusiveMinimum in draft-04 has different semantics and a different implementation than exclusiveMinimum in draft-06+.

If all keywords have a unique URI, I can automate wiring keywords using the vocabulary definition file. That would mean I can support relatively simple vocabularies without any code. For example, a vocabulary that renames an existing keyword and/or adds meta-data keywords would require no code. The only thing you would have to add code for is a custom keyword implementation.

Here's an example of how my implementation effectively works. The second block is pure boilerplate that wouldn't be necessary if each keyword had a URI.

// `keywords.*` are keyword implementations
addKeyword("https://json-schema.org/keyword/exclusiveMaximum", keywords.exclusiveMaximum);
addKeyword("https://json-schema.org/keyword/exclusiveMinimum", keywords.exclusiveMinimum);
addKeyword("https://json-schema.org/keyword/maximum", keywords.maximum);
addKeyword("https://json-schema.org/keyword/minimum", keywords.minimum);
...

// This would be unnecessary
addVocabulary("https://json-schema.org/draft/2019-09/vocab/validation", {
  "exclusiveMaximum": "https://json-schema.org/keyword/exclusiveMaximum",
  "exclusiveMinimum": "https://json-schema.org/keyword/exclusiveMinimum",
  "maximum": "https://json-schema.org/keyword/maximum",
  "minimum": "https://json-schema.org/keyword/minimum",
  ...
});

handrews commented 3 years ago

I'm closing this issue- it was an attempt to focus on getting something out for OAS 3.1, and that turned out not to be necessary. It was specifically not meant to be a starting point for a larger discussion, but that crept in anyway.

For any future approach, it should emerge from community experience, not something I made up off the top of my head.

json-schema-org / json-schema-spec

Minimalist vocabulary file for expressing keyword dependencies. #996