custom vocabulary: annotation on validation failure

[karenetheridge]

As a new custom $vocabulary extension, introduce a keyword that is purely annotative in the sense that it does not influence the outcome of validation, but is a sort of anti-annotation whose value is included in the error output when the instance does not validate at that position in the schema. Its inclusion in the error output acts as a sort of hint providing more insight into the nature of the validation failure (i.e. acting like "description" would in annotations for a valid instance).

ref: https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.9.1

Where and how this information should be included in the result is TBD. Should it be its own "error" node in the list of errors as in https://json-schema.org/draft/2019-09/json-schema-core.html#rfc.section.10.4.2? In that case, the evaluation of the keyword would have to either: be able to know whether validation at this schema location has already failed, or provide its annotation ("anti-annotation"?) in all cases and then the overall evaluator (collecting results at each location) would have to know to discard this data if validation succeeded.

[karenetheridge]

This was also alluded to here -- https://github.com/json-schema-org/json-schema-spec/issues/643#issuecomment-441238264 -- although it wasn't recognized there that there is a paradox between producing annotations and having a validation failure.

[karenetheridge]

We could achieve the desired results by always returning annotations, even on validation failure, and then the calling application can piece together the data as required.

for example, the data 5 and schema:

{
  description: "more information about what this value is for",
  multipleOf: 2
}

..would generate this, in "basic" format, if annotations and errors were collected:

{
  valid: false,
  errors: [
    {
      instanceLocation: "",
      keywordLocation: "/multipleOf",
      error: "not a multiple of 5"
    }
  ],
  annotations: [
    {
      instanceLocation: "",
      keywordLocation: "/description",
      annotation: "more information about what this value is for"
    }
  ]
}

..which the application can combine by grouping together all the errors and annotations for each instanceLocation, and that lets the user see the error message together with the description.

e.g. the application might transform to this:

• validation information for location "" (data: 5):
  • via "/multipleOf": "not a multiple of 5"
  • via "/description": "more information about what this value is for"
• validation information for location ...
  • ...

However! this violates the spec (https://json-schema.org/draft/2019-09/json-schema-core.html#rfc.section.7.7.1.2):

Schema objects that produce a false assertion result MUST NOT produce any annotation results, whether from their own keywords or from keywords in subschemas.

[karenetheridge]

Instead of annotating the validation result directly, a keyword could be defined as annotating the errors collected at that subschema itself:

As a keyword to be evaluated after all error-producing keywords have been processed, add an annotation field to the errors produced there. This is an extra property that would be returned in the "basic", "detailed" and "verbose" output formats. A separate annotation is not produced, because the schema failed at that location so no annotations can be returned.

Therefore this vocabulary doesn't just extend the meta-schema itself, but also extends the schema used to validate output.