Wrong unevaluatedProperties validation errors

[vassilvk]

Consider the following schema which uses unevaluatedProperties in order to extend a closed schema following the official JSON schema documentation:

schema.json

{
  "allOf": [
    {
      "type": "object",
      "properties": {
        "street_address": { "type": "string" },
        "city": { "type": "string" },
        "state": { "type": "string" }
      },
      "required": ["street_address", "city", "state"]
    }
  ],
  "properties": {
    "type": { "enum": ["residential", "business"] }
  },
  "required": ["type"],
  "unevaluatedProperties": false
}

When running jv against the following valid document:

doc.json

{
  "street_address": "123 Willow Road",
  "city": "Middletown",
  "state": "KY",
  "type": "residential"
}

... jv validates the document as expected:

$ jv schema.json doc.json
$

Now, if we make the document invalid by modifying one of the subschema properties - for example, by setting the street_address to a number:

{
  "street_address": 12,
  "city": "Middletown",
  "state": "KY",
  "type": "residential"
}

... jv correctly detects the street_address type mismatch, however it also yields an unevaluatedProperties not allowed error for every property that's declared by the subschema, although these properties are correct according to the schema:

$ jv schema.json doc.json
[I#] [S#] doesn't validate with file:///Users/vassilk/projects/test/json/schema.json#
  [I#] [S#/allOf/0] allOf failed
    [I#/street_address] [S#/allOf/0/properties/street_address/type] expected string, but got number
  [I#/state] [S#/unevaluatedProperties] not allowed
  [I#/street_address] [S#/unevaluatedProperties] not allowed
  [I#/city] [S#/unevaluatedProperties] not allowed

[santhosh-tekuri]

@vassilvk

the second document you have specified is invalid;

here allOf has failed. so any properties evaluated by allOf are still considered as unevaluated; only if the validation succeeds, the properties are considered evaluated;

[edit]

the documentation you mentioned clearly says:

unevaluatedProperties works by collecting any properties that are successfully validated when processing the schemas and using those as the allowed list of properties.

[santhosh-tekuri]

from spec:

https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#name-unevaluatedproperties

The behavior of this keyword depends on the annotation results of adjacent keywords that apply to the instance location being validated.

but annotation results are generated only when an instance is valid against schema. see https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#name-annotations

JSON Schema can annotate an instance with information, whenever the instance validates against the schema object containing the annotation, and all of its parent schema objects.

[santhosh-tekuri]

to better understand consider following example:

consider following files:

schema.json: ( allOf is wrapped in not )

{
  "allOf": [ { "not":
    {
      "type": "object",
      "properties": {
        "street_address": { "type": "string" },
        "city": { "type": "string" },
        "state": { "type": "string" }
      },
      "required": ["street_address", "city", "state"]
    }
  }],
  "properties": {
    "type": { "enum": ["residential", "business"] }
  },
  "required": ["type"],
  "unevaluatedProperties": false
}

{
  "street_address": 123,
  "city": "Middletown",
  "state": "KY",
  "type": "residential"
}

here street_address is number so allOf fails which implies not succeeds. still validator considers that none of the properties are evaluated

see jv output below:

jv schema.json doc.json
[I#] [S#] doesn't validate with file:///Users/santhosh/gh/santhosh-tekuri/jsonschema/cmd/jv/schema.json#
  [I#/street_address] [S#/unevaluatedProperties] not allowed
  [I#/city] [S#/unevaluatedProperties] not allowed
  [I#/state] [S#/unevaluatedProperties] not allowed

if you want you can also check results at https://www.jsonschemavalidator.net

[vassilvk]

@vassilvk the second document you have specified is invalid;

Yes, this was the main point. The second document has one invalid property, and I expect to see a validation error only for that one property. Seeing validation errors for all the other properties (which are in fact valid) makes no sense and is potentially confusing.

From customer's perspective, the experience is the following:

• They enter a number for street address (which they shouldn't, but they do).
• They get an error correctly indicating that address is not supposed to be a number.
• Then they get messages that address, city, and state are not allowed.

This last bit is technically wrong. I understand that your goal is to stick to the specification, and that's great. I am just surprised at how impractical unevaluatedProperties becomes when it comes to subschemas.

[santhosh-tekuri]

in the beginning, this library used to show only single error and stop validation on first error.

later ability to show multiple errors is added.

It is always not trivial to show errors from customer's perspective.

for example when missing a semicolon in a program, compiler shows tons of errors. it is similar situation here.

we can do stop validating keywords that have dependent on other keywords. but it is little complicated to implement and maintain.

[vassilvk]

I understand. Thank you for the quick response and thank you for a great library!