I've been looking for a C# JSON Schema validator / parser and found this repo that claims "NJsonSchema is a .NET library to read, generate and validate JSON Schema draft v4+ schemas.". So I wanted to test it out.
NJsonSchema.Demo Failures
After initially looking at the repository I found this module directory and decided to test it against the JSON Schema Test Data Suite repository. It also looked like the directory structure of the Tests directory matched the official test suite repository but with 8 year old test files (draft-04).
Running this program I initially received 11 failures, which looks to be the expected result from the Program.cs.
NJsonSchema.Demo Execution Output
_I modified the output to reduce some of the verbosity and make it more readable._
```shell
File: Tests/maxLength.json
Test: two supplementary Unicode code points is long enough
Expected: True
Result: False
File: Tests/multipleOf.json
File: Tests/minLength.json
Test: one supplementary Unicode code point is not long enough
Expected: False
Result: True
File: Tests/maxProperties.json
File: Tests/required.json
File: Tests/enum.json
File: Tests/maxItems.json
File: Tests/pattern.json
File: Tests/minItems.json
File: Tests/properties.json
Test: patternProperty invalidates property
Expected: False
Result: True
File: Tests/ref.json
Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'.
Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'.
Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'.
Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'.
Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'.
Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'.
File: Tests/items.json
File: Tests/additionalItems.json
File: Tests/not.json
File: Tests/additionalProperties.json
File: Tests/maximum.json
File: Tests/uniqueItems.json
File: Tests/minimum.json
File: Tests/definitions.json
File: Tests/oneOf.json
File: Tests/type.json
File: Tests/default.json
File: Tests/minProperties.json
File: Tests/refRemote.json
Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/integer.json' with the full JSON path 'http://localhost:1234/integer.json'.
Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/integer.json' with the full JSON path 'http://localhost:1234/integer.json'.
Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/integer' with the full JSON path 'http://localhost:1234/subSchemas.json#/integer'.
Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/integer' with the full JSON path 'http://localhost:1234/subSchemas.json#/integer'.
Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/refToInteger' with the full JSON path 'http://localhost:1234/subSchemas.json#/refToInteger'.
Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/refToInteger' with the full JSON path 'http://localhost:1234/subSchemas.json#/refToInteger'.
Exception: System.InvalidOperationException => Could not resolve the JSON path 'folderInteger.json' within the file path 'folderInteger.json'.
Exception: System.InvalidOperationException => Could not resolve the JSON path 'folderInteger.json' within the file path 'folderInteger.json'.
File: Tests/anyOf.json
File: Tests/allOf.json
File: Tests/patternProperties.json
Test: an invalid due to the other is invalid
Expected: False
Result: True
File: Tests/dependencies.json
Test: missing dependency
Expected: False
Result: True
Test: missing dependency
Expected: False
Result: True
Test: missing other dependency
Expected: False
Result: True
Test: missing both dependencies
Expected: False
Result: True
Test: wrong type
Expected: False
Result: True
Test: wrong type other
Expected: False
Result: True
Test: wrong type both
Expected: False
Result: True
Passes: 241
Fails: 11
Exceptions: 14
```
Validation Attempts on Schema Versions 4+
I imported the newest test data suite from the JSON Schema Test Data Suite repository and modified NJsonSchema.Demo/Program.cs to accept these files. Upon executing the program I was met with an alarming amount of validation failures when validating against the data suite test files.
Does this repository not support versions later than 4 like they claim or have I done something incorrect?
Program.cs: The following code exert has been modified to remove comments, unnecessary stdout prints, program imports & namespace. It has also been modified to have explicit schema validation result variables.
static void Main(string[] args) {
var files = Directory.GetFiles("schemas");
foreach(var file in files){
ValidateTestFile(file);
}
}
static private void ValidateTestFile(string testFile){
var fileData = JArray.Parse(File.ReadAllText(testFile));
var jobjData = fileData.OfType<JObject>();
var suiteIndex = 0;
foreach (var suite in jobjData){
try{
var schema = JsonSchema.FromJsonAsync(suite["schema"]!.ToString()).GetAwaiter().GetResult();
foreach (var test in suite["tests"]!.OfType<JObject>())
{
var testDescription = test["description"]!.Value<string>();
var expectedResult = test["valid"]!.Value<bool>();
var validationResult = false;
try{
validationResult = schema.Validate(test["data"]!).Count == 0;
var validationSuccess = validationResult == expectedResult;
if(!validationSuccess)
// print to stdout
}catch(Exception e)
// print to stdout
}
suiteIndex++;
}catch(Exception e){
// print to stdout
}
}
}
Newer Draft Test Attempt Execution Output
```
File: draft2020-12/maxLength.json
Suite:
Test: two graphemes is long enough
Validation Failure => Expected (True) != Result (False)
Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'maxLength', line 3, position 18.
File: draft2020-12/multipleOf.json
Suite: by int
Suite: by number
Suite: by small number
Suite: float division = inf
Suite: small multiple of large integer
File: draft2020-12/minLength.json
Suite: minLength validation
Test: one grapheme is not long enough
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'minLength', line 3, position 18.
File: draft2020-12/maxProperties.json
Suite: maxProperties validation
Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'maxProperties', line 3, position 22.
Suite: maxProperties = 0 means the object is empty
Test: one property is invalid
Validation Failure => Expected (False) != Result (True)
File: draft2020-12/required.json
Suite: required validation
Test: ignores arrays
Validation Failure => Expected (True) != Result (False)
Test: ignores strings
Validation Failure => Expected (True) != Result (False)
Test: ignores other non-objects
Validation Failure => Expected (True) != Result (False)
Suite: required default validation
Suite: required with empty array
Suite: required with escaped characters
Suite: required properties whose names are Javascript object property names
Test: ignores arrays
Validation Failure => Expected (True) != Result (False)
Test: ignores other non-objects
Validation Failure => Expected (True) != Result (False)
File: draft2020-12/unevaluatedProperties.json
Suite: unevaluatedProperties true
Suite: unevaluatedProperties schema
Test: with invalid unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties false
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with adjacent properties
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with adjacent patternProperties
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with adjacent additionalProperties
Suite: unevaluatedProperties with nested properties
Test: with additional properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with nested patternProperties
Test: with additional properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with nested additionalProperties
Suite: unevaluatedProperties with nested unevaluatedProperties
Suite: unevaluatedProperties with anyOf
Test: when one matches and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: when two match and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with oneOf
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with not
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with if/then/else
Test: when if is true and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: when if is false and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with if/then/else, then not defined
Test: when if is true and has no unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: when if is true and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: when if is false and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with if/then/else, else not defined
Test: when if is true and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: when if is false and has no unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: when if is false and has unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties with dependentdraft2020-12
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0]', line 10, position 8.
Suite: unevaluatedProperties with $ref
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties before $ref
Test: with unevaluated properties
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Could not resolve the JSON path './baseSchema' because no document path is available.
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0].properties.foo', line 6, position 19.
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[1].properties.foo', line 9, position 19.
Suite: nested unevaluatedProperties, outer false, inner true, properties outside
Suite: nested unevaluatedProperties, outer false, inner true, properties inside
Suite: nested unevaluatedProperties, outer true, inner false, properties outside
Test: with no nested unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: with nested unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: nested unevaluatedProperties, outer true, inner false, properties inside
Test: with nested unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: cousin unevaluatedProperties, true and false, true with properties
Test: with no nested unevaluated properties
Validation Failure => Expected (False) != Result (True)
Test: with nested unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: cousin unevaluatedProperties, true and false, false with properties
Test: with nested unevaluated properties
Validation Failure => Expected (False) != Result (True)
Suite: property is evaluated in an uncle schema to unevaluatedProperties
Test: uncle keyword evaluation is not significant
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0].properties.foo', line 7, position 19.
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0].properties.foo', line 7, position 19.
Suite: unevaluatedProperties + single cyclic ref
Test: Unevaluated on 1st level is invalid
Validation Failure => Expected (False) != Result (True)
Test: Unevaluated on 2nd level is invalid
Validation Failure => Expected (False) != Result (True)
Test: Unevaluated on 3rd level is invalid
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[1].properties.b', line 24, position 17.
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'oneOf[1].properties.a', line 63, position 17.
Suite: non-object instances are valid
Suite: unevaluatedProperties with null valued instance properties
Suite: unevaluatedProperties not affected by propertyNames
Test: string property is invalid
Validation Failure => Expected (False) != Result (True)
Suite: unevaluatedProperties can see annotations from if without then and else
Test: invalid in case if is evaluated
Validation Failure => Expected (False) != Result (True)
File: draft2020-12/infinite-loop-detection.json
Suite: evaluating the same schema location against the same data location twice is not a sign of an infinite loop
File: draft2020-12/enum.json
Suite: simple enum validation
Suite: heterogeneous enum validation
Suite: heterogeneous enum-with-null validation
Suite: enums in properties
Suite: enum with escaped characters
Suite: enum with false does not match 0
Suite: enum with [false] does not match [0]
Suite: enum with true does not match 1
Suite: enum with [true] does not match [1]
Suite: enum with 0 does not match false
Suite: enum with [0] does not match [false]
Test: [0.0] is valid
Validation Failure => Expected (True) != Result (False)
Suite: enum with 1 does not match true
Suite: enum with [1] does not match [true]
Test: [1.0] is valid
Validation Failure => Expected (True) != Result (False)
Suite: nul characters in strings
File: draft2020-12/maxItems.json
Suite: maxItems validation
Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'maxItems', line 3, position 17.
File: draft2020-12/minContains.json
Suite: minContains without contains is ignored
Suite: minContains=1 with contains
Test: empty data
Validation Failure => Expected (False) != Result (True)
Test: no elements match
Validation Failure => Expected (False) != Result (True)
Suite: minContains=2 with contains
Test: empty data
Validation Failure => Expected (False) != Result (True)
Test: all elements match, invalid minContains
Validation Failure => Expected (False) != Result (True)
Test: some elements match, invalid minContains
Validation Failure => Expected (False) != Result (True)
Suite: minContains=2 with contains with a decimal value
Test: one element matches, invalid minContains
Validation Failure => Expected (False) != Result (True)
Suite: maxContains = minContains
Test: empty data
Validation Failure => Expected (False) != Result (True)
Test: all elements match, invalid minContains
Validation Failure => Expected (False) != Result (True)
Test: all elements match, invalid maxContains
Validation Failure => Expected (False) != Result (True)
Suite: maxContains < minContains
Test: empty data
Validation Failure => Expected (False) != Result (True)
Test: invalid minContains
Validation Failure => Expected (False) != Result (True)
Test: invalid maxContains
Validation Failure => Expected (False) != Result (True)
Test: invalid maxContains and minContains
Validation Failure => Expected (False) != Result (True)
Suite: minContains = 0
Suite: minContains = 0 with maxContains
Test: too many
Validation Failure => Expected (False) != Result (True)
File: draft2020-12/if-then-else.json
Suite: ignore if without then or else
Suite: ignore then without if
Suite: ignore else without if
Suite: if and then without else
Test: invalid through then
Validation Failure => Expected (False) != Result (True)
Suite: if and else without then
Test: invalid through else
Validation Failure => Expected (False) != Result (True)
Suite: validate against correct branch, then vs else
Test: invalid through then
Validation Failure => Expected (False) != Result (True)
Test: invalid through else
Validation Failure => Expected (False) != Result (True)
Suite: non-interference across combined draft2020-12
Suite: if with boolean schema true
Test: boolean schema true in if always chooses the then path (invalid)
Validation Failure => Expected (False) != Result (True)
Suite: if with boolean schema false
Test: boolean schema false in if always chooses the else path (invalid)
Validation Failure => Expected (False) != Result (True)
Suite: if appears at the end when serialized (keyword processing sequence)
Test: no redirects to then and fails
Validation Failure => Expected (False) != Result (True)
Test: invalid redirects to else and fails
Validation Failure => Expected (False) != Result (True)
File: draft2020-12/dynamicRef.json
Suite: A $dynamicRef to a $dynamicAnchor in the same schema resource behaves like a normal $ref to an $anchor
Test: An array containing non-strings is invalid
Validation Failure => Expected (False) != Result (True)
Suite: A $dynamicRef to an $anchor in the same schema resource behaves like a normal $ref to an $anchor
Test: An array containing non-strings is invalid
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Could not resolve the JSON path '#items' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'intermediate-scope' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'extended' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'extended' because no document path is available.
Suite: multiple dynamic paths to the $dynamicRef keyword
Test: number list with string values
Validation Failure => Expected (False) != Result (True)
Test: string list with number values
Validation Failure => Expected (False) != Result (True)
Suite: after leaving a dynamic scope, it is not used by a $dynamicRef
Test: string matches /$defs/thingy, but the $dynamicRef does not stop here
Validation Failure => Expected (False) != Result (True)
Test: first_scope is not in dynamic scope for the $dynamicRef
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Could not resolve the JSON path 'tree.json' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'extendible-dynamic-ref.json' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'extendible-dynamic-ref.json' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'extendible-dynamic-ref.json' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'http://localhost:1234/draft2020-12/detached-dynamicref.json#/$defs/foo' with the full JSON path 'http://localhost:1234/draft2020-12/detached-dynamicref.json#/$defs/foo'.
Suite: $dynamicRef points to a boolean schema
Test: follow $dynamicRef to a false schema
Validation Failure => Expected (False) != Result (True)
File: draft2020-12/content.json
Suite: validation of string-encoded content based on media type
Suite: validation of binary string-encoding
Suite: validation of binary-encoded media type documents
Suite: validation of binary-encoded media type documents with schema
File: draft2020-12/pattern.json
Suite: pattern validation
Suite: pattern is not anchored
File: draft2020-12/boolean_schema.json
Schema Parse Exception Thrown => Unexpected character encountered while parsing value: T. Path '', line 0, position 0.
Schema Parse Exception Thrown => Unexpected character encountered while parsing value: F. Path '', line 0, position 0.
File: draft2020-12/minItems.json
Suite: minItems validation
Schema Parse Exception Thrown => Input string '1.0' is not a valid integer. Path 'minItems', line 3, position 17.
File: draft2020-12/properties.json
Suite: object properties validation
Suite: properties, patternProperties, additionalProperties interaction
Test: patternProperty invalidates property
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'properties.foo', line 4, position 15.
Suite: properties with escaped characters
Suite: properties with null valued instance properties
Suite: properties whose names are Javascript object property names
File: draft2020-12/prefixItems.json
Suite: a schema given for prefixItems
Test: wrong types
Validation Failure => Expected (False) != Result (True)
Suite: prefixItems with boolean draft2020-12
Test: array with two items is invalid
Validation Failure => Expected (False) != Result (True)
Suite: additional items are allowed by default
Suite: prefixItems with null instance elements
File: draft2020-12/dependentRequired.json
Suite: single dependency
Test: missing dependency
Validation Failure => Expected (False) != Result (True)
Suite: empty dependents
Suite: multiple dependents required
Test: missing dependency
Validation Failure => Expected (False) != Result (True)
Test: missing other dependency
Validation Failure => Expected (False) != Result (True)
Test: missing both dependencies
Validation Failure => Expected (False) != Result (True)
Suite: dependencies with escaped characters
Test: CRLF missing dependent
Validation Failure => Expected (False) != Result (True)
Test: quoted quotes missing dependent
Validation Failure => Expected (False) != Result (True)
File: draft2020-12/ref.json
Suite: root pointer ref
Suite: relative pointer ref to object
Suite: relative pointer ref to array
Test: mismatch array
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Could not resolve the path '#/$defs/tilde~0field'.
Suite: nested refs
Test: nested ref valid
Validation Exception Thrown => The schema reference path '#/$defs/b' has not been resolved.
Test: nested ref invalid
Validation Exception Thrown => The schema reference path '#/$defs/b' has not been resolved.
Suite: ref applies alongside sibling keywords
Test: ref valid, maxItems invalid
Validation Failure => Expected (False) != Result (True)
Schema Parse Exception Thrown => Could not resolve the JSON path 'https://json-schema.org/draft/2020-12/schema' with the full JSON path 'https://json-schema.org/draft/2020-12/schema'.
Suite: property named $ref that is not a reference
Suite: property named $ref, containing an actual $ref
Schema Parse Exception Thrown => Could not resolve the path '#/$defs/bool'.
Schema Parse Exception Thrown => Could not resolve the path '#/$defs/bool'.
Schema Parse Exception Thrown => Could not resolve the JSON path 'node' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the path '#/$defs/foo%22bar'.
Suite: ref creates new scope when adjacent to keywords
Test: referenced subschema doesn't see annotations from properties
Validation Failure => Expected (False) != Result (True)
Suite: naive replacement of $ref with its destination is not correct
Schema Parse Exception Thrown => Could not resolve the JSON path 'schema-relative-uri-defs2.json' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'schema-refs-absolute-uris-defs2.json' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/b/d.json' with the full JSON path 'http://example.com/b/d.json'.
Schema Parse Exception Thrown => Could not resolve the JSON path 'int.json' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path '#bigint' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-1234-ffff-ffff-4321feebdaed' because no document path is available.
Suite: simple URN base URI with JSON pointer
Suite: URN base URI with NSS
Suite: URN base URI with r-component
Suite: URN base URI with q-component
Schema Parse Exception Thrown => Could not resolve the JSON path 'https://json-schema.org/draft/2020-12/schema' with the full JSON path 'https://json-schema.org/draft/2020-12/schema'.
Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-1234-0000-0000-4321feebdaed#/$defs/bar' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-1234-ff00-00ff-4321feebdaed#something' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-4321-ffff-ffff-1234feebdaed' because no document path is available.
Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/ref/if' with the full JSON path 'http://example.com/ref/if'.
Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/ref/then' with the full JSON path 'http://example.com/ref/then'.
Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/ref/else' with the full JSON path 'http://example.com/ref/else'.
Schema Parse Exception Thrown => Could not resolve the JSON path '/absref/foobar.json' because no document path is available.
Suite: $id with file URI still resolves pointers - *nix
Suite: $id with file URI still resolves pointers - windows
Suite: empty tokens in $ref json-pointer
File: draft2020-12/vocabulary.json
Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'properties.badProperty', line 5, position 24.
Suite: ignore unrecognized optional vocabulary
Done
```
Overview
I've been looking for a C# JSON Schema validator / parser and found this repo that claims "NJsonSchema is a .NET library to read, generate and validate JSON Schema draft v4+ schemas.". So I wanted to test it out.
NJsonSchema.DemoFailuresAfter initially looking at the repository I found this module directory and decided to test it against the JSON Schema Test Data Suite repository. It also looked like the directory structure of the
Testsdirectory matched the official test suite repository but with 8 year old test files (draft-04).Running this program I initially received 11 failures, which looks to be the expected result from the
Program.cs.
_I modified the output to reduce some of the verbosity and make it more readable._ ```shell File: Tests/maxLength.json Test: two supplementary Unicode code points is long enough Expected: True Result: False File: Tests/multipleOf.json File: Tests/minLength.json Test: one supplementary Unicode code point is not long enough Expected: False Result: True File: Tests/maxProperties.json File: Tests/required.json File: Tests/enum.json File: Tests/maxItems.json File: Tests/pattern.json File: Tests/minItems.json File: Tests/properties.json Test: patternProperty invalidates property Expected: False Result: True File: Tests/ref.json Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'. Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'. Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'. Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'. Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'. Exception: System.InvalidOperationException => Could not resolve the path '#/tilda~0field'. File: Tests/items.json File: Tests/additionalItems.json File: Tests/not.json File: Tests/additionalProperties.json File: Tests/maximum.json File: Tests/uniqueItems.json File: Tests/minimum.json File: Tests/definitions.json File: Tests/oneOf.json File: Tests/type.json File: Tests/default.json File: Tests/minProperties.json File: Tests/refRemote.json Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/integer.json' with the full JSON path 'http://localhost:1234/integer.json'. Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/integer.json' with the full JSON path 'http://localhost:1234/integer.json'. Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/integer' with the full JSON path 'http://localhost:1234/subSchemas.json#/integer'. Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/integer' with the full JSON path 'http://localhost:1234/subSchemas.json#/integer'. Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/refToInteger' with the full JSON path 'http://localhost:1234/subSchemas.json#/refToInteger'. Exception: System.InvalidOperationException => Could not resolve the JSON path 'http://localhost:1234/subSchemas.json#/refToInteger' with the full JSON path 'http://localhost:1234/subSchemas.json#/refToInteger'. Exception: System.InvalidOperationException => Could not resolve the JSON path 'folderInteger.json' within the file path 'folderInteger.json'. Exception: System.InvalidOperationException => Could not resolve the JSON path 'folderInteger.json' within the file path 'folderInteger.json'. File: Tests/anyOf.json File: Tests/allOf.json File: Tests/patternProperties.json Test: an invalid due to the other is invalid Expected: False Result: True File: Tests/dependencies.json Test: missing dependency Expected: False Result: True Test: missing dependency Expected: False Result: True Test: missing other dependency Expected: False Result: True Test: missing both dependencies Expected: False Result: True Test: wrong type Expected: False Result: True Test: wrong type other Expected: False Result: True Test: wrong type both Expected: False Result: True Passes: 241 Fails: 11 Exceptions: 14 ```NJsonSchema.DemoExecution OutputValidation Attempts on Schema Versions 4+
I imported the newest test data suite from the JSON Schema Test Data Suite repository and modified
NJsonSchema.Demo/Program.csto accept these files. Upon executing the program I was met with an alarming amount of validation failures when validating against the data suite test files.Does this repository not support versions later than 4 like they claim or have I done something incorrect?
Program Modification Details
Directory Structure
Program.cs: The following code exert has been modified to remove comments, unnecessary stdout prints, program imports & namespace. It has also been modified to have explicit schema validation result variables.
Newer Draft Test Attempt Execution Output
``` File: draft2020-12/maxLength.json Suite: Test: two graphemes is long enough Validation Failure => Expected (True) != Result (False) Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'maxLength', line 3, position 18. File: draft2020-12/multipleOf.json Suite: by int Suite: by number Suite: by small number Suite: float division = inf Suite: small multiple of large integer File: draft2020-12/minLength.json Suite: minLength validation Test: one grapheme is not long enough Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'minLength', line 3, position 18. File: draft2020-12/maxProperties.json Suite: maxProperties validation Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'maxProperties', line 3, position 22. Suite: maxProperties = 0 means the object is empty Test: one property is invalid Validation Failure => Expected (False) != Result (True) File: draft2020-12/required.json Suite: required validation Test: ignores arrays Validation Failure => Expected (True) != Result (False) Test: ignores strings Validation Failure => Expected (True) != Result (False) Test: ignores other non-objects Validation Failure => Expected (True) != Result (False) Suite: required default validation Suite: required with empty array Suite: required with escaped characters Suite: required properties whose names are Javascript object property names Test: ignores arrays Validation Failure => Expected (True) != Result (False) Test: ignores other non-objects Validation Failure => Expected (True) != Result (False) File: draft2020-12/unevaluatedProperties.json Suite: unevaluatedProperties true Suite: unevaluatedProperties schema Test: with invalid unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties false Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with adjacent properties Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with adjacent patternProperties Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with adjacent additionalProperties Suite: unevaluatedProperties with nested properties Test: with additional properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with nested patternProperties Test: with additional properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with nested additionalProperties Suite: unevaluatedProperties with nested unevaluatedProperties Suite: unevaluatedProperties with anyOf Test: when one matches and has unevaluated properties Validation Failure => Expected (False) != Result (True) Test: when two match and has unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with oneOf Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with not Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with if/then/else Test: when if is true and has unevaluated properties Validation Failure => Expected (False) != Result (True) Test: when if is false and has unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with if/then/else, then not defined Test: when if is true and has no unevaluated properties Validation Failure => Expected (False) != Result (True) Test: when if is true and has unevaluated properties Validation Failure => Expected (False) != Result (True) Test: when if is false and has unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with if/then/else, else not defined Test: when if is true and has unevaluated properties Validation Failure => Expected (False) != Result (True) Test: when if is false and has no unevaluated properties Validation Failure => Expected (False) != Result (True) Test: when if is false and has unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties with dependentdraft2020-12 Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0]', line 10, position 8. Suite: unevaluatedProperties with $ref Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties before $ref Test: with unevaluated properties Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Could not resolve the JSON path './baseSchema' because no document path is available. Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0].properties.foo', line 6, position 19. Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[1].properties.foo', line 9, position 19. Suite: nested unevaluatedProperties, outer false, inner true, properties outside Suite: nested unevaluatedProperties, outer false, inner true, properties inside Suite: nested unevaluatedProperties, outer true, inner false, properties outside Test: with no nested unevaluated properties Validation Failure => Expected (False) != Result (True) Test: with nested unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: nested unevaluatedProperties, outer true, inner false, properties inside Test: with nested unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: cousin unevaluatedProperties, true and false, true with properties Test: with no nested unevaluated properties Validation Failure => Expected (False) != Result (True) Test: with nested unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: cousin unevaluatedProperties, true and false, false with properties Test: with nested unevaluated properties Validation Failure => Expected (False) != Result (True) Suite: property is evaluated in an uncle schema to unevaluatedProperties Test: uncle keyword evaluation is not significant Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0].properties.foo', line 7, position 19. Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[0].properties.foo', line 7, position 19. Suite: unevaluatedProperties + single cyclic ref Test: Unevaluated on 1st level is invalid Validation Failure => Expected (False) != Result (True) Test: Unevaluated on 2nd level is invalid Validation Failure => Expected (False) != Result (True) Test: Unevaluated on 3rd level is invalid Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'allOf[1].properties.b', line 24, position 17. Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'oneOf[1].properties.a', line 63, position 17. Suite: non-object instances are valid Suite: unevaluatedProperties with null valued instance properties Suite: unevaluatedProperties not affected by propertyNames Test: string property is invalid Validation Failure => Expected (False) != Result (True) Suite: unevaluatedProperties can see annotations from if without then and else Test: invalid in case if is evaluated Validation Failure => Expected (False) != Result (True) File: draft2020-12/infinite-loop-detection.json Suite: evaluating the same schema location against the same data location twice is not a sign of an infinite loop File: draft2020-12/enum.json Suite: simple enum validation Suite: heterogeneous enum validation Suite: heterogeneous enum-with-null validation Suite: enums in properties Suite: enum with escaped characters Suite: enum with false does not match 0 Suite: enum with [false] does not match [0] Suite: enum with true does not match 1 Suite: enum with [true] does not match [1] Suite: enum with 0 does not match false Suite: enum with [0] does not match [false] Test: [0.0] is valid Validation Failure => Expected (True) != Result (False) Suite: enum with 1 does not match true Suite: enum with [1] does not match [true] Test: [1.0] is valid Validation Failure => Expected (True) != Result (False) Suite: nul characters in strings File: draft2020-12/maxItems.json Suite: maxItems validation Schema Parse Exception Thrown => Input string '2.0' is not a valid integer. Path 'maxItems', line 3, position 17. File: draft2020-12/minContains.json Suite: minContains without contains is ignored Suite: minContains=1 with contains Test: empty data Validation Failure => Expected (False) != Result (True) Test: no elements match Validation Failure => Expected (False) != Result (True) Suite: minContains=2 with contains Test: empty data Validation Failure => Expected (False) != Result (True) Test: all elements match, invalid minContains Validation Failure => Expected (False) != Result (True) Test: some elements match, invalid minContains Validation Failure => Expected (False) != Result (True) Suite: minContains=2 with contains with a decimal value Test: one element matches, invalid minContains Validation Failure => Expected (False) != Result (True) Suite: maxContains = minContains Test: empty data Validation Failure => Expected (False) != Result (True) Test: all elements match, invalid minContains Validation Failure => Expected (False) != Result (True) Test: all elements match, invalid maxContains Validation Failure => Expected (False) != Result (True) Suite: maxContains < minContains Test: empty data Validation Failure => Expected (False) != Result (True) Test: invalid minContains Validation Failure => Expected (False) != Result (True) Test: invalid maxContains Validation Failure => Expected (False) != Result (True) Test: invalid maxContains and minContains Validation Failure => Expected (False) != Result (True) Suite: minContains = 0 Suite: minContains = 0 with maxContains Test: too many Validation Failure => Expected (False) != Result (True) File: draft2020-12/if-then-else.json Suite: ignore if without then or else Suite: ignore then without if Suite: ignore else without if Suite: if and then without else Test: invalid through then Validation Failure => Expected (False) != Result (True) Suite: if and else without then Test: invalid through else Validation Failure => Expected (False) != Result (True) Suite: validate against correct branch, then vs else Test: invalid through then Validation Failure => Expected (False) != Result (True) Test: invalid through else Validation Failure => Expected (False) != Result (True) Suite: non-interference across combined draft2020-12 Suite: if with boolean schema true Test: boolean schema true in if always chooses the then path (invalid) Validation Failure => Expected (False) != Result (True) Suite: if with boolean schema false Test: boolean schema false in if always chooses the else path (invalid) Validation Failure => Expected (False) != Result (True) Suite: if appears at the end when serialized (keyword processing sequence) Test: no redirects to then and fails Validation Failure => Expected (False) != Result (True) Test: invalid redirects to else and fails Validation Failure => Expected (False) != Result (True) File: draft2020-12/dynamicRef.json Suite: A $dynamicRef to a $dynamicAnchor in the same schema resource behaves like a normal $ref to an $anchor Test: An array containing non-strings is invalid Validation Failure => Expected (False) != Result (True) Suite: A $dynamicRef to an $anchor in the same schema resource behaves like a normal $ref to an $anchor Test: An array containing non-strings is invalid Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Could not resolve the JSON path '#items' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'intermediate-scope' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'list' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'extended' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'extended' because no document path is available. Suite: multiple dynamic paths to the $dynamicRef keyword Test: number list with string values Validation Failure => Expected (False) != Result (True) Test: string list with number values Validation Failure => Expected (False) != Result (True) Suite: after leaving a dynamic scope, it is not used by a $dynamicRef Test: string matches /$defs/thingy, but the $dynamicRef does not stop here Validation Failure => Expected (False) != Result (True) Test: first_scope is not in dynamic scope for the $dynamicRef Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Could not resolve the JSON path 'tree.json' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'extendible-dynamic-ref.json' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'extendible-dynamic-ref.json' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'extendible-dynamic-ref.json' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'http://localhost:1234/draft2020-12/detached-dynamicref.json#/$defs/foo' with the full JSON path 'http://localhost:1234/draft2020-12/detached-dynamicref.json#/$defs/foo'. Suite: $dynamicRef points to a boolean schema Test: follow $dynamicRef to a false schema Validation Failure => Expected (False) != Result (True) File: draft2020-12/content.json Suite: validation of string-encoded content based on media type Suite: validation of binary string-encoding Suite: validation of binary-encoded media type documents Suite: validation of binary-encoded media type documents with schema File: draft2020-12/pattern.json Suite: pattern validation Suite: pattern is not anchored File: draft2020-12/boolean_schema.json Schema Parse Exception Thrown => Unexpected character encountered while parsing value: T. Path '', line 0, position 0. Schema Parse Exception Thrown => Unexpected character encountered while parsing value: F. Path '', line 0, position 0. File: draft2020-12/minItems.json Suite: minItems validation Schema Parse Exception Thrown => Input string '1.0' is not a valid integer. Path 'minItems', line 3, position 17. File: draft2020-12/properties.json Suite: object properties validation Suite: properties, patternProperties, additionalProperties interaction Test: patternProperty invalidates property Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'properties.foo', line 4, position 15. Suite: properties with escaped characters Suite: properties with null valued instance properties Suite: properties whose names are Javascript object property names File: draft2020-12/prefixItems.json Suite: a schema given for prefixItems Test: wrong types Validation Failure => Expected (False) != Result (True) Suite: prefixItems with boolean draft2020-12 Test: array with two items is invalid Validation Failure => Expected (False) != Result (True) Suite: additional items are allowed by default Suite: prefixItems with null instance elements File: draft2020-12/dependentRequired.json Suite: single dependency Test: missing dependency Validation Failure => Expected (False) != Result (True) Suite: empty dependents Suite: multiple dependents required Test: missing dependency Validation Failure => Expected (False) != Result (True) Test: missing other dependency Validation Failure => Expected (False) != Result (True) Test: missing both dependencies Validation Failure => Expected (False) != Result (True) Suite: dependencies with escaped characters Test: CRLF missing dependent Validation Failure => Expected (False) != Result (True) Test: quoted quotes missing dependent Validation Failure => Expected (False) != Result (True) File: draft2020-12/ref.json Suite: root pointer ref Suite: relative pointer ref to object Suite: relative pointer ref to array Test: mismatch array Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Could not resolve the path '#/$defs/tilde~0field'. Suite: nested refs Test: nested ref valid Validation Exception Thrown => The schema reference path '#/$defs/b' has not been resolved. Test: nested ref invalid Validation Exception Thrown => The schema reference path '#/$defs/b' has not been resolved. Suite: ref applies alongside sibling keywords Test: ref valid, maxItems invalid Validation Failure => Expected (False) != Result (True) Schema Parse Exception Thrown => Could not resolve the JSON path 'https://json-schema.org/draft/2020-12/schema' with the full JSON path 'https://json-schema.org/draft/2020-12/schema'. Suite: property named $ref that is not a reference Suite: property named $ref, containing an actual $ref Schema Parse Exception Thrown => Could not resolve the path '#/$defs/bool'. Schema Parse Exception Thrown => Could not resolve the path '#/$defs/bool'. Schema Parse Exception Thrown => Could not resolve the JSON path 'node' because no document path is available. Schema Parse Exception Thrown => Could not resolve the path '#/$defs/foo%22bar'. Suite: ref creates new scope when adjacent to keywords Test: referenced subschema doesn't see annotations from properties Validation Failure => Expected (False) != Result (True) Suite: naive replacement of $ref with its destination is not correct Schema Parse Exception Thrown => Could not resolve the JSON path 'schema-relative-uri-defs2.json' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'schema-refs-absolute-uris-defs2.json' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/b/d.json' with the full JSON path 'http://example.com/b/d.json'. Schema Parse Exception Thrown => Could not resolve the JSON path 'int.json' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path '#bigint' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-1234-ffff-ffff-4321feebdaed' because no document path is available. Suite: simple URN base URI with JSON pointer Suite: URN base URI with NSS Suite: URN base URI with r-component Suite: URN base URI with q-component Schema Parse Exception Thrown => Could not resolve the JSON path 'https://json-schema.org/draft/2020-12/schema' with the full JSON path 'https://json-schema.org/draft/2020-12/schema'. Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-1234-0000-0000-4321feebdaed#/$defs/bar' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-1234-ff00-00ff-4321feebdaed#something' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'urn:uuid:deadbeef-4321-ffff-ffff-1234feebdaed' because no document path is available. Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/ref/if' with the full JSON path 'http://example.com/ref/if'. Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/ref/then' with the full JSON path 'http://example.com/ref/then'. Schema Parse Exception Thrown => Could not resolve the JSON path 'http://example.com/ref/else' with the full JSON path 'http://example.com/ref/else'. Schema Parse Exception Thrown => Could not resolve the JSON path '/absref/foobar.json' because no document path is available. Suite: $id with file URI still resolves pointers - *nix Suite: $id with file URI still resolves pointers - windows Suite: empty tokens in $ref json-pointer File: draft2020-12/vocabulary.json Schema Parse Exception Thrown => Unexpected initial token 'Boolean' when populating object. Expected JSON object or array. Path 'properties.badProperty', line 5, position 24. Suite: ignore unrecognized optional vocabulary Done ```