search

json-schema-org / JSON-Schema-Test-Suite

A language agnostic test suite for the JSON Schema specifications
MIT License
603 stars 205 forks source link

Clarify how test runners should handle `format`. #702

Closed big-andy-coates closed 8 months ago

big-andy-coates commented 8 months ago

Unde the How to Implement a Test Runner - Additional Assumptions section in the main README.md, it alludes to special handling required for format tests:

In particular, tests within the optional/format subdirectory may require implementations to change the way they treat the "format"keyword (particularly on older drafts which did not have a notion of vocabularies).

But doesn't give any specifics as to what is actually needed.

@erosb has been kind enough to raise an issue for us to correct our test runner in terms of how format.json vs optional/format/*.json tests are handled, including a link to an associated Slack conversation.

This all seems legit. However, it feels like this information should also be included in this repos docs.

gregsdennis commented 8 months ago

The tests (specifically the 2019-09+ ones) don't specify the format assertion vocab, so implementations are required to configure themselves for format validation on these tests. For the non-optional tests, such configuration should be turned off and format should only generate annotations (if anything).

The configuration rule applies to all drafts, but the later drafts have the capability of using the format assertion vocab.