json-schema-org / JSON-Schema-Test-Suite

A language agnostic test suite for the JSON Schema specifications
MIT License
625 stars 209 forks source link

added tests for checking the entire resource for anchors instead of just the local subschema #679

Closed gregsdennis closed 1 year ago

gregsdennis commented 1 year ago

See Slack conversation

Have tested against my implementation

karenetheridge commented 1 year ago

I'm not a huge fan of the draft6 remotes being added here, since now I need to modify my test harness to skip those (I only support draft7 and up), but I can't really say "don't do that" -- so I'll go ahead and make the appropriate mods.

..and after doing so, tests pass in my implementation.

gregsdennis commented 1 year ago

Yeah, the draft6 and draft7 ref files are the same, but it didn't feel right referencing one from the other, and I didn't think it was right to put it in the "unversioned" parent folder, either.

Julian commented 1 year ago

These tests actually slightly break the promised API for remotes -- specifically we "support" / tell implementers there are 2 ways they can know what additional schema resources and at what remote URLs might need to be available for running tests:

These tests though do something subtly different -- specifically, they use an $id of e.g. "http://localhost:1234/draft2019-09/detached-ref" but they are not available at remotes/draft2019-09/detached-ref, they're instead available at remotes/draft2019-09/detached-ref*.json*.

You can see the effects of this in the new failures of a bunch of implementations in Bowtie.

It's true that implementations may need/want to support discovering different $ids found internally, but given that's not the purpose of this test, it seems to me we should change the $ids here.

gregsdennis commented 1 year ago

they're instead available at remotes/draft2019-09/detached-ref*.json*

I assume the * is meant to make italics?

I thought about this while testing, and I had looked at a few other remotes that seemed to have this same pattern of .json in the file name but not in the $id. I didn't look at any $refs to those files, though. I'll have a look again and post.

You can see the effects of this in the new failures of a bunch of implementations in Bowtie.

Interestingly, mine is among the errored ones. Not sure what that's about. "Works on my machine."