test_reference_in_meta fails with jsonschema > 4.15.0

[viraptor]

When trying the tests on v1.8.0 with python 3.10 on darwin-intel, I get the following failure:

======================================================================
ERROR: test_reference_in_meta (tests.test_jsonmerge.TestGetSchema)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jm/jsonmerge-1.8.0/.eggs/jsonschema-4.16.0-py3.10.egg/jsonschema/validators.py", line 898, in resolve_from_url
    document = self.store[url]
  File "/tmp/jm/jsonmerge-1.8.0/.eggs/jsonschema-4.16.0-py3.10.egg/jsonschema/_utils.py", line 28, in __getitem__
    return self.store[self.normalize(uri)]
KeyError: 'schema_2.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/jm/jsonmerge-1.8.0/.eggs/jsonschema-4.16.0-py3.10.egg/jsonschema/validators.py", line 901, in resolve_from_url
    document = self.resolve_remote(url)
  File "/tmp/jm/jsonmerge-1.8.0/.eggs/jsonschema-4.16.0-py3.10.egg/jsonschema/validators.py", line 1007, in resolve_remote
    with urlopen(uri) as url:
  File "/usr/lib64/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.10/urllib/request.py", line 503, in open
    req = Request(fullurl, data)
  File "/usr/lib64/python3.10/urllib/request.py", line 322, in __init__
    self.full_url = url
  File "/usr/lib64/python3.10/urllib/request.py", line 348, in full_url
    self._parse()
  File "/usr/lib64/python3.10/urllib/request.py", line 377, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'schema_2.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/jm/jsonmerge-1.8.0/tests/test_jsonmerge.py", line 1996, in test_reference_in_meta
    mschema = merger.get_schema(merge_options={
  File "/tmp/jm/jsonmerge-1.8.0/jsonmerge/__init__.py", line 364, in get_schema
    return walk.descend(schema).val
  File "/tmp/jm/jsonmerge-1.8.0/jsonmerge/__init__.py", line 86, in descend
    rv = self.work(strategy, schema, *args, **opts)
  File "/tmp/jm/jsonmerge-1.8.0/jsonmerge/__init__.py", line 213, in work
    rv = strategy.get_schema(self, schema, **kwargs)
  File "/tmp/jm/jsonmerge-1.8.0/jsonmerge/strategies.py", line 123, in get_schema
    item = dict(walk.resolve_subschema_option_refs(metadataSchema))
  File "/tmp/jm/jsonmerge-1.8.0/jsonmerge/__init__.py", line 153, in resolve_subschema_option_refs
    subschema = w._resolve_refs(JSONValue(subschema), resolve_base=True).val
  File "/tmp/jm/jsonmerge-1.8.0/jsonmerge/__init__.py", line 169, in _resolve_refs
    with self.resolver.resolving(ref) as resolved:
  File "/usr/lib64/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/tmp/jm/jsonmerge-1.8.0/.eggs/jsonschema-4.16.0-py3.10.egg/jsonschema/validators.py", line 841, in resolving
    url, resolved = self.resolve(ref)
  File "/tmp/jm/jsonmerge-1.8.0/.eggs/jsonschema-4.16.0-py3.10.egg/jsonschema/validators.py", line 887, in resolve
    return url, self._remote_cache(url)
  File "/tmp/jm/jsonmerge-1.8.0/.eggs/jsonschema-4.16.0-py3.10.egg/jsonschema/validators.py", line 903, in resolve_from_url
    raise exceptions.RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: unknown url type: 'schema_2.json'
----------------------------------------------------------------------

[avian2]

Hi. This might be an incompatibility with some recent jsonschema. What is the jsonschema version you are using?

[viraptor]

jsonschema-4.16.0 - latest version.

Tested previous versions and 4.9.0 is the first one that fails the tests (up till latest). Versions < 4.9.0 pass everything.

[viraptor]

If it helps, after bisecting, this is the commit in jsonschema which breaks the tests: https://github.com/python-jsonschema/jsonschema/commit/c9a3667507be5ba75fbf57589cbf094e881625af

[avian2]

I've looked a bit into the failure of the test_reference_in_meta test.

The resolver's base_uri isn't getting set, because in the test we have id and $ref on the base level and in that case id gets ignored.

        meta_schema = {
            'id': 'http://example.com/schema_1.json',
            '$ref': 'schema_2.json#/definitions/meta'
        }

This is as per id_of_ignore_ref() used in the old Draft4Validator, which is still the default in jsonmerge. This behavior was introduced in https://github.com/python-jsonschema/jsonschema/commit/bf92aed44eb317188fba68e558d0429c68259dee and first released in jsonschema 4.15.0.

Since $ref is a relative URI and there is no base_uri, the resolve fails. The test works if $ref is a full URI.

There are some other test failures related to reference resolution that appear and disappear with other jsonschema 4.x releases, which might be the reason why your bisect arrived at a different commit.

It's been a while since I looked at this code and I can't remember what use case this test was trying to test in the first place. At the moment it seems to me that it shouldn't have worked at all. I also see there's already a comment in the resolve_subschema_option_refs method that this test hits that the code is kind of dodgy.

I need to think some more about this.

[avian2]

This issue should be fixed now. Can you check if it works for you?