[Feature] Allow jsonschema 4 as dependency

[der-eismann]

Is this your first time submitting a feature request?

• [X] I have read the expectations for open source contributors
• [X] I have searched the existing issues, and I could not find an existing issue for this feature
• [X] I am requesting a straightforward change to existing dbt-semantic-interfaces functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I wanted to install dbt-core together with Airflow 2.6.3, using their constraints file. However the pinned usage of jsonschema 3 in this project prevented me from this, as the constraints require version 4:

The conflict is caused by:
    apache-airflow[amazon,cncf-kubernetes,statsd] 2.6.3 depends on jsonschema>=4.0.0
    flask-appbuilder 4.3.1 depends on jsonschema<5 and >=3
    apache-airflow 2.6.3 depends on jsonschema>=4.0.0
    connexion[flask] 2.14.2 depends on jsonschema<5 and >=2.5.1
    dbt-semantic-interfaces 0.2.0 depends on jsonschema~=3.0
    The user requested (constraint) jsonschema==4.18.0

jsonschema 4 has been out for ~2 years and this project is quite new, so I don't know why it's relying on old software.

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

yes

Anything else?

No response

[Jstein77]

Hey @der-eismann! Thanks for flagging this! We pinned dbt-semantic-interfaces to jsonschema 3 as a holdover from when this package was part of metricflow, and was imported by Transform's old cloud service. dbt-core depends on hologram which depends on jsonschema>=3.0, which doesn't specify an upper bound so we should be able to upgrade.

I chatted with @QMalcolm about what it would take to upgrade. We're not sure how much of lift this will be, because we don't yet know what will break when we upgrade. We'll do some investigation on our side and get back to you.

[der-eismann]

That sounds great, thank you very much!

[QMalcolm]

So I tried upgrading to jsonschema 4 locally for this project to see what would break, and it was unfortunately quite a few things. More details in the github actions of #136. It looks like we'll have to do more than just allow for jsonschema 4, but we'll have to migrate to jsonschema 4. I.e. jsonschema~=4.0 instead of jsonschema>=3.0,<5.0

[tjni]

What will break in dbt-core >= 1.6.0 if we do not include this dependency? In nixpkgs, we only package the latest version of jsonschema, and, because we do not use a constraint solver, we don't package earlier versions. I'm exploring if updating is possible (perhaps with some minor degradation), or if we should hold off until this is addressed.

[angad]

Ran against this issue where I was not able to run dbt-core >= 1.6.0 with Cloud Composer 2.5.3. I pinned requirements to specific dbt-bigquery 1.5.6 (which needs dbt-core 1.5.6, which does not depend on dbt-semantic-interfaces).

airflow-dbt
dbt-bigquery==1.5.6
agate==1.6.3

[zli06160]

Hello, I also have a similar problem while installing dbt-core under the constraints https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.11.txt.

The conflict is caused by:
    dbt-semantic-interfaces 0.2.0 depends on jsonschema~=3.0
    The user requested (constraint) jsonschema==4.18.0

Impossible to resolve the conflict by changing the versions slightly.

jsonschema==4.*.* is pretty necessary for Apache Airflow users.

[codecae]

This issue seems to revolve around the deprecation of the jsonschema.RefResolver in jsonschema 4.18.

See release notes for 4.18 for warning of this deprecation: https://github.com/python-jsonschema/jsonschema/releases/tag/v4.18.0

If this library is pinned to use >=4.0<4.18 and airflow users tweak their constraints to be permissive of 4.17.3, it should be good to go for airflow>=2.6.3.

[codecae]

@QMalcolm @der-eismann

I believe I may have resolved this compatibility issue in #160 by following the jsonschema migration guide from RefResolver to referencing.Registry

https://python-jsonschema.readthedocs.io/en/latest/referencing/#migrating-from-refresolver

I'll get the rest of the stuff for the PR completed, but I have passing tests locally using jsonschema 4.19.1

[der-eismann]

Awesome, thanks!