search

dbt-labs / hologram

A library for automatically generating Draft 7 JSON Schemas from Python dataclasses
MIT License
9 stars 13 forks source link

Core pins dataclasses to outdated version #39

Closed jmcarp closed 3 years ago

jmcarp commented 3 years ago

Describe the bug

https://github.com/fishtown-analytics/dbt/blob/dev/margaret-mead/core/setup.py#L72 pins dataclasses to ==0.6, which was released in 2018. This can cause version conflicts with other packages.

Steps To Reproduce

Install dbt with any package that expects a more recent version of dataclasses.

Expected behavior

No version conflict.

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

Which database are you using dbt with?

jtcohen6 commented 3 years ago

Hey @jmcarp, thanks for the issue.

We lock down dataclasses==0.6.0 in dbt-core because it's also pinned in dbt's of our critical dependencies, hologram. We made that change in October 2019 to avoid unpleasant errors when users had installed newer, incompatible versions of low-lying dependencies like jsonschema (https://github.com/fishtown-analytics/dbt/issues/1817, https://github.com/fishtown-analytics/hologram/pull/24).

As I understand it, dataclasses is an interesting case because it's just a backport of py37-style dataclass functionality for python 3.6. While we want to maintain dbt's py36 compatibility for the foreseeable future, we're not doing a lot of net-new development in this area. It looks like the changes between 0.6 and 0.8 are pretty minor, so I'm not opposed to relaxing the requirement, but we'd need to make the change in hologram as well.

You mention that this is causing version conflicts with other packages that you have installed. Would you be able to share some of them? Are they things that community members are frequently installing alongside dbt?

jmcarp commented 3 years ago

In my environment, it looks like the conflict comes from rich: https://github.com/willmcgugan/rich/blob/3473658d13a4e88e1e58a7be116ae6975ca13cf6/pyproject.toml#L29. I can ask them to relax the requirement too/instead. But in general, I think it's preferable to use flexible versioning requirements where possible. This is kind of a rough onboarding experience:

ERROR: hologram 0.0.10 has requirement dataclasses==0.6; python_version < "3.7", but you'll have dataclasses 0.8 which is incompatible.
ERROR: hologram 0.0.10 has requirement jsonschema<3.2,>=3.0, but you'll have jsonschema 3.2.0 which is incompatible.
ERROR: dbt-core 0.18.1 has requirement dataclasses==0.6; python_version < "3.7", but you'll have dataclasses 0.8 which is incompatible.
ERROR: dbt-core 0.18.1 has requirement idna<2.10, but you'll have idna 2.10 which is incompatible.
ERROR: dbt-core 0.18.1 has requirement requests<2.24.0,>=2.18.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: snowflake-connector-python 2.2.10 has requirement boto3<1.15,>=1.4.4, but you'll have boto3 1.16.59 which is incompatible.
ERROR: snowflake-connector-python 2.2.10 has requirement cryptography<3.0.0,>=2.5.0, but you'll have cryptography 3.0 which is incompatible.
ERROR: snowflake-connector-python 2.2.10 has requirement idna<2.10, but you'll have idna 2.10 which is incompatible.
ERROR: snowflake-connector-python 2.2.10 has requirement requests<2.24.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: dbt-snowflake 0.18.1 has requirement cryptography<3,>=2, but you'll have cryptography 3.0 which is incompatible.
ERROR: dbt-redshift 0.18.1 has requirement boto3<1.12,>=1.4.4, but you'll have boto3 1.16.59 which is incompatible.
ERROR: dbt-redshift 0.18.1 has requirement botocore<1.15,>=1.5.0, but you'll have botocore 1.19.59 which is incompatible.
ERROR: dbt-bigquery 0.18.1 has requirement google-api-core<1.17.0,>=1.16.0, but you'll have google-api-core 1.22.1 which is incompatible.
ERROR: dbt-bigquery 0.18.1 has requirement google-cloud-bigquery<1.26.0,>=1.25.0, but you'll have google-cloud-bigquery 1.26.1 which is incompatible.
ERROR: dbt-bigquery 0.18.1 has requirement google-cloud-core<1.4,>=1.3.0, but you'll have google-cloud-core 1.4.1 which is incompatible.
ERROR: dbt-bigquery 0.18.1 has requirement googleapis-common-protos<1.7.0,>=1.6.0, but you'll have googleapis-common-protos 1.52.0 which is incompatible.
ERROR: dbt-bigquery 0.18.1 has requirement protobuf<3.12,>=3.6.0, but you'll have protobuf 3.13.0 which is incompatible.
jtcohen6 commented 3 years ago

@jmcarp Heard. I'm going to transfer this issue to the hologram repo, since that's where the actual dependency is coming from, and where the change would need to take place.

jtcohen6 commented 3 years ago

Copying here from https://github.com/fishtown-analytics/dbt/issues/3150:

If you wanted to bump the dataclasses dependency in dbt, you could:

  1. Fork hologram, bump its dataclasses dependency, open a PR against it
  2. Open a PR against fishtown-analytics/dbt that replaces the versioned hologram dependency with a git+ pointer to your fork branch
  3. Assuming all CI checks pass in the dbt PR, we can merge the hologram PR, cut a new version, and update the dbt PR to point to that new version.

By the by, both of the dataclasses dependencies (in hologram and in dbt-core) are written as:

'dataclasses==0.6;python_version<"3.7"'

The dataclass functionality is native to python 3.7 and above; the dataclasses library is just a backport for python 3.6. So users should only be experiencing version conflict if they're using python 3.6. Also, python 3.6 end of life is December 2021 (9 months from now).