search

Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[BUG] Transitive dependency on `pydantic < 2.0.0` due to `snowflake==0.5.1` #95

Closed maxim-mityutko closed 5 months ago

maxim-mityutko commented 6 months ago

Describe the bug snowflake==0.5.1 has a transitive dependency on pydantic < 2.0.0. The snowflake dependency is always required if plugins functionality is enabled. Hence Brickflow usage is limited to the cases when no extra dependencies are required or if said dependencies does not require pydantic >= 2.0.0

To Reproduce View snowflake package dependencies.

Expected behavior Compatibility with pydantic >= 2.0.0

Screenshots

snowflake==0.5.1
├── snowflake-core [required: ==0.5.1, installed: 0.5.1]
│   ├── atpublic [required: >=4, installed: 4.0]
│   ├── pydantic [required: >=1.10.7,<2.0.0, installed: 1.10.13]
│   │   └── typing-extensions [required: >=4.2.0, installed: 4.9.0]

Cloud Information

Desktop (please complete the following information):

asingamaneni commented 6 months ago

can I close this, as we will be working on this in #93 ?

maxim-mityutko commented 6 months ago

I think so, it is mentioned in the other issue anyways. So the paper trail is there.

maxim-mityutko commented 5 months ago

This can be resolved by:

  1. Bump snowflake dependency version to 0.6.0 in project dependencies.
  2. Bump the Airflow version to at least 2.7.3 in project dependencies.
  3. Update versions in: https://github.com/Nike-Inc/brickflow/blob/6ba515b9eb291195089fd54ea701cfcb0cd97838/brickflow/engine/task.py#L823-L825
asingamaneni commented 5 months ago

@maxim-mityutko thanks for the solution, can you please raise the PR for the same and also update the pyproject.toml file!