search

astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
340 stars 42 forks source link

Unable to use the Python Astro SDK with Airflow 2.3.0 #351

Closed tatiana closed 2 years ago

tatiana commented 2 years ago

Describe the bug Airflow 2.3.0 uses newer versions of SQLAlchemy (1.4,<1.4.10) than the ones supported by Astro ("SQLAlchemy>=1.3.18,<=1.3.24"): https://github.com/astro-projects/astro/blob/main/pyproject.toml#L23 https://github.com/apache/airflow/blob/main/setup.cfg#L165

Version

To Reproduce

Steps to reproduce the behavior:

  1. Create a virtualenv:

    python3.9 -m venv myenv

  2. Create a requirements.txt with:

    apache-airflow==2.3.0
astro-projects==0.8.2

  3. Install the requirements.txt file

    pip install -r requirements.txt

Expected behavior The user is able to install the newest version of Airflow and of the Astro library. Currently, the user is not able to install the dependencies of this file.

Additional details

$ pip install -r requirements.txt 
Collecting apache-airflow==2.3.0
  Using cached apache_airflow-2.3.0-py3-none-any.whl (5.2 MB)
Collecting astro-projects==0.8.2
  Using cached astro_projects-0.8.2-py3-none-any.whl (54 kB)
Collecting apache-airflow-providers-http
  Using cached apache_airflow_providers_http-2.1.2-py3-none-any.whl (21 kB)
Collecting apache-airflow-providers-imap
  Using cached apache_airflow_providers_imap-2.2.3-py3-none-any.whl (16 kB)
Collecting apache-airflow-providers-sqlite
  Using cached apache_airflow_providers_sqlite-2.1.3-py3-none-any.whl (16 kB)
Collecting importlib-metadata>=1.7
  Using cached importlib_metadata-4.11.3-py3-none-any.whl (18 kB)
Collecting flask-login<0.5,>=0.3
  Using cached Flask_Login-0.4.1-py2.py3-none-any.whl
Collecting python-nvd3>=0.15.0
  Using cached python_nvd3-0.15.0-py3-none-any.whl
Collecting tenacity>=6.2.0
  Using cached tenacity-8.0.1-py3-none-any.whl (24 kB)
Collecting setproctitle>=1.1.8
  Downloading setproctitle-1.2.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (29 kB)
Collecting tabulate>=0.7.5
  Using cached tabulate-0.8.9-py3-none-any.whl (25 kB)
Collecting cryptography>=0.9.3
  Using cached cryptography-37.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Collecting python-daemon>=2.2.4
  Using cached python_daemon-2.3.0-py2.py3-none-any.whl (35 kB)
Collecting apache-airflow-providers-ftp
  Using cached apache_airflow_providers_ftp-2.1.2-py3-none-any.whl (15 kB)
Collecting werkzeug>=1.0.1,~=1.0
  Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting flask-session<=0.4.0,>=0.3.1
  Using cached Flask_Session-0.4.0-py2.py3-none-any.whl (7.5 kB)
Collecting packaging>=14.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pathspec~=0.9.0
  Using cached pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
Collecting flask<2.0,>=1.1.0
  Using cached Flask-1.1.4-py2.py3-none-any.whl (94 kB)
Collecting sqlalchemy<1.4.10,>=1.4
  Using cached SQLAlchemy-1.4.9-cp38-cp38-manylinux2014_x86_64.whl (1.5 MB)
Collecting colorlog<5.0,>=4.0.2
  Using cached colorlog-4.8.0-py2.py3-none-any.whl (10 kB)
Collecting blinker
  Using cached blinker-1.4-py3-none-any.whl
Collecting termcolor>=1.1.0
  Using cached termcolor-1.1.0-py3-none-any.whl
Collecting cattrs!=1.7.*,~=1.1
  Using cached cattrs-1.10.0-py3-none-any.whl (29 kB)
Collecting marshmallow-oneofschema>=2.0.1
  Using cached marshmallow_oneofschema-3.0.1-py2.py3-none-any.whl (5.8 kB)
Collecting itsdangerous<2.0,>=1.1.0
  Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting markupsafe<2.1.0,>=1.1.1
  Using cached MarkupSafe-2.0.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
Collecting croniter>=0.3.17
  Using cached croniter-1.3.4-py2.py3-none-any.whl (17 kB)
Collecting python-slugify>=5.0
  Using cached python_slugify-6.1.2-py2.py3-none-any.whl (9.4 kB)
Collecting flask-appbuilder==3.4.5
  Using cached Flask_AppBuilder-3.4.5-py3-none-any.whl (1.9 MB)
Collecting deprecated>=1.2.13
  Using cached Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
Collecting graphviz>=0.12
  Using cached graphviz-0.20-py3-none-any.whl (46 kB)
Collecting pygments>=2.0.1
  Using cached Pygments-2.12.0-py3-none-any.whl (1.1 MB)
Collecting dill>=0.2.2
  Using cached dill-0.3.4-py2.py3-none-any.whl (86 kB)
Collecting flask-caching<2.0.0,>=1.5.0
  Using cached Flask_Caching-1.10.1-py3-none-any.whl (34 kB)
Collecting pluggy>=1.0
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting importlib-resources>=5.2
  Downloading importlib_resources-5.7.1-py3-none-any.whl (28 kB)
Collecting psutil>=4.2.0
  Using cached psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
Collecting unicodecsv>=0.14.1
  Using cached unicodecsv-0.14.1-py3-none-any.whl
Collecting typing-extensions>=3.7.4
  Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting attrs<21.0,>=20.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting httpx
  Using cached httpx-0.22.0-py3-none-any.whl (84 kB)
Collecting pendulum>=2.0
  Using cached pendulum-2.1.2-cp38-cp38-manylinux1_x86_64.whl (155 kB)
Collecting cron-descriptor>=1.2.24
  Using cached cron_descriptor-1.2.24-py3-none-any.whl
Collecting alembic<2.0,>=1.5.1
  Using cached alembic-1.7.7-py3-none-any.whl (210 kB)
Collecting gunicorn>=20.1.0
  Using cached gunicorn-20.1.0-py3-none-any.whl (79 kB)
Collecting jsonschema<5.0,>=3.2.0
  Using cached jsonschema-4.4.0-py3-none-any.whl (72 kB)
Collecting markdown>=3.0
  Using cached Markdown-3.3.6-py3-none-any.whl (97 kB)
Collecting python-dateutil>=2.3
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting rich>=12.3.0
  Using cached rich-12.3.0-py3-none-any.whl (232 kB)
Collecting sqlalchemy-jsonfield>=1.0
  Using cached SQLAlchemy_JSONField-1.0.0-py3-none-any.whl (10 kB)
Collecting lockfile>=0.12.2
  Using cached lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting argcomplete>=1.10
  Using cached argcomplete-2.0.0-py2.py3-none-any.whl (37 kB)
Collecting connexion[flask,swagger-ui]>=2.10.0
  Using cached connexion-2.13.1-py2.py3-none-any.whl (95 kB)
Collecting jinja2<3.1,>=2.10.1
  Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting flask-wtf<0.15,>=0.14.3
  Using cached Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
Collecting lazy-object-proxy
  Using cached lazy_object_proxy-1.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60 kB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apache-airflow to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements_upgrade.txt (line 1) and -r requirements_upgrade.txt (line 2) because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow 2.3.0 depends on sqlalchemy<1.4.10 and >=1.4
    astro-projects 0.8.2 depends on SQLAlchemy<=1.3.24 and >=1.3.18
pgzmnk commented 2 years ago

Dito. From a docker-compose install:

The conflict is caused by:
    The user requested SQLAlchemy
    apache-airflow 2.3.0 depends on sqlalchemy<1.4.10 and >=1.4
    astro-projects 0.8.3 depends on SQLAlchemy<=1.3.24 and >=1.3.18
kaxil commented 2 years ago

@pgzmnk Hey Plinio, can you help test 0.8.4b1, please? pip install astro-sdk-python==0.8.4b1 - This adds support for Airflow 2.3.0 and is compatible with 2.2.x as well.