search

dlt-hub / verified-sources

Contribute to dlt verified sources 🔥
https://dlthub.com/docs/walkthroughs/add-a-verified-source
Apache License 2.0
50 stars 40 forks source link

SQL database -> databricks init errors on SQLAlchemy==1.4.52 possibly due to library version conflicts #439

Closed adrianbr closed 2 months ago

adrianbr commented 2 months ago

dlt version

0.4.7

Source name

sql_database

Describe the problem

Happened to user and I was able to reproduce it

if I have SQLAlchemy==1.4.52 installed and try to init dlt init sql_database databricksthen i get the error

Looking up the init scripts in https://github.com/dlt-hub/verified-sources.git...
cannot import name 'Row' from 'sqlalchemy' (/Users/adrian/PycharmProjects/dlt-hub-data-warehous/dags/env/lib/python3.9/site-packages/sqlalchemy/__init__.py)
NOTE: Please refer to https://dlthub.com/docs/reference/command-line-interface#dlt-init for further assistance
(env) MacBook-Pro:dags adrian$ pip show dlt
Name: dlt
Version: 0.4.7

If I install SQLAlchemy==2.0.8 and repeat, the init succeeds and I can then install the requirements, which installs 1.4.52 again.

Expected behavior

Steps to reproduce

How you are using the source?

I run this source in production.

Operating system

macOS, Windows

Runtime environment

Local

Python version

3.9

dlt destination

databricks

Additional information

adrianbr commented 2 months ago

more info, seems a library issue source is for sqlalchemy 2.0+ while the databricks destination requires 1.4+ and doesn't work with 2.0+

Slack thread https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1714476331585079

rudolfix commented 2 months ago

@adrianbr our source should work with 1.4.x let me take a look