SQLAlchemy Version Bug

[jonathandurantalkiatry]

First Stop

• [X] I have read and agreed to the ThoughtSpot Dev Tools EULA.
• [X] I understand that if I have a critical issue with ThoughtSpot, I should reach out to my designated Account Executive and Solutions Consultant team instead of cs_tools.
• [X] I searched the cs_tools documentation, with the integrated search.

Platform Configuration

Description

When running cs_tools tools searchable gather, sqlalchemy==1.4.50 is installed which is incompatible with sqlmodel. I think this causes an upgrade to sqlalchemy==2.X.X but this breaks due to deprecated libraries. Specifically, from sqlalchemy.util.compat import string_types is no longer available in sqlalchemy==2.X.X.

Traceback:

16s Run /home/runner/.config/cs_tools/.cs_tools/bin/cs_tools tools searchable gather \ /home/runner/.config/cs_tools/.cs_tools/bin/cs_tools tools searchable gather \ --config default \ --syncer snowflake://$HOME/snowflake-syncer.toml shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib [08:34:16] INFO registering syncer: snowflake syncer.py:49 INFO installing package: wheel register.py:51 [08:34:17] INFO Collecting wheel _updater.py:69 INFO Downloading _updater.py:69 wheel-0.42.0-py3-none-any.whl.metadata (2.2
kB)
INFO Downloading wheel-0.42.0-py3-none-any.whl (65 _updater.py:69 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 65.4/65.4 kB 2.9 MB/s eta 0:00:00
INFO Installing collected packages: wheel _updater.py:69 INFO Successfully installed wheel-0.42.0 _updater.py:69 INFO installing package: pyarrow == 10.0.1 register.py:51 [08:34:18] INFO Collecting pyarrow==10.0.1 _updater.py:69 INFO Downloading _updater.py:69 pyarrow-10.0.1-cp39-cp39-manylinux_2_17_x86_6
4.manylinux2014_x86_64.whl (35.9 MB)
[08:34:19] INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 35.9/35.9 MB 276.8 MB/s eta 0:00:00
INFO Collecting numpy>=1.16.6 (from _updater.py:69 pyarrow==10.0.1)
INFO Downloading _updater.py:69 numpy-1.26.2-cp39-cp39-manylinux_2_17_x86_64.
manylinux2014_x86_64.whl.metadata (61 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 61.2/61.2 kB 224.9 MB/s eta 0:00:00
INFO Downloading _updater.py:69 numpy-1.26.2-cp39-cp39-manylinux_2_17_x86_64.
manylinux2014_x86_64.whl (18.2 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 18.2/18.2 MB 295.6 MB/s eta 0:00:00
INFO Installing collected packages: numpy, pyarrow _updater.py:69 [08:34:23] INFO Successfully installed numpy-1.26.2 _updater.py:69 pyarrow-10.0.1
INFO installing package: cryptography == 40.0.2 register.py:51 [08:34:24] INFO Collecting cryptography==40.0.2 _updater.py:69 INFO Downloading _updater.py:69 cryptography-40.0.2-cp36-abi3-manylinux_228
x86_64.whl (3.7 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 3.7/3.7 MB 16.9 MB/s eta 0:00:00
INFO Collecting cffi>=1.12 (from _updater.py:69 cryptography==40.0.2)
INFO Downloading _updater.py:69 cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl.metadata (1.5 kB)
INFO Collecting pycparser (from _updater.py:69 cffi>=1.12->cryptography==40.0.2)
INFO Downloading _updater.py:69 pycparser-2.21-py2.py3-none-any.whl (118 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 118.7/118.7 kB 254.7 MB/s eta 0:00:00
INFO Downloading _updater.py:69 cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl (443 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 443.4/443.4 kB 41.5 MB/s eta 0:00:00
[08:34:25] INFO Installing collected packages: pycparser, _updater.py:69 cffi, cryptography
INFO Successfully installed cffi-1.16.0 _updater.py:69 cryptography-40.0.2 pycparser-2.21
INFO installing package: snowflake-sqlalchemy == register.py:51 1.4.6
[08:34:26] INFO Collecting snowflake-sqlalchemy==1.4.6 _updater.py:69 INFO Downloading _updater.py:69 snowflake_sqlalchemy-1.4.6-py2.py3-none-any.w
hl (32 kB)
INFO Collecting sqlalchemy<2.0.0,>=1.4.0 (from _updater.py:69 snowflake-sqlalchemy==1.4.6)
INFO Downloading _updater.py:69 SQLAlchemy-1.4.50-cp39-cp39-manylinux_2_5_x86
_64.manylinux1_x86_64.manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl.metadata (10.0 kB)
[08:34:27] INFO Collecting snowflake-connector-python<4.0.0 _updater.py:69 (from snowflake-sqlalchemy==1.4.6)
INFO Downloading _updater.py:69 snowflake_connector_python-3.6.0-cp39-cp39-ma
nylinux_2_17_x86_64.manylinux2014_x86_64.whl.
metadata (58 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 58.6/58.6 kB 7.2 MB/s eta 0:00:00
INFO Collecting asn1crypto<2.0.0,>0.24.0 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 asn1crypto-1.5.1-py2.py3-none-any.whl (105
kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 105.0/105.0 kB 26.4 MB/s eta 0:00:00
INFO Requirement already satisfied: _updater.py:69 cffi<2.0.0,>=1.9 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (1.16.0)
INFO Requirement already satisfied: _updater.py:69 cryptography<42.0.0,>=3.1.0 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (40.0.2)
INFO Collecting pyOpenSSL<24.0.0,>=16.2.0 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 pyOpenSSL-23.3.0-py3-none-any.whl.metadata
(12 kB)
INFO Collecting pyjwt<3.0.0 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 PyJWT-2.8.0-py3-none-any.whl.metadata (4.2
kB)
INFO Collecting pytz (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 pytz-2023.3.post1-py2.py3-none-any.whl.metada
ta (22 kB)
INFO Collecting requests<3.0.0 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 requests-2.31.0-py3-none-any.whl.metadata
(4.6 kB)
INFO Requirement already satisfied: packaging in _updater.py:69 /home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (23.2)
INFO Collecting charset-normalizer<4,>=2 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 charsetnormalizer-3.3.2-cp39-cp39-manylinux
2_17_x86_64.manylinux2014_x86_64.whl.metadata
(33 kB)
INFO Requirement already satisfied: idna<4,>=2.5 _updater.py:69 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (3.6)
INFO Requirement already satisfied: _updater.py:69 certifi>=2017.4.17 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (2023.11.17)
INFO Requirement already satisfied: _updater.py:69 typing-extensions<5,>=4.3 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6) (4.9.0)
INFO Collecting filelock<4,>=3.5 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 filelock-3.13.1-py3-none-any.whl.metadata
(2.8 kB)
INFO Collecting sortedcontainers>=2.4.0 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 sortedcontainers-2.4.0-py2.py3-none-any.whl
(29 kB)
INFO Collecting platformdirs<4.0.0,>=2.6.0 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 platformdirs-3.11.0-py3-none-any.whl.metadata
(11 kB)
INFO Collecting tomlkit (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 tomlkit-0.12.3-py3-none-any.whl.metadata (2.7
kB)
INFO Collecting urllib3<2.0.0,>=1.21.1 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 urllib3-1.26.18-py2.py3-none-any.whl.metadata
(48 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 48.9/48.9 kB 226.4 MB/s eta 0:00:00
INFO Requirement already satisfied: _updater.py:69 greenlet!=0.4.17 in
/home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
sqlalchemy<2.0.0,>=1.4.0->snowflake-sqlalchem
y==1.4.6) (3.0.3)
INFO Requirement already satisfied: pycparser in _updater.py:69 /home/runner/.config/cs_tools/.cs_tools/lib/p
ython3.9/site-packages (from
cffi<2.0.0,>=1.9->snowflake-connector-python<
4.0.0->snowflake-sqlalchemy==1.4.6) (2.21)
[08:34:28] INFO Collecting cryptography<42.0.0,>=3.1.0 (from _updater.py:69 snowflake-connector-python<4.0.0->snowflake-s
qlalchemy==1.4.6)
INFO Downloading _updater.py:69 cryptography-41.0.7-cp37-abi3-manylinux_228
x86_64.whl.metadata (5.2 kB)
INFO Downloading _updater.py:69 snowflake_connector_python-3.6.0-cp39-cp39-ma
nylinux_2_17_x86_64.manylinux2014_x86_64.whl
(2.5 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 2.5/2.5 MB 131.3 MB/s eta 0:00:00
INFO Downloading _updater.py:69 SQLAlchemy-1.4.50-cp39-cp39-manylinux_2_5_x86
_64.manylinux1_x86_64.manylinux_2_17_x86_64.m
anylinux2014_x86_64.whl (1.6 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 1.6/1.6 MB 280.8 MB/s eta 0:00:00
INFO Downloading _updater.py:69 charsetnormalizer-3.3.2-cp39-cp39-manylinux
2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 142.3/142.3 kB 288.0 MB/s eta 0:00:00
INFO Downloading filelock-3.13.1-py3-none-any.whl _updater.py:69 (11 kB)
INFO Downloading _updater.py:69 platformdirs-3.11.0-py3-none-any.whl (17 kB)
INFO Downloading PyJWT-2.8.0-py3-none-any.whl (22 _updater.py:69 kB)
INFO Downloading pyOpenSSL-23.3.0-py3-none-any.whl _updater.py:69 (58 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 58.8/58.8 kB 240.8 MB/s eta 0:00:00
INFO Downloading _updater.py:69 cryptography-41.0.7-cp37-abi3-manylinux_228
x86_64.whl (4.4 MB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 4.4/4.4 MB 301.8 MB/s eta 0:00:00
INFO Downloading requests-2.31.0-py3-none-any.whl _updater.py:69 (62 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 62.6/62.6 kB 232.1 MB/s eta 0:00:00
INFO Downloading _updater.py:69 urllib3-1.26.18-py2.py3-none-any.whl (143 kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 143.8/143.8 kB 273.7 MB/s eta 0:00:00
INFO Downloading _updater.py:69 pytz-2023.3.post1-py2.py3-none-any.whl (502
kB)
INFO ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ _updater.py:69 502.5/502.5 kB 305.8 MB/s eta 0:00:00
INFO Downloading tomlkit-0.12.3-py3-none-any.whl _updater.py:69 (37 kB)
INFO Installing collected packages: _updater.py:69 sortedcontainers, pytz, asn1crypto, urllib3,
tomlkit, sqlalchemy, pyjwt, platformdirs,
filelock, charset-normalizer, requests,
cryptography, pyOpenSSL,
snowflake-connector-python,
snowflake-sqlalchemy
[08:34:29] INFO Attempting uninstall: sqlalchemy _updater.py:69 INFO Found existing installation: SQLAlchemy _updater.py:69 2.0.24
INFO Uninstalling SQLAlchemy-2.0.24: _updater.py:69 INFO Successfully uninstalled SQLAlchemy-2.0.24 _updater.py:69 [08:34:30] INFO Attempting uninstall: cryptography _updater.py:69 INFO Found existing installation: cryptography _updater.py:69 40.0.2
INFO Uninstalling cryptography-40.0.2: _updater.py:69 INFO Successfully uninstalled cryptography-40.0.2 _updater.py:69 ERROR pip's dependency resolver does not currently _updater.py:69 take into account all the packages that are
installed. This behaviour is the source of
the following dependency conflicts.
INFO sqlmodel 0.0.14 requires _updater.py:69 SQLAlchemy<2.1.0,>=2.0.0, but you have
sqlalchemy 1.4.50 which is incompatible.
INFO Successfully installed asn1crypto-1.5.1 _updater.py:69 charset-normalizer-3.3.2 cryptography-41.0.7
filelock-3.13.1 platformdirs-3.11.0
pyOpenSSL-23.3.0 pyjwt-2.8.0
pytz-2023.3.post1 requests-2.31.0
snowflake-connector-python-3.6.0
snowflake-sqlalchemy-1.4.6
sortedcontainers-2.4.0 sqlalchemy-1.4.50
tomlkit-0.12.3 urllib3-1.26.18
╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/cs_tools │ │ /cli/main.py:144 in run │ │ │ │ 141 │ app.add_typer(_log.app) │ │ 142 │ │ │ 143 │ try: │ │ ❱ 144 │ │ return_code = app(standalone_mode=False) │ │ 145 │ │ │ 146 │ except (click.Abort, typer.Abort): │ │ 147 │ │ return_code = 0 │ │ │ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/typer/ma │ │ in.py:328 in call │ │ │ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/typer/ma │ │ in.py:311 in call │ │ │ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/click/co │ │ re.py:1157 in call │ │ │ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/typer/co │ │ re.py:778 in main │ │ │ │ ... 9 frames hidden ... │ │ in exec_module:850 │ │ in _call_with_frames_removed:228 │ │ │ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/cs_tools │ │ /sync/snowflake/syncer.py:9 in │ │ │ │ 6 import enum │ │ 7 import uuid │ │ 8 │ │ ❱ 9 from snowflake.sqlalchemy import URL │ │ 10 from pydantic import root_validator, Field, BaseModel │ │ 11 import pyarrow.parquet as pq │ │ 12 import sqlalchemy as sa │ │ │ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/snowflak │ │ e/sqlalchemy/init.py:30 in │ │ │ │ 27 │ VARCHAR, │ │ 28 ) │ │ 29 │ │ ❱ 30 from . import base, snowdialect │ │ 31 from .custom_commands import ( │ │ 32 │ AWSBucket, │ │ 33 │ AzureContainer, │ │ │ │ /home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/snowflak │ │ e/sqlalchemy/base.py:13 in │ │ │ │ 10 from sqlalchemy.schema import Sequence, Table │ │ 11 from sqlalchemy.sql import compiler, expression │ │ 12 from sqlalchemy.sql.elements import quoted_name │ │ ❱ 13 from sqlalchemy.util.compat import string_types │ │ 14 │ │ 15 from .custom_commands import AWSBucket, AzureContainer, ExternalStage │ │ 16 from .util import _set_connection_interpolate_empty_sequences │ ╰──────────────────────────────────────────────────────────────────────────────╯ ImportError: cannot import name 'string_types' from 'sqlalchemy.util.compat'
(/home/runner/.config/cs_tools/.cs_tools/lib/python3.9/site-packages/sqlalchemy/ util/compat.py)

[boonhapus]

Thanks @jonathandurantalkiatry ! I see this actually as a snowflake-sqlalchemy issue. The Snowflake team has some work to do to achieve true sqlalchemy 2.0 compatibility, but I've worked around it in the 1.5.0 branch .

This branch is still in a very healthy beta phase at the moment, but should be available before the end of the month.

[boonhapus]

@jonathandurantalkiatry I think I've fixed this as part of v1.4.15 in case this is blocking you.