search

snowflakedb / snowflake-sqlalchemy

Snowflake SQLAlchemy
https://pypi.python.org/pypi/snowflake-sqlalchemy/
Apache License 2.0
231 stars 152 forks source link

Unpinned snowflake python connector causing broken imports #129

Closed jtalmi closed 1 year ago

jtalmi commented 4 years ago

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using (python --version)?

Snowflake Connector for Python Version: 2.0.3, Python Version: 3.6.8, Platform: Linux-4.15.0-1043-aws-x86_64-with-Ubuntu-18.04-bionic

  1. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')?

Snowflake Connector for Python Version: 2.0.3, Python Version: 3.6.8, Platform: Linux-4.15.0-1043-aws-x86_64-with-Ubuntu-18.04-bionic

  1. What are the component versions in the environment (pip list)? 

    adal                       1.2.1      
aioboto3                   6.4.1      
aiobotocore                0.10.4     
aiocontextvars             0.2.2      
aiohttp                    3.6.0      
alabaster                  0.7.12     
alembic                    0.8.10     
amqp                       2.4.2      
analytics-python           1.2.9      
apache-airflow             1.10.2     
approvaltests              0.1.14     
asn1crypto                 0.24.0     
async-generator            1.10       
async-timeout              3.0.1      
asyncio                    3.4.3      
asyncpg                    0.19.0     
atlasclient                0.1.2      
attrs                      19.3.0     
aws-xray-sdk               0.95       
azure-common               1.1.18     
azure-cosmos               3.0.2      
azure-datalake-store       0.0.19     
azure-mgmt-datalake-nspkg  3.0.1      
azure-mgmt-datalake-store  0.4.0      
azure-mgmt-nspkg           3.0.2      
azure-mgmt-resource        1.2.2      
azure-nspkg                3.0.2      
azure-storage              0.36.0     
azure-storage-blob         2.1.0      
azure-storage-common       2.1.0      
Babel                      2.6.0      
bcrypt                     3.1.6      
billiard                   3.5.0.4    
bleach                     2.1.4      
blessings                  1.7        
blinker                    1.4        
boto                       2.49.0     
boto3                      1.9.181    
botocore                   1.12.253   
cachetools                 3.1.0      
cassandra-driver           3.17.0     
cdecimal                   2.3        
celery                     4.1.1      
certifi                    2019.3.9   
cffi                       1.12.2     
chardet                    3.0.4      
click                      6.7        
cloudant                   0.5.10     
codecov                    2.0.15     
colorama                   0.4.1      
coloredlogs                10.0       
colour-runner              0.1.1      
configparser               3.5.3      
contextlib2                0.6.0.post1
contextvars                2.4        
cookies                    2.2.1      
coverage                   4.5.4      
croniter                   0.3.27     
cryptography               2.6.1      
cucco                      2.2.1      
cx-Oracle                  7.1.2      
Cython                     0.29.6     
datadog                    0.29.3     
decorator                  4.3.2      
deepdiff                   4.0.8      
defusedxml                 0.5.0      
dill                       0.2.9      
docker                     3.7.0      
docker-pycreds             0.4.0      
docopt                     0.6.2      
docutils                   0.14       
ecdsa                      0.13       
elasticsearch              5.0.1      
elasticsearch-dsl          5.4.0      
engarde                    0.4.0      
entrypoints                0.3        
enum34                     1.1.6      
fastavro                   0.9.6      
filelock                   3.0.12     
Flask                      0.12.4     
Flask-Admin                1.5.2      
Flask-AppBuilder           1.12.1     
Flask-Babel                0.12.2     
Flask-Bcrypt               0.7.1      
Flask-Caching              1.3.3      
Flask-Login                0.4.1      
Flask-OpenID               1.2.5      
Flask-SQLAlchemy           2.3.2      
flask-swagger              0.2.13     
Flask-WTF                  0.14.2     
flower                     0.9.2      
freezegun                  0.3.11     
funcsigs                   1.0.0      
future                     0.16.0     
fuzzywuzzy                 0.17.0     
gino                       0.8.3      
gitdb2                     2.0.5      
GitPython                  2.1.11     
google-ads                 3.2.0      
google-api-core            1.8.1      
google-api-python-client   1.7.8      
google-auth                1.6.3      
google-auth-httplib2       0.0.3      
google-auth-oauthlib       0.2.0      
google-cloud-bigquery      1.10.0     
google-cloud-bigtable      0.31.0     
google-cloud-container     0.2.1      
google-cloud-core          0.28.1     
google-cloud-spanner       1.8.0      
google-resumable-media     0.3.2      
googleapis-common-protos   1.6.0      
grpc-google-iam-v1         0.11.4     
grpcio                     1.19.0     
grpcio-gcp                 0.2.2      
gspread                    2.0.0      
gunicorn                   19.9.0     
hdfs                       2.2.2      
hmsclient                  0.1.1      
html5lib                   1.0.1      
httplib2                   0.12.1     
humanfriendly              4.18       
idna                       2.8        
idna-ssl                   1.1.0      
ijson                      2.3        
imagesize                  1.1.0      
immutables                 0.11       
importlib-metadata         0.23       
inflection                 0.3.1      
iso8601                    0.1.12     
isodate                    0.6.0      
itsdangerous               1.1.0      
JayDeBeApi                 1.1.1      
jeepney                    0.4        
Jinja2                     2.8.1      
jira                       2.0.0      
jmespath                   0.9.4      
JPype1                     0.6.3      
json-merge-patch           0.2        
jsondiff                   1.1.1      
jsonpickle                 1.1        
keyring                    18.0.0     
kombu                      4.3.0      
kubernetes                 8.0.1      
ldap3                      2.5.2      
lockfile                   0.12.2     
lxml                       4.3.2      
Mako                       1.0.7      
Markdown                   2.6.11     
MarkupSafe                 1.1.1      
mock                       2.0.0      
mongomock                  3.15.0     
monotonic                  1.5        
more-itertools             7.2.0      
moto                       1.3.5      
msrest                     0.4.29     
msrestazure                0.4.34     
multi-key-dict             2.0.3      
multidict                  4.5.2      
mysqlclient                1.4.2.post1
ndg-httpsclient            0.5.1      
nose                       1.3.7      
nose-ignore-docstring      0.2        
nose-timer                 0.7.5      
ntlm-auth                  1.2.0      
numpy                      1.16.0     
oauth2client               4.1.2      
oauthlib                   3.0.1      
ordered-set                3.1.1      
ordereddict                1.1        
packaging                  19.0       
pandas                     0.23.4     
pandas-gbq                 0.9.0      
parameterized              0.7.0      
paramiko                   2.4.2      
pbr                        5.1.3      
pendulum                   1.4.4      
pinotdb                    0.2.2      
pip                        19.0.3     
pluggy                     0.13.0     
protobuf                   3.7.0      
psutil                     5.6.1      
psycopg2                   2.7.7      
py                         1.8.0      
pyaml                      18.11.0    
pyasn1                     0.4.5      
pyasn1-modules             0.2.4      
pycparser                  2.19       
pycryptodome               3.7.3      
pycryptodomex              3.7.3      
pydata-google-auth         0.1.3      
pydruid                    0.5.2      
Pygments                   2.3.1      
PyHive                     0.6.1      
PyJWT                      1.7.1      
pykerberos                 1.2.1      
pymongo                    3.7.2      
pymssql                    2.1.4      
PyNaCl                     1.3.0      
pynamodb                   4.0.0      
pyOpenSSL                  19.0.0     
pyparsing                  2.3.1      
pyperclip                  1.5.27     
pysftp                     0.2.9      
PySmbClient                0.1.5      
pytest                     3.2.1      
python-daemon              2.1.2      
python-dateutil            2.8.0      
python-editor              1.0.4      
python-http-client         3.1.0      
python-jenkins             1.4.0      
python-jose                2.0.2      
python-nvd3                0.15.0     
python-slugify             3.0.0      
python3-openid             3.1.0      
pytz                       2018.9     
pytzdata                   2018.9     
pywinrm                    0.3.0      
PyYAML                     3.12       
qds-sdk                    1.11.0     
raven                      5.22.0     
redis                      3.3.11     
rednose                    1.3.0      
requests                   2.7.0      
requests-aws4auth          0.9        
requests-futures           0.9.4      
requests-kerberos          0.12.0     
requests-mock              1.5.2      
requests-ntlm              1.1.0      
requests-oauthlib          1.2.0      
requests-toolbelt          0.9.1      
responses                  0.10.5     
room-image-classification  2.3.3      
rootpath                   0.1.1      
rsa                        4.0        
s3transfer                 0.2.1      
sasl                       0.2.1      
SecretStorage              3.1.1      
sendgrid                   5.6.0      
sentinels                  1.0.0      
setproctitle               1.1.10     
setuptools                 40.6.2     
simplejson                 3.8.2      
six                        1.12.0     
slackclient                1.3.1      
smmap2                     2.0.5      
snakebite                  2.11.0     
snowballstemmer            1.2.1      
snowflake-connector-python 1.7.8      
snowflake-sqlalchemy       1.1.15     
Sphinx                     1.8.5      
sphinx-argparse            0.2.5      
Sphinx-PyPI-upload         0.2.1      
sphinx-rtd-theme           0.4.3      
sphinxcontrib-httpdomain   1.7.0      
sphinxcontrib-websupport   1.1.0      
SQLAlchemy                 1.3.10     
sqlparse                   0.3.0      
sshtunnel                  0.1.4      
statsd                     3.3.0      
tabulate                   0.8.2      
tenacity                   4.8.0      
termcolor                  1.1.0      
termstyle                  0.1.11     
text-unidecode             1.2        
thrift                     0.11.0     
thrift-sasl                0.3.0      
toml                       0.10.0     
tornado                    5.1.1      
tox                        3.14.0     
tqdm                       4.36.1     
typing-extensions          3.7.4.1    
tzlocal                    1.5.1      
unicodecsv                 0.14.1     
uritemplate                3.0.0      
urllib3                    1.24.1     
uvloop                     0.13.0     
vertica-python             0.9.0      
vine                       1.2.0      
virtualenv                 16.7.7     
webencodings               0.5.1      
websocket-client           0.54.0     
Werkzeug                   0.14.1     
wrapt                      1.11.1     
WTForms                    2.2.1      
xlrd                       1.2.0      
xmltodict                  0.12.0     
yarl                       1.3.0      
zdesk                      2.7.1      
zipp                       0.6.0      
zope.deprecation           4.4.0

  2. What did you do?

    engine = create_engine(os.getenv('AIRFLOW_CONN_SNOWFLAKE_URL'))
engine.connect()

  3. What did you expect to see? We should be able to connect.

  4. What did you see instead?

The connector breaks when trying to connect because it's passing in a cert_reqs parameter to urllib3, but I believe urllib3 <1.25.0 does not support this parameter.

If we only import snowflake-sqlalchemy, this will install the latest version of the snowflake python connector, which in this case has breaking changes.

  1. Can you set logging to DEBUG and collect the logs? When running engine.connect():
>     with self.engine.connect():

tests/unit/airflow/operators/test_merge_stg_snowflake_operator.py:41: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib/python3.6/site-packages/sqlalchemy/engine/base.py:2206: in connect
    return self._connection_cls(self, **kwargs)
env/lib/python3.6/site-packages/sqlalchemy/engine/base.py:103: in __init__
    else engine.raw_connection()
env/lib/python3.6/site-packages/sqlalchemy/engine/base.py:2306: in raw_connection
    self.pool.unique_connection, _connection
env/lib/python3.6/site-packages/sqlalchemy/engine/base.py:2275: in _wrap_pool_connect
    return fn()
env/lib/python3.6/site-packages/sqlalchemy/pool/base.py:303: in unique_connection
    return _ConnectionFairy._checkout(self)
env/lib/python3.6/site-packages/sqlalchemy/pool/base.py:760: in _checkout
    fairy = _ConnectionRecord.checkout(pool)
env/lib/python3.6/site-packages/sqlalchemy/pool/base.py:492: in checkout
    rec = pool._do_get()
env/lib/python3.6/site-packages/sqlalchemy/pool/impl.py:139: in _do_get
    self._dec_overflow()
env/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
env/lib/python3.6/site-packages/sqlalchemy/util/compat.py:153: in reraise
    raise value
env/lib/python3.6/site-packages/sqlalchemy/pool/impl.py:136: in _do_get
    return self._create_connection()
env/lib/python3.6/site-packages/sqlalchemy/pool/base.py:308: in _create_connection
    return _ConnectionRecord(self)
env/lib/python3.6/site-packages/sqlalchemy/pool/base.py:437: in __init__
    self.__connect(first_connect_check=True)
env/lib/python3.6/site-packages/sqlalchemy/pool/base.py:639: in __connect
    connection = pool._invoke_creator(self)
env/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py:114: in connect
    return dialect.connect(*cargs, **cparams)
env/lib/python3.6/site-packages/sqlalchemy/engine/default.py:481: in connect
    return self.dbapi.connect(*cargs, **cparams)
env/lib/python3.6/site-packages/snowflake/connector/__init__.py:33: in Connect
    return SnowflakeConnection(**kwargs)
env/lib/python3.6/site-packages/snowflake/connector/connection.py:187: in __init__
    self.connect(**kwargs)
env/lib/python3.6/site-packages/snowflake/connector/connection.py:484: in connect
    self.__open_connection()
env/lib/python3.6/site-packages/snowflake/connector/connection.py:702: in __open_connection
    self.__authenticate(auth_instance)
env/lib/python3.6/site-packages/snowflake/connector/connection.py:933: in __authenticate
    session_parameters=self._session_parameters,
env/lib/python3.6/site-packages/snowflake/connector/auth.py:197: in authenticate
    socket_timeout=self._rest._connection.login_timeout)
env/lib/python3.6/site-packages/snowflake/connector/network.py:533: in _post_request
    _include_retry_params=_include_retry_params)
env/lib/python3.6/site-packages/snowflake/connector/network.py:612: in fetch
    **kwargs)
env/lib/python3.6/site-packages/snowflake/connector/network.py:714: in _request_exec_wrapper
    raise e
env/lib/python3.6/site-packages/snowflake/connector/network.py:652: in _request_exec_wrapper
    **kwargs)
env/lib/python3.6/site-packages/snowflake/connector/network.py:913: in _request_exec
    raise err
env/lib/python3.6/site-packages/snowflake/connector/network.py:802: in _request_exec
    auth=SnowflakeAuth(token),
env/lib/python3.6/site-packages/requests/sessions.py:465: in request
    resp = self.send(prep, **send_kwargs)
env/lib/python3.6/site-packages/requests/sessions.py:573: in send
    r = adapter.send(request, **kwargs)
env/lib/python3.6/site-packages/requests/adapters.py:370: in send
    timeout=timeout
env/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py:544: in urlopen
    body=body, headers=headers)
env/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py:341: in _make_request
    self._validate_conn(conn)
env/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py:761: in _validate_conn
    conn.connect()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7409b58908>

    def connect(self):
        # Add certificate verification
        conn = self._new_conn()

        resolved_cert_reqs = resolve_cert_reqs(self.cert_reqs)
        resolved_ssl_version = resolve_ssl_version(self.ssl_version)

        hostname = self.host
        if getattr(self, '_tunnel_host', None):
            # _tunnel_host was added in Python 2.6.3
            # (See: http://hg.python.org/cpython/rev/0f57b30a152f)

            self.sock = conn
            # Calls self._set_hostport(), so self.host is
            # self._tunnel_host below.
            self._tunnel()
            # Mark this connection as not reusable
            self.auto_open = 0

            # Override the host with the one we're requesting data from.
            hostname = self._tunnel_host

        is_time_off = datetime.date.today() < RECENT_DATE
        if is_time_off:
            warnings.warn((
                'System time is way off (before {0}). This will probably '
                'lead to SSL verification errors').format(RECENT_DATE),
                SystemTimeWarning
            )

        # Wrap socket using verification with the root certs in
        # trusted_root_certs
        self.sock = ssl_wrap_socket(conn, self.key_file, self.cert_file,
                                    cert_reqs=resolved_cert_reqs,
                                    ca_certs=self.ca_certs,
                                    server_hostname=hostname,
>                                   ssl_version=resolved_ssl_version)
E       TypeError: ssl_wrap_socket_with_ocsp() got an unexpected keyword argument 'cert_reqs'
smtakeda commented 4 years ago

Please try v2.0.3 and use the latest requests.

sfc-gh-stakeda commented 4 years ago

This duplicates https://github.com/snowflakedb/snowflake-connector-python/issues/238 Should be fixed in v2.1.0

sfc-gh-stakeda commented 4 years ago

@jtalmi , can you please try v2.1.0 and let us know?

github-actions[bot] commented 1 year ago

To clean up and re-prioritize bugs and feature requests we are closing all issues older than 6 months as of Apr 1, 2023. If there are any issues or feature requests that you would like us to address, please re-create them. For urgent issues, opening a support case with this link Snowflake Community is the fastest way to get a response