search

snowflakedb / snowflake-connector-python

Snowflake Connector for Python
https://pypi.python.org/pypi/snowflake-connector-python/
Apache License 2.0
582 stars 468 forks source link

SNOW-169155: snowflake.connector.errors.OperationalError: 250003: Failed to execute request: 'SSLSocket' object has no attribute 'connection' #324

Closed jasonlcy2020 closed 3 years ago

jasonlcy2020 commented 4 years ago

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using (python --version)? Python 3.7.3
  2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? Linux-4.15.0-1050-azure-x86_64-with-debian-stretch-sid
  3. What are the component versions in the environment (pip list)? 
    
Package             Version               
------------------- ----------------------
asn1crypto          0.24.0                
backcall            0.1.0                 
boto                2.49.0                
boto3               1.9.162               
botocore            1.12.163              
certifi             2019.3.9              
cffi                1.12.2                
chardet             3.0.4                 
cryptography        2.6.1                 
cycler              0.10.0                
Cython              0.29.6                
decorator           4.4.0                 
docutils            0.14                  
idna                2.8                   
ipykernel           5.1.0                 
ipython             7.4.0                 
ipython-genutils    0.2.0                 
jedi                0.13.3                
jmespath            0.9.4                 
jupyter-client      5.2.4                 
jupyter-core        4.4.0                 
kiwisolver          1.1.0                 
matplotlib          3.0.3                 
numpy               1.16.2                
pandas              0.24.2                
parso               0.3.4                 
patsy               0.5.1                 
pexpect             4.6.0                 
pickleshare         0.7.5                 
pip                 19.0.3                
prompt-toolkit      2.0.9                 
psycopg2            2.7.6.1               
ptyprocess          0.6.0                 
pyarrow             0.13.0                
pycparser           2.19                  
pycurl              7.43.0                
Pygments            2.3.1                 
pygobject           3.20.0                
pyOpenSSL           19.0.0                
pyparsing           2.4.6                 
PySocks             1.6.8                 
python-apt          1.1.0b1+ubuntu0.16.4.8
python-dateutil     2.8.0                 
pytz                2018.9                
pyzmq               18.0.0                
requests            2.21.0                
s3transfer          0.2.1                 
scikit-learn        0.20.3                
scipy               1.2.1                 
seaborn             0.9.0                 
setuptools          40.8.0                
six                 1.12.0                
ssh-import-id       5.5                   
statsmodels         0.9.0                 
tornado             6.0.2                 
traitlets           4.3.2                 
unattended-upgrades 0.1                   
urllib3             1.24.1                
virtualenv          16.4.1                
wcwidth             0.1.7                 
wheel               0.33.1                
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

4. What did you do?
Installed snowflake-connector-python on databricks and just trying to connect but failed. It was ok all the time until today:

dbutils.library.installPyPI('snowflake-connector-python', version='2.2.7') import snowflake.connector conn = snowflake.connector.connect(....

5. What did you expect to see?
connection object returned

6. What did you see instead?
Error as titled.

7. Can you set logging to DEBUG and collect the logs?

Installed snowflake-connector-python=2.2.7 Installed azure-cli-core Installed azure-common 2020-06-18 09:16:58,249 - MainThread ssl_wrap_socket.py:91 - inject_into_urllib3() - DEBUG - Injecting ssl_wrap_socket_with_ocsp 2020-06-18 09:16:58,250 - MainThread auth.py:52 - () - DEBUG - Failed to import keyring module. err=[No module named 'keyring'] 2020-06-18 09:16:58,250 - MainThread auth.py:72 - () - DEBUG - cache directory: /root/.cache/snowflake /local_disk0/pythonVirtualEnvDirs/virtualEnv-791d3104-b985-4b9f-b2b0-7e4ceb1830a0/lib/python3.7/site-packages/snowflake/connector/options.py:39: UserWarning: You have an incompatible version of 'pyarrow' installed, please install a version that adheres to: 'pyarrow<0.18.0,>=0.17.0; extra == "pandas"' warn_incompatible_dep('pyarrow', _installed_pyarrow.version, _expected_version) 2020-06-18 09:17:01,247 - MainThread connection.py:180 - init() - INFO - Snowflake Connector for Python Version: 2.2.7, Python Version: 3.7.3, Platform: Linux-4.15.0-1050-azure-x86_64-with-debian-stretch-sid 2020-06-18 09:17:01,247 - MainThread connection.py:487 - connect() - DEBUG - connect 2020-06-18 09:17:01,247 - MainThread connection.py:725 - config() - DEBUG - config 2020-06-18 09:17:01,247 - MainThread connection.py:820 - config() - INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity. 2020-06-18 09:17:01,247 - MainThread connection.py:836 - config() - INFO - Setting use_openssl_only mode to False 2020-06-18 09:17:01,248 - MainThread converter.py:138 - init() - DEBUG - use_numpy: False 2020-06-18 09:17:01,248 - MainThread connection.py:649 - __open_connection() - DEBUG - REST API object was created: grab.southeast-asia.azure.snowflakecomputing.com:443 2020-06-18 09:17:01,248 - MainThread auth.py:137 - authenticate() - DEBUG - authenticate 2020-06-18 09:17:01,248 - MainThread auth.py:172 - authenticate() - DEBUG - assertion content: ***** 2020-06-18 09:17:01,248 - MainThread auth.py:184 - authenticate() - DEBUG - account=grab, user=[REDACTED], database=STGFDW, schema=ANALYTICS, warehouse=FDWANALYTICSWH, role=ANALYST, request_id=f54c8219-fbac-4d0f-a284-9570aa874eb0 2020-06-18 09:17:01,248 - MainThread auth.py:210 - authenticate() - DEBUG - body['data']: {'CLIENT_APP_ID': 'PythonConnector', 'CLIENT_APP_VERSION': '2.2.7', 'SVN_REVISION': None, 'ACCOUNT_NAME': 'grab', 'LOGIN_NAME': '[REDACTED]', 'CLIENT_ENVIRONMENT': {'APPLICATION': 'PythonConnector', 'OS': 'Linux', 'OS_VERSION': 'Linux-4.15.0-1050-azure-x86_64-with-debian-stretch-sid', 'PYTHON_VERSION': '3.7.3', 'PYTHON_RUNTIME': 'CPython', 'PYTHON_COMPILER': 'GCC 5.4.0 20160609', 'OCSP_MODE': 'FAIL_OPEN', 'TRACING': 10, 'LOGIN_TIMEOUT': 120, 'NETWORK_TIMEOUT': None, 'CLIENT_STORE_TEMPORARY_CREDENTIAL': None}, 'SESSION_PARAMETERS': {'CLIENT_SESSION_KEEP_ALIVE_HEARTBEAT_FREQUENCY': 900, 'CLIENT_PREFETCH_THREADS': 4}} 2020-06-18 09:17:01,248 - MainThread network.py:940 - _use_requests_session() - DEBUG - Active requests sessions: 1, idle: 0 2020-06-18 09:17:01,248 - MainThread network.py:636 - _request_exec_wrapper() - DEBUG - remaining request timeout: 120, retry cnt: 1 2020-06-18 09:17:01,249 - MainThread network.py:784 - _request_exec() - DEBUG - socket timeout: 60 2020-06-18 09:17:01,342 - MainThread ocsp_snowflake.py:382 - reset_cache_dir() - DEBUG - cache directory: /root/.cache/snowflake 2020-06-18 09:17:01,392 - MainThread ssl_wrap_socket.py:406 - ssl_wrap_socket_with_ocsp() - DEBUG - OCSP Mode: FAIL_OPEN, OCSP response cache file name: None 2020-06-18 09:17:01,392 - MainThread ocsp_snowflake.py:421 - reset_ocsp_response_cache_uri() - DEBUG - ocsp_response_cache_uri: file:///root/.cache/snowflake/ocsp_response_cache.json 2020-06-18 09:17:01,392 - MainThread ocsp_snowflake.py:423 - reset_ocsp_response_cache_uri() - DEBUG - OCSP_VALIDATION_CACHE size: 0 2020-06-18 09:17:01,392 - MainThread ocsp_snowflake.py:217 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP response cache server is enabled: http://ocsp.snowflakecomputing.com/ocsp_response_cache.json 2020-06-18 09:17:01,392 - MainThread ocsp_snowflake.py:251 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP dynamic cache server RETRY URL: None 2020-06-18 09:17:01,396 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 2020-06-18 09:17:01,397 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:08:26+00:00 2020-06-18 09:17:01,401 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 16:58:09+00:00 2020-06-18 09:17:01,407 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:44:21+00:00 2020-06-18 09:17:01,409 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:08:26+00:00 2020-06-18 09:17:01,412 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 2020-06-18 09:17:01,415 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 16:58:09+00:00 2020-06-18 09:17:01,417 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 2020-06-18 09:17:01,420 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 16:58:09+00:00 2020-06-18 09:17:01,422 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-06-18 09:17:01,423 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:40:26+00:00 2020-06-18 09:17:01,427 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 2020-06-18 09:17:01,429 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 2020-06-18 09:17:01,432 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-06-18 09:17:01,435 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-06-18 09:17:01,438 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-06-18 09:17:01,441 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 2020-06-18 09:17:01,443 - MainThread ocsp_asn1crypto.py:205 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-06-18 09:17:01,443 - MainThread ocsp_snowflake.py:459 - read_ocsp_response_cache_file() - DEBUG - Read OCSP response cache file: /root/.cache/snowflake/ocsp_response_cache.json, count=97 2020-06-18 09:17:01,444 - MainThread network.py:900 - _request_exec() - DEBUG - Hit retryable client error. Retrying... Ignore the following error stack: 'SSLSocket' object has no attribute 'connection' Traceback (most recent call last): File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-791d3104-b985-4b9f-b2b0-7e4ceb1830a0/lib/python3.7/site-packages/snowflake/connector/network.py", line 808, in _request_exec auth=SnowflakeAuth(token), File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-791d3104-b985-4b9f-b2b0-7e4ceb1830a0/lib/python3.7/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, send_kwargs) File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-791d3104-b985-4b9f-b2b0-7e4ceb1830a0/lib/python3.7/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, kwargs) File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-791d3104-b985-4b9f-b2b0-7e4ceb1830a0/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/databricks/python/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/databricks/python/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/databricks/python/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/databricks/python/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect ssl_context=context) File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-791d3104-b985-4b9f-b2b0-7e4ceb1830a0/lib/python3.7/site-packages/snowflake/connector/ssl_wrap_socket.py", line 411, in ssl_wrap_socket_with_ocsp ).validate(server_hostname, ret.connection) AttributeError: 'SSLSocket' object has no attribute 'connection'

asadkarim7 commented 4 years ago

Hi I am having a same issue as well. My build was working fine yesterday. @sfc-gh-stakeda can you help

sfc-gh-mkeller commented 4 years ago

We saw this issue yesterday with the new release of requests, but you don't have that. Let me look into this!

bradfordlittooysonos commented 4 years ago

Hi, we had this issue as well. Fixing requests to 2.23.0 or lower seems to fix the issue for us.

Seems like the most recent release of requests broke this module: https://github.com/psf/requests/blob/master/HISTORY.md#2240-2020-06-17

Hope this helps.

sfc-gh-mkeller commented 4 years ago

Note that we pin our version against new releases, in this case:

https://github.com/snowflakedb/snowflake-connector-python/blob/7314a85bcf9711b0279c8c94f570fe25dc067b08/setup.py#L176

seoyeonberry commented 4 years ago

I am from the issue #325. In my case, requests==2.22.0 but got the same error.

sfc-gh-mkeller commented 4 years ago

I'm still unable to reproduce this issue without upgrading my requests version to 2.24.0 Is it possible @jasonlcy2020 @asadkarim7 that dbutils.library.installPyPI('snowflake-connector-python', version='2.2.7') installs the newest version of requests?

asadkarim7 commented 4 years ago

@sfc-gh-mkeller I changed my requirement.txt file to include requests==2.23.0, just like @bradfordlittooysonos suggested and my build didn't break this time. It ran successfully

sfc-gh-mkeller commented 4 years ago

@seoyeonberry @jasonlcy2020 Could you guys insert the following right before creating the connection and post it's output, please?

import pkg_resources
print(pkg_resources.working_set.by_key['requests'])
print(pkg_resources.working_set.by_key['urllib3'])
seoyeonberry commented 4 years ago

@sfc-gh-mkeller I am working an another virtual environment and ran the requirements.txt again. The version is slightly different but got the same error message. Here it is.

requests 2.24.0 urllib3 1.24.3

sfc-gh-mkeller commented 4 years ago

@seoyeonberry We don't support requests 2.24.0 yet. You need to downgrade it to 2.23.0

seoyeonberry commented 4 years ago

@sfc-gh-mkeller I downgraded requests to 2.23.0 and it worked. This is kind of strange because requests was even not on the requirements.txt file. Did Snowflake make changes recently? with 2.24.0 it worked well until last week.

sfc-gh-mkeller commented 4 years ago

We made no changes, requests=2.24.0 came out yesterday https://pypi.org/project/requests/#history

seoyeonberry commented 4 years ago

Then my assumption is that one of the packages in the requirements.txt file(snowflake connector or sth) installs requests=2.24.0 default since yesterday. FYI, I checked the log and it says that the program got broken yesterday.

sfc-gh-mkeller commented 4 years ago

It's great that we found the source of the issue. I'll learn my lesson and next time when we see an internal error like this then I'll proactively announce it here too.

chyzzqo2 commented 4 years ago

Would better if you stop monkey patching requests.

sfc-gh-mkeller commented 4 years ago

That'd require requests to add a hook for our custom verification. This is not a bad idea! Have we ever tried reaching out @sfc-gh-hchaturvedi ?

jdotjdot commented 4 years ago

You could also vendor requests in your package.

On Fri, Jun 19, 2020 at 19:12 Mark Keller notifications@github.com wrote:

That'd require requests to add a hook for our custom verification. This is not a bad idea! Have we ever tried reaching out @sfc-gh-hchaturvedi https://github.com/sfc-gh-hchaturvedi ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/snowflakedb/snowflake-connector-python/issues/324#issuecomment-646892555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKEOQL3UZR6TMYGJKBIPQ3RXPWHRANCNFSM4OBMNSDQ .

WolVecz commented 4 years ago

Please look at swetashre's linked problem. Literally just importing the python snowflake-connector breaks the use of the Boto3 queries.

yousefissa commented 4 years ago

Any updates on this? Still experiencing it with requests being pinned to 2.23.0 and snowflake-connector-python=2.2.4

jasonlcy91 commented 3 years ago

I dont experience error anymore

yousefissa commented 3 years ago

Which versions are you using?

manugarri commented 3 years ago

hey guys i maintain an alternate lite version of snowconn python connector here my hope is we can add changes to the package that eventually will be added to the official one.

Here is the repo in case someone wants to help. I would like to add:

jdotjdot commented 3 years ago

When will newer versions of requests be supported? With the updates to dependency resolution in pip, it's going to become very hard to keep requests pinned to older versions.

sfc-gh-mkeller commented 3 years ago

When will newer versions of requests be supported? With the updates to dependency resolution in pip, it's going to become very hard to keep requests pinned to older versions.

I don't know 👀 https://github.com/snowflakedb/snowflake-connector-python/pull/557

WolVecz commented 3 years ago

It is a pretty big problem for us to be pinned to PyArrow at 0.17 and Requests at 2.23. The number of environmental issues I have run into, solely because of the Snowflake-connector has been rough. Would love to see these resolved.

jdotjdot commented 3 years ago

Agreed, it's a really huge problem now. If there was any way to connect to Snowflake using more standard tools then we'd just use that, but as it is, we have to use this connector to connect.

We're at the point where we're considering writing a proxy service that just forwards requests to Snowflake using a different language's client library or using the JDBC driver which... is obviously not what customers want to be doing.

manugarri commented 3 years ago

another weird thing, this package boto3 requirements are 

snowflake-connector-python 2.1.3 requires boto3<1.11.0,>=1.4.4, but you'll have boto3 1.16.35 which is incompatible.

But the latest boto3 release is 1.16.35, how does that even work?

potiuk commented 3 years ago

It is a serious problem for Airflow 2.0 (which is due to be released this week). We introduced separation between different providers (we have 60+ airflow-providers-* packages including snowflake). Installing snowflake provider breaks almost all other providers instantly (because we introduced automated discovery of providers and we import some of the classes from every provider to check out what are the capabilities of those).

We just decided to release snowflake provider but immediately yank it until the problem is solved.

We are tracking it in https://github.com/apache/airflow/issues/12881

CameronGibson commented 3 years ago

So in my case I am adding a Twilio and SendGrid layer for failure alerts to staff, all I needed to do was change the ordering layer to have Twilio and SendGrids layer merge before our Snowflake Connector layer. Hope this helps somebody.

sfc-gh-hkapre commented 3 years ago

This should have been resolved in v2.3.8 which vendored requests and urllib3. Please reopen if the issues with requests are still occurring.

docbyte86 commented 2 years ago

I still see the issue while installing pip package snowflake-connector-python 2.7.4. As a dependency the pip is installing 2.27.1, which is buggy.

Collecting requests<3.0.0 Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) |████████████████████████████████| 63 kB 1.7 MB/s

Only a work around, downgrading requests package to 2.23.0 is solving the problem