snowflakedb / snowflake-connector-python

Snowflake Connector for Python
https://pypi.python.org/pypi/snowflake-connector-python/
Apache License 2.0
601 stars 473 forks source link

SNOW-1774037: OCSP check fails when execption is not of type RevocationCheckError #2094

Open knowhoper opened 3 weeks ago

knowhoper commented 3 weeks ago

Python version

3.10.3

Operating system and processor architecture

macOS-14.7-arm64-arm-64bit

Installed packages

asn1crypto==1.5.1
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
cryptography==43.0.3
filelock==3.16.1
idna==3.10
packaging==24.1
platformdirs==4.3.6
pycparser==2.22
PyJWT==2.9.0
pyOpenSSL==24.2.1
pytz==2024.2
requests==2.32.3
sortedcontainers==2.4.0
tomlkit==0.13.2
typing_extensions==4.12.2
urllib3==2.2.3
snowflake-connector-python==3.12.3

What did you do?

OCSP errors fail when the Exception thrown is not of type RevocationCheckError. This is on any invocation of the connector in our infrastructure - currently trying to diagnose why we have OCSP issues.

Fails with error

 File "/opt/pysetup/.venv/lib/python3.10/site-packages/snowflake/connector/ocsp_snowflake.py", line 1147, in verify_fail_open
    if ex_obj.errno is ER_OCSP_RESPONSE_CERT_STATUS_REVOKED:
AttributeError: 'TypeError' object has no attribute 'errno'

### What did you expect to see?

A more concise OCSP error or results from the Snowflake service.

https://github.com/snowflakedb/snowflake-connector-python/blob/main/src/snowflake/connector/ocsp_snowflake.py#L1147

This line fails when the type passed is of type Exception as Exception has no property errno. This is causing issues in our stack due to some OCSP error in our infrastructure we can't diagnose due to the error routine being incorrect in the connector

### Can you set logging to DEBUG and collect the logs?

```bash
import logging
import os

for logger_name in ('snowflake.connector',):
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)
sfc-gh-dszmolka commented 3 weeks ago

hi - thanks for raising this issue with us. taking a look. do you think it would be possible to post a full(er) stack of the error thrown? we supposed to raise a RevocationCheckError when the OCSP server is not reachable, so really curious what else is going on here.

edit: re: to diagnose why you're having OCSP errors, this could be helpful:

knowhoper commented 2 weeks ago

Hi @sfc-gh-dszmolka , thank you for the follow up. Yes, we are currently in the process of diagnosing the OCSP issue within our environment, suspect its a timeout due to some missing whitelisting. Errors started appearing with no code changes on our side so assuming something in either our SF tenant or GCP infra has changed. Will report back findings.

Regardless, I suspect the error routine there is incorrect, which is making pin pointing this issue harder.

Thanks again.

knowhoper commented 2 weeks ago

Here are the results from the OCSP check, noting the failure on

OCSP_RESPONDER: ocsp.pki.goog:80: URL Check: Failed: HTTP/1.1 404 Not Found

=========Connectivity diagnostic report================================
INITIAL: Specified snowflake account: redacted
INITIAL: Host based on specified account: redacted.us-central1.gcp.snowflakecomputing.com

=========Proxy information - These are best guesses, not guarantees====
PROXY: Proxies with Env vars removed(SYSTEM PROXIES): {}
PROXY: Proxies with Env vars restored(ENV PROXIES): {}
PROXY: There is likely a proxy because the issuer for www.google.com is not correct. Got C=US; O=Google Trust Services; CN=WE2 and expected one of ('C=US; O=Google Trust Services LLC', 'C=US; O=Amazon', 'C=US; O=DigiCert Inc')
PROXY: Proxy check using invalid URL did not show proxy: Review result, but you can probably ignore: Result: HTTPSConnectionPool(host='ireallyshouldnotexistatallanywhere.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<snowflake.connector.vendored.urllib3.connection.HTTPSConnection object at 0x3e16d64b8b80>: Failed to establish a new connection: [Errno -2] Name or service not known'))

=========Snowflake URL information=====================================
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: nslookup results: public ip: 34.70.63.80
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com:443: URL Check: Connected Successfully
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: Cert info:
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: subject: {'C': 'US', 'ST': 'California', 'L': 'San Mateo', 'O': 'Snowflake Inc.', 'CN': '*.us-central1.gcp.snowflakecomputing.com'}
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: issuer: {'C': 'US', 'O': 'DigiCert Inc', 'CN': 'DigiCert Global G2 TLS RSA SHA256 2020 CA1'}
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: serialNumber: 2619509689619176879997026077985058025
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: version: 2
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: notBefore: 2024-01-29 00:00:00
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: notAfter: 2025-01-28 23:59:59
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: subjectAltName: DNS:*.us-central1.gcp.snowflakecomputing.com, DNS:*.snowflake.app, DNS:*.gcpuscentral1.snowflake.app, DNS:*.snowflakecomputing.com, DNS:*.global.snowflakecomputing.com, DNS:us-central1.gcp.snowflakecomputing.com, DNS:snowflake.app, DNS:gcpuscentral1.snowflake.app, DNS:snowflakecomputing.com, DNS:global.snowflakecomputing.com
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: ocspURLs: ['ocsp.digicert.com', 'cacerts.digicert.com']

=========Snowflake Stage information===================================
We retrieved stage info from the allowlist
STAGE: storage.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: storage.googleapis.com:443: URL Check: Connected Successfully
STAGE: storage.googleapis.com: Cert info:
STAGE: storage.googleapis.com: subject: {'CN': 'storage.googleapis.com'}
STAGE: storage.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: storage.googleapis.com: serialNumber: 45079917743581688039704620458248828968
STAGE: storage.googleapis.com: version: 2
STAGE: storage.googleapis.com: notBefore: 2024-10-21 08:40:59
STAGE: storage.googleapis.com: notAfter: 2025-01-13 08:40:58
STAGE: storage.googleapis.com: subjectAltName: DNS:storage.googleapis.com
STAGE: storage.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: storage.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com:443: URL Check: Connected Successfully
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: Cert info:
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: version: 2
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: storage-upload.googleapis.com: nslookup results: ['142.250.66.187', '172.217.24.59']
STAGE: storage-upload.googleapis.com:443: URL Check: Connected Successfully
STAGE: storage-upload.googleapis.com: Cert info:
STAGE: storage-upload.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: storage-upload.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: storage-upload.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: storage-upload.googleapis.com: version: 2
STAGE: storage-upload.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: storage-upload.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: storage-upload.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: storage-upload.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: storage-upload.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com:443: URL Check: Connected Successfully
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: Cert info:
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: version: 2
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: storage-download.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: storage-download.googleapis.com:443: URL Check: Connected Successfully
STAGE: storage-download.googleapis.com: Cert info:
STAGE: storage-download.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: storage-download.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: storage-download.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: storage-download.googleapis.com: version: 2
STAGE: storage-download.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: storage-download.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: storage-download.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: storage-download.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: storage-download.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com:443: URL Check: Connected Successfully
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: Cert info:
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: version: 2
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: www.googleapis.com: nslookup results: ['142.251.221.74', '142.250.71.74', '142.250.66.234', '142.250.76.106', '172.217.167.74', '172.217.167.106', '142.250.66.202', '142.250.67.10', '142.250.204.10']
STAGE: www.googleapis.com:443: URL Check: Connected Successfully
STAGE: www.googleapis.com: Cert info:
STAGE: www.googleapis.com: subject: {'CN': 'upload.video.google.com'}
STAGE: www.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WR2'}
STAGE: www.googleapis.com: serialNumber: 249523738985507380678070046713553759211
STAGE: www.googleapis.com: version: 2
STAGE: www.googleapis.com: notBefore: 2024-10-07 08:25:41
STAGE: www.googleapis.com: notAfter: 2024-12-30 08:25:40
STAGE: www.googleapis.com: subjectAltName: DNS:upload.video.google.com, DNS:*.clients.google.com, DNS:*.docs.google.com, DNS:*.drive.google.com, DNS:*.gdata.youtube.com, DNS:*.googleapis.com, DNS:*.photos.google.com, DNS:*.youtube-3rd-party.com, DNS:upload.google.com, DNS:*.upload.google.com, DNS:upload.youtube.com, DNS:*.upload.youtube.com, DNS:uploads.stage.gdata.youtube.com, DNS:bg-call-donation.goog, DNS:bg-call-donation-alpha.goog, DNS:bg-call-donation-canary.goog, DNS:bg-call-donation-dev.goog
STAGE: www.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: www.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']

=========Snowflake OCSP information====================================
We were able to retrieve system allowlist.
These OCSP hosts came from the certificate and the allowlist.
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.47
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.5
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.122
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.11
OCSP_RESPONDER: ocsp.snowflakecomputing.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: cacerts.digicert.com: nslookup results: ['152.195.38.76']
OCSP_RESPONDER: cacerts.digicert.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.digicert.com: nslookup results: ['152.195.38.76']
OCSP_RESPONDER: ocsp.digicert.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: e5.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: e5.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: e7.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: e7.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: e6.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: e6.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.r2m01.amazontrust.com: nslookup results: ['18.67.98.168']
OCSP_RESPONDER: ocsp.r2m01.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: e8.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: e8.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.rootg2.amazontrust.com: nslookup results: ['13.35.146.220']
OCSP_RESPONDER: ocsp.rootg2.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: e9.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: e9.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.rootca1.amazontrust.com: nslookup results: ['13.35.146.220']
OCSP_RESPONDER: ocsp.rootca1.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: r10.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: r10.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.r2m03.amazontrust.com: nslookup results: ['18.67.98.168']
OCSP_RESPONDER: ocsp.r2m03.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.pki.goog: nslookup results: ['172.217.24.35']
OCSP_RESPONDER: ocsp.pki.goog:80: URL Check: Failed: HTTP/1.1 404 Not Found
Date: Thu, 07 Nov 2024 23:04:58 GMT
Content-Type: text/html; charset=UTF-8
Server: ocsp_responder
Content-Length: 1561
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/</code> was not found on this server.  <ins>That’s all we know.</ins>

OCSP_RESPONDER: ocsp.sca1b.amazontrust.com: nslookup results: ['18.67.95.191', '18.67.95.118', '18.67.95.61', '18.67.95.7']
OCSP_RESPONDER: ocsp.sca1b.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: r11.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: r11.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: o.pki.goog: nslookup results: ['172.217.24.35']
OCSP_RESPONDER: o.pki.goog:80: URL Check: Failed: HTTP/1.1 404 Not Found
Date: Thu, 07 Nov 2024 23:04:58 GMT
Content-Type: text/html; charset=UTF-8
Server: ocsp_responder
Content-Length: 1561
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/</code> was not found on this server.  <ins>That’s all we know.</ins>

OCSP_RESPONDER: r14.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: r14.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.r2m02.amazontrust.com: nslookup results: ['18.67.98.168']
OCSP_RESPONDER: ocsp.r2m02.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: r12.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: r12.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: r13.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: r13.o.lencr.org:80: URL Check: Connected Successfully

=========Snowflake Out of bound telemetry check========================
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: nslookup results: public ip: 54.191.121.111
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: nslookup results: public ip: 44.235.223.145
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: nslookup results: public ip: 44.229.173.185
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com:443: URL Check: Connected Successfully
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: Cert info:
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: subject: {'C': 'US', 'ST': 'California', 'L': 'San Mateo', 'O': 'Snowflake Inc.', 'CN': 'client-telemetry.snowflakecomputing.com'}
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: issuer: {'C': 'US', 'O': 'DigiCert Inc', 'CN': 'DigiCert Global G2 TLS RSA SHA256 2020 CA1'}
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: serialNumber: 14635881134541535973514408807947601738
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: version: 2
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: notBefore: 2024-03-22 00:00:00
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: notAfter: 2025-03-22 23:59:59
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: subjectAltName: DNS:client-telemetry.snowflakecomputing.com
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
sfc-gh-dszmolka commented 1 week ago

this issue seems to be coming from SnowCD itself, and is unrelated to the problem. (it looks to be trying to check http://ocsp.pki.goog OCSP Responder, but there's no OCSP Responder endpoint on / path of the said server. Please ignore this error line from SnowCD.

do you think it would be possible to post a full(er) stack of the error thrown?

Is it possible to share it, from the original issue you saw? the stack of exceptions from the the python application, not just a single error line?

flekkk commented 1 week ago

Hello, everyone. On our project there was exactly the same problem described above knowhoper. The problem appeared also unexpectedly and without any changes in code/infrastructure. The error in the logs fully corresponds to the error attached by knowhoper.

So far it turned out that everything works fine with insecure_mode=True.

I would be very grateful for any help and information related to this issue.

All the logs we were able to get:

DEFAULT 2024-11-11T11:44:41.372083Z [ERROR 14 2024-11-11 11:44:41,370] - snowflake.connector.file_transfer_agent 606 An exception was raised in <bound method SnowflakeStorageClient.prepare_upload of <snowflake.connector.gcs_storage_client.SnowflakeGCSRestClient object at 0x3e9339ee1990>>
ERROR 2024-11-11T11:44:41.372115Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1125, in validate_by_direct_connection self.process_ocsp_response(issuer, cert_id, ocsp_response) File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 310, in process_ocsp_response self.verify_signature( File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 372, in verify_signature public_key.verify( TypeError: _EllipticCurvePublicKey.verify() takes 4 positional arguments but 5 were given
DEFAULT 2024-11-11T11:44:41.372120Z During handling of the above exception, another exception occurred:
ERROR 2024-11-11T11:44:41.372129Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/snowflake/connector/file_transfer_agent.py", line 603, in function_and_callback_wrapper work(*args, **kwargs),
DEFAULT 2024-11-11T11:44:41.372132Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372135Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/storage_client.py", line 229, in prepare_upload
DEFAULT 2024-11-11T11:44:41.372138Z self.preprocess()
DEFAULT 2024-11-11T11:44:41.372141Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/storage_client.py", line 195, in preprocess
DEFAULT 2024-11-11T11:44:41.372144Z file_header = self.get_file_header(
DEFAULT 2024-11-11T11:44:41.372147Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372150Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/gcs_storage_client.py", line 344, in get_file_header
DEFAULT 2024-11-11T11:44:41.372153Z response = self._send_request_with_retry(
DEFAULT 2024-11-11T11:44:41.372156Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372160Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/storage_client.py", line 294, in _send_request_with_retry
DEFAULT 2024-11-11T11:44:41.372162Z response = session.request(verb, url, **rest_kwargs)
DEFAULT 2024-11-11T11:44:41.372165Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372168Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/requests/sessions.py", line 589, in request
DEFAULT 2024-11-11T11:44:41.372171Z resp = self.send(prep, **send_kwargs)
DEFAULT 2024-11-11T11:44:41.372174Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372177Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/requests/sessions.py", line 703, in send
DEFAULT 2024-11-11T11:44:41.372180Z r = adapter.send(request, **kwargs)
DEFAULT 2024-11-11T11:44:41.372182Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372186Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/requests/adapters.py", line 486, in send
DEFAULT 2024-11-11T11:44:41.372188Z resp = conn.urlopen(
DEFAULT 2024-11-11T11:44:41.372191Z ^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372196Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 715, in urlopen
DEFAULT 2024-11-11T11:44:41.372199Z httplib_response = self._make_request(
DEFAULT 2024-11-11T11:44:41.372202Z ^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372204Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 404, in _make_request
DEFAULT 2024-11-11T11:44:41.372207Z self._validate_conn(conn)
DEFAULT 2024-11-11T11:44:41.372211Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 1058, in _validate_conn
DEFAULT 2024-11-11T11:44:41.372214Z conn.connect()
DEFAULT 2024-11-11T11:44:41.372217Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connection.py", line 419, in connect
DEFAULT 2024-11-11T11:44:41.372220Z self.sock = ssl_wrap_socket(
DEFAULT 2024-11-11T11:44:41.372223Z ^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372226Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ssl_wrap_socket.py", line 91, in ssl_wrap_socket_with_ocsp
DEFAULT 2024-11-11T11:44:41.372229Z ).validate(server_hostname, ret.connection)
DEFAULT 2024-11-11T11:44:41.372232Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372235Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 999, in validate
DEFAULT 2024-11-11T11:44:41.372238Z return self._validate(
DEFAULT 2024-11-11T11:44:41.372241Z ^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372244Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1012, in _validate
DEFAULT 2024-11-11T11:44:41.372247Z results = self._validate_certificates_sequential(
DEFAULT 2024-11-11T11:44:41.372249Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372252Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1205, in _validate_certificates_sequential
DEFAULT 2024-11-11T11:44:41.372255Z r = self.validate_by_direct_connection(
DEFAULT 2024-11-11T11:44:41.372258Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372261Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1140, in validate_by_direct_connection
DEFAULT 2024-11-11T11:44:41.372264Z err = self.verify_fail_open(ex, telemetry_data)
DEFAULT 2024-11-11T11:44:41.372266Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372269Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1159, in verify_fail_open
DEFAULT 2024-11-11T11:44:41.372272Z if ex_obj.errno is ER_OCSP_RESPONSE_CERT_STATUS_REVOKED:
DEFAULT 2024-11-11T11:44:41.372275Z ^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.488608Z AttributeError: 'TypeError' object has no attribute 'errno'

ERROR 2024-11-11T11:44:41.490209Z Traceback (most recent call last): File ... in _write_dataframe success, nchunks, nrows, _ = write_pandas(con, df, self.datapoint_table_name)
DEFAULT 2024-11-11T11:44:41.490212Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.490216Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/pandas_tools.py", line 389, in write_pandas
DEFAULT 2024-11-11T11:44:41.490219Z cursor.execute(upload_sql, _is_internal=True)
DEFAULT 2024-11-11T11:44:41.490222Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/cursor.py", line 1059, in execute
DEFAULT 2024-11-11T11:44:41.490225Z data = sf_file_transfer_agent.result()
DEFAULT 2024-11-11T11:44:41.490228Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.490231Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/file_transfer_agent.py", line 740, in result
DEFAULT 2024-11-11T11:44:41.490240Z Error.errorhandler_wrapper(
DEFAULT 2024-11-11T11:44:41.490243Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/errors.py", line 284, in errorhandler_wrapper
DEFAULT 2024-11-11T11:44:41.490247Z handed_over = Error.hand_to_other_handler(
DEFAULT 2024-11-11T11:44:41.490250Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.490253Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/errors.py", line 339, in hand_to_other_handler
DEFAULT 2024-11-11T11:44:41.490256Z cursor.errorhandler(connection, cursor, error_class, error_value)
DEFAULT 2024-11-11T11:44:41.490259Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/errors.py", line 215, in default_errorhandler
DEFAULT 2024-11-11T11:44:41.490262Z raise error_class(
DEFAULT 2024-11-11T11:44:41.524363Z snowflake.connector.errors.OperationalError: 253003: While putting file(s) there was an error: 'AttributeError("'TypeError' object has no attribute 'errno'")', this might be caused by your access to the blob storage provider, or by Snowflake.

Some important dependencies:

python 3.11

asn1crypto==1.5.1
attrs==23.2.0
Babel==2.15.0
beautifulsoup4==4.12.2
certifi==2023.11.17
cryptography==41.0.7
filelock==3.13.1
flake8==6.1.0
httplib2==0.22.0
numpy==1.26.3
packaging==23.2
pandas==2.1.4
pyOpenSSL==23.3.0
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.11.4
snowflake-connector-python==3.12.3
snowflake-snowpark-python==1.11.1
snowflake-sqlalchemy==1.5.1
SQLAlchemy==1.4.51
sqlparse==0.4.4
tomlkit==0.12.3
urllib3==2.1.0
sfc-gh-dszmolka commented 1 week ago

hi @flekkk thank you, this response was very useful! looks like, at least in your case, something causes the public key verification to somehow get 5 arguments:

ERROR 2024-11-11T11:44:41.372115Z Traceback (most recent call last): 
File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1125, in validate_by_direct_connection self.process_ocsp_response(issuer, cert_id, ocsp_response) 
File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 310, in process_ocsp_response self.verify_signature( 
File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 372, in verify_signature public_key.verify( TypeError: _EllipticCurvePublicKey.verify() takes 4 positional arguments but 5 were given

despite only four should be passed:

            public_key.verify(
                signature, digest, padding.PKCS1v15(), utils.Prehashed(chosen_hash)
            )

and we don't have error handling for this situation as it should never happen.

Anyways; if you're up for a little more debugging and gathering more info, I created a patched version of ocsp_asn1crypto.py with some more logging and a dumb but hopefully effective error handling, attaching here: patched_ocsp_asn1crypto.py.txt

0.1 make sure you're running Snowflake PythonConnector 3.12.3, as the patch was made for this version. Should work with other 3.12.x too. I see you already have it. 0.2 remove insecure_mode or set it to False

  1. optionally, take a backup of /usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py
  2. replace the above file with the patched version
  3. either use this article to enable DEBUG logging in your original Python application

or you can just use this very simple one to connect to Snowflake and PUT a file (I see the error happened in the PUT), if using the test app please make sure to replace the account details, database, stage, etc:

import snowflake.connector

import logging
import os

for logger_name in ['snowflake','botocore']:
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.FileHandler('python_connector.log')
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)

conn = snowflake.connector.connect(
    user='REPLACE_USER',
    password='REPLACE_PASSWORD',
    account='REPLACE-ACCOUNT',
    database='REPLACE_DB',
    schema='REPLACE_SCHEMA'
    )

conn.cursor().execute("PUT file:///PATH/TO/test.txt @REPLACE_STAGE OVERWRITE=TRUE")

Main thing is, that either way, the DEBUG level logs need to be turned on.

  1. Reproduce the issue, i.e. run the test script above or run your original one which has DEBUG logs enabled. It will log tons of events into python_connector.log where it's executed.
  2. Now, I will need to see the logs somehow :) So either share it here (and then make sure to sanitize it, remove any sensitive data) or raise a Support Case with us and reference this GH Issue.
  3. You can revert DEBUG logs or even insecure_mode, but if you decide to keep the patched version of the module, then it hopefully the simple error handling will prevent the unhandled exception and allow your app to run, until we fix this properly.

Thanks for your help here !

opqpop commented 1 week ago

Hi, this seems to still happen for me even when upgraded to 3.12.3, started happening 11/8/24

any thoughts on what's wrong? it only happens for a very specific query that i do, only fetches 400 rows and I'm not sure why it happens

using insecure_mode=True solves the issue

def snowflake_conn(keep_alive=False, **kwargs):
    """
    Connect to Snowflake
        account: looks like "{}.us-central1.gcp"
    """

    keys = {
        "account": cfgutil.get_env("SNOWFLAKE_ACCOUNT"),
        "user": cfgutil.get_env("SNOWFLAKE_USER"),
        "password": cfgutil.get_env("SNOWFLAKE_PASSWORD"),
        "warehouse": cfgutil.get_env("SNOWFLAKE_WAREHOUSE", "COMPUTE_WH"),
        "database": cfgutil.get_env("SNOWFLAKE_DB_NAME", "EDW"),
    }

    # useful debug
    # print(keys)

    logger.info("Creating snowflake connection")

    conn = snowflake.connector.connect(
        account=keys["account"],
        user=keys["user"],
        password=keys["password"],
        warehouse=keys["warehouse"],
        database=keys["database"],
        client_session_keep_alive=keep_alive,
        # fixes weird 11/14/24 issue causing nftpulse fetches to not
        # work for 
        # https://github.com/snowflakedb/snowflake-connector-python/issues/2094
        insecure_mode=True,
        **kwargs,
    )
    return conn

    # Acquire a connection from the pool within each task
def run_query_with_pool(
    sql_query: str,
    sql_values: Optional[Union[Tuple, List[Dict]]] = None,
    convert_to_df=True,
) -> Any:
    if not POOL_INITIALIZED:
        init_snowflake_conn_pool()

    conn = connections.get()
    try:
        cursor = conn.cursor()
        cursor.execute(sql_query, sql_values)
        rows = cursor.fetchall()

        if convert_to_df:
            df = pd.DataFrame(
                rows,
                columns=[
                    # lowercase col names makes things easier
                    desc[0].lower()
                    for desc in cursor.description
                ],
            )
            return df
        else:
            return rows
    except Exception as err:
        breakpoint()
        print(1)
    finally:
        connections.put(conn)  # Return the connection to the pool
sfc-gh-dszmolka commented 1 week ago

@opqpop & folks, this looks to be some issue in GCP US CENTRAL, when verifying the certificate during Snowflake/GCP Bucket connection. So far, could not reproduce with an account in GCP US CENTRAL, so if any of you has the possibility to try with the method described in this comment in the environment where the issue reproduces for you, and shares the debug logs, that could potentially massively help. Thanks in advance !

knowhoper commented 1 day ago

Hi, @sfc-gh-dszmolka please see attached logs. You will see the error and debug information in there. Noting the logs were flooded with the line [DEBUG]: Verifying the attached certificate is signed by the issuer. Valid Not After: 2024-12-15 15:50:28+00:00, so cleaned that out.

Any help would be much appreciated. formatted.log

sfc-gh-dszmolka commented 23 hours ago

this is super helpful @knowhoper , thank you ! so from the logs it looks like one of the certificates (sha256 fingerprint: 9c:3f:2f:d1:1c:57:d7:c6:49:ad:5a:09:32:c0:f0:d2:97:56:f6:a0:a1:c7:4c:43:e1:e8:9a:62:d6:4c:d3:20 , name: WE2 issued by GTS Root R4) has a ECDSA type key.

This is unexpected. The Python Connector seems to be unprepared for this type of certificate, we expect them to be signed with a RSA key. During handling the certificate, it looks as if it is parsed into several parts I don't think this works well with verify_signature .

I'm not entirely sure how you end up with connecting to the regional GCS endpoint (gcpuscentral1-5fd82fl-stage.storage.googleapis.com) instead of the generic one like i'm seeing in my attempts (storage.googleapis.com). I'll keep experimenting to see if I can reproduce the issue and see the same regional GCS endpoint instead of the generic one, but the direction is quite straightforward I'm hoping.

While the bug is being investigated then later fixed, if you're up for a bit more experimenting, here's another patched version of ocsp_asn1crypto.py: patched_ocsp_asn1crypto_handle_exception.py.txt

It has no other change just to add a generic way of handling any kind of exception which might come from verify_signature. Just replace the original ocsp_asn1crypto.py with this patched version and I would be very curious to hear if it actually helps or not. It doesn't fix anything, but I'm hoping that at least by handling the exception better, we can get past the OCSP part. Of course other ,100% working approach is to just have insecure_mode=True which avoids the entire path which involves OCSP.

edit: also, because no matter how I try to connect to gcpuscentral1-5fd82fl-stage.storage.googleapis.com, I'm always getting a different cert signed by a different (WR2) intermediary CA , which has RSA key and PythonConnector can happily work with it. You for some reason are getting a cert which has the WE2 intermediary CA as a signer, which has ECDSA key - breaking the stuff here. You seem to also have OCSP_VALIDATION_CACHE size: 314, I have OCSP_VALIDATION_CACHE size: 300. This should be the same, since the OCSP Cache file is generated and downloaded from a central location.

Just in case this is caused by a bad / old cache, can you please try removing contents of /root/.cache/snowflake/ before starting up the application to test? This is prior to replacing ocsp_asn1crypto.py with the patched version; to see if it helps.

knowhoper commented 11 hours ago

Hi @sfc-gh-dszmolka, I've added the patched file in and don't receive any error with this. See attached logs. patched.log

sfc-gh-dszmolka commented 8 hours ago

great to hear that @knowhoper and thank you for testing it so quickly + sharing the logs ! We'll fix it in the driver so a patch would not be needed. I'll keep this thread updated with the progress.