search

snowflakedb / snowflake-connector-python

Snowflake Connector for Python
https://pypi.python.org/pypi/snowflake-connector-python/
Apache License 2.0
584 stars 468 forks source link

SNOW-694457: Connector is leaking HTTP(S)_PROXY environment variables #1329

Open ozturkberkay opened 1 year ago

ozturkberkay commented 1 year ago

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using?

    3.8.12

  2. What operating system and processor architecture are you using?

    macOS-10.16-x86_64-i386-64bit

  3. What are the component versions in the environment (pip freeze)?

    snowflake-connector-python==2.8.1

  4. What did you do?

    snowflake_config = {
    "account": "ACCOUNT",
    "user": "USER",
    "password": "PASSWORD",
    "role": "ROLE",
    "warehouse": "WAREHOUSE",
    "database": "DATABASE",
    "schema": "SCHEMA",
    "proxy_host": "PROXY",
    "proxy_port": "PROXY_PORT",
    "autocommit": True,
    "client_session_keep_alive": False,
}

print(os.environ.get("HTTP_PROXY"), os.environ.get("HTTPS_PROXY"))

conn = snowflake.connector.connect(**snowflake_config)
conn.close()

print(os.environ.get("HTTP_PROXY"), os.environ.get("HTTPS_PROXY"))
    None None
http://PROXY:PROXY_PORT http://PROXY:PROXY_PORT

  5. What did you expect to see?

    None None
None None

    Snowflake should not change the global environment variables. This has caused very hard to debug issues on our end. We have the same issue when using the connection in a context manager as well. The code responsible for this is here.

  6. Can you set logging to DEBUG and collect the logs?

    INFO:snowflake.connector.connection:Snowflake Connector for Python Version: 2.8.0, Python Version: 3.8.12, Platform: macOS-10.16-x86_64-i386-64bit
DEBUG:snowflake.connector.connection:connect
DEBUG:snowflake.connector.connection:__config
INFO:snowflake.connector.connection:This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
INFO:snowflake.connector.connection:Setting use_openssl_only mode to False
DEBUG:snowflake.connector.converter:use_numpy: False
DEBUG:snowflake.connector.connection:REST API object was created: ACCOUNT.REGION.snowflakecomputing.com:443
DEBUG:snowflake.connector.auth:authenticate
DEBUG:snowflake.connector.auth:assertion content: *********
DEBUG:snowflake.connector.auth:account=ACCOUNT, user=ACCOUNT, database=DATABASE, schema=SCHEMA, warehouse=COMPUTE_WH, role=PUBLIC, request_id=2de28877-ef87-4c1e-987d-9c078c6d833d
DEBUG:snowflake.connector.auth:body['data']: {'CLIENT_APP_ID': 'PythonConnector', 'CLIENT_APP_VERSION': '2.8.0', 'SVN_REVISION': None, 'ACCOUNT_NAME': 'ACCOUNT', 'LOGIN_NAME': 'USER', 'CLIENT_ENVIRONMENT': {'APPLICATION': 'PythonConnector', 'OS': 'Darwin', 'OS_VERSION': 'macOS-10.16-x86_64-i386-64bit', 'PYTHON_VERSION': '3.8.12', 'PYTHON_RUNTIME': 'CPython', 'PYTHON_COMPILER': 'Clang 10.0.0 ', 'OCSP_MODE': 'FAIL_OPEN', 'TRACING': 10, 'LOGIN_TIMEOUT': 120, 'NETWORK_TIMEOUT': None}, 'SESSION_PARAMETERS': {'AUTOCOMMIT': True, 'CLIENT_SESSION_KEEP_ALIVE': False, 'CLIENT_PREFETCH_THREADS': 4}}
DEBUG:snowflake.connector.auth:Timeout set to 120
DEBUG:snowflake.connector.vendored.urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG:snowflake.connector.vendored.urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG:snowflake.connector.network:Session status for SessionPool 'ACCOUNT.region.snowflakecomputing.com', SessionPool 1/1 active sessions
DEBUG:snowflake.connector.network:remaining request timeout: 120, retry cnt: 1
DEBUG:snowflake.connector.network:Request guid: 932e3a2d-3b0c-4cc7-986f-82c34d3c2640
DEBUG:snowflake.connector.network:socket timeout: 60
DEBUG:snowflake.connector.vendored.urllib3.connectionpool:Starting new HTTPS connection (1): ACCOUNT.REGION.snowflakecomputing.com:443
DEBUG:filelock:Attempting to acquire lock 140683005407344 on /Users/berkay/Library/Caches/Snowflake/ocsp_cache.lock
DEBUG:filelock:Lock 140683005407344 acquired on /Users/berkay/Library/Caches/Snowflake/ocsp_cache.lock
DEBUG:filelock:Attempting to acquire lock 140685157499760 on /Users/berkay/Library/Caches/Snowflake/ocsp_cache.lock
DEBUG:filelock:Timeout on acquiring lock 140685157499760 on /Users/berkay/Library/Caches/Snowflake/ocsp_cache.lock
DEBUG:snowflake.connector.cache:acquiring /Users/berkay/Library/Caches/Snowflake/ocsp_cache.lock timed out, skipping saving...
DEBUG:filelock:Attempting to release lock 140683005407344 on /Users/berkay/Library/Caches/Snowflake/ocsp_cache.lock
DEBUG:filelock:Lock 140683005407344 released on /Users/berkay/Library/Caches/Snowflake/ocsp_cache.lock
DEBUG:snowflake.connector.ocsp_snowflake:cache directory: /Users/berkay/Library/Caches/Snowflake
DEBUG:snowflake.connector.ssl_wrap_socket:OCSP Mode: FAIL_OPEN, OCSP response cache file name: None
DEBUG:snowflake.connector.ocsp_snowflake:ocsp_response_cache_uri: file:///Users/berkay/Library/Caches/Snowflake/ocsp_response_cache.json
DEBUG:snowflake.connector.ocsp_snowflake:OCSP_VALIDATION_CACHE size: 0
DEBUG:snowflake.connector.ocsp_snowflake:OCSP response cache server is enabled: http://ocsp.snowflakecomputing.com/ocsp_response_cache.json
DEBUG:snowflake.connector.ocsp_snowflake:OCSP dynamic cache server RETRY URL: None
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-09-15 19:41:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-09-15 19:45:34+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-22 07:00:00+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-01-15 00:00:00+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 16:47:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-09-15 19:41:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-09-15 19:41:06+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-11-18 15:50:09+00:00
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-07-15 15:48:31+00:00
DEBUG:snowflake.connector.ocsp_snowflake:Read OCSP response cache file: /Users/berkay/Library/Caches/Snowflake/ocsp_response_cache.json, count=196
DEBUG:snowflake.connector.ocsp_snowflake:validating certificate: ACCOUNT.REGION.snowflakecomputing.com
DEBUG:snowflake.connector.ocsp_asn1crypto:# of certificates: 4
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict([('common_name', '*.va3.us-east-1.snowflakecomputing.com')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')])
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')])
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]), issuer: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')])
DEBUG:snowflake.connector.ocsp_asn1crypto:subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Starfield Technologies, Inc.'), ('organizational_unit_name', 'Starfield Class 2 Certification Authority')])
DEBUG:snowflake.connector.ocsp_asn1crypto:reading certificate bundle: /Users/berkay/miniconda3/envs/snowflakebug/lib/python3.8/site-packages/certifi/cacert.pem
DEBUG:snowflake.connector.ocsp_asn1crypto:not found issuer_der: OrderedDict([('country_name', 'US'), ('organization_name', 'Starfield Technologies, Inc.'), ('organizational_unit_name', 'Starfield Class 2 Certification Authority')])
DEBUG:snowflake.connector.ocsp_snowflake:hit cache for subject: OrderedDict([('common_name', '*.va3.us-east-1.snowflakecomputing.com')])
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-09-15 19:41:06+00:00
DEBUG:snowflake.connector.ocsp_snowflake:hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')])
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-09-15 19:45:34+00:00
DEBUG:snowflake.connector.ocsp_snowflake:hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')])
DEBUG:snowflake.connector.ocsp_asn1crypto:Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-22 07:00:00+00:00
DEBUG:snowflake.connector.ocsp_snowflake:hit cache for subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')])
DEBUG:snowflake.connector.ocsp_snowflake:writing OCSP response cache file to /Users/berkay/Library/Caches/Snowflake/ocsp_response_cache.json
DEBUG:snowflake.connector.ocsp_snowflake:encoding OCSP response cache to JSON
DEBUG:snowflake.connector.ocsp_snowflake:ok
DEBUG:snowflake.connector.vendored.urllib3.connectionpool:https://ACCOUNT.us-east-1.snowflakecomputing.com:443 "POST /session/v1/login-request?request_id=ID&databaseName=DATABASE&schemaName=SCHEMA&warehouse=WAREHOUSE&roleName=ROLE&request_guid=GUID HTTP/1.1" 200 1488
DEBUG:snowflake.connector.network:SUCCESS
DEBUG:snowflake.connector.network:Session status for SessionPool 'ACCOUNT.region.snowflakecomputing.com', SessionPool 0/1 active sessions
DEBUG:snowflake.connector.network:ret[code] = None, after post request
DEBUG:snowflake.connector.auth:completed authentication
DEBUG:snowflake.connector.auth:token = ******
DEBUG:snowflake.connector.auth:master_token = ******
DEBUG:snowflake.connector.auth:id_token = NULL
DEBUG:snowflake.connector.auth:mfa_token = NULL
INFO:snowflake.connector.connection:closed
DEBUG:snowflake.connector.telemetry:Closing telemetry client.
DEBUG:snowflake.connector.telemetry:Sending 1 logs to telemetry. Data is {'logs': [{'message': {'driver_type': 'PythonConnector', 'driver_version': '2.8.0', 'source': 'PythonConnector', 'type': 'client_imported_packages', 'value': "{'pathlib', 'uuid', 'matplotlib', 'subprocess', 'pstats', 'certifi', 'zipimport', 'bcrypt', 'tzlocal', 'posix', 'packaging', 'ssl', 'quopri', 'opcode', 'asn1crypto', 'builtins', 'select', 'errno', 'tty', 'chardet', 'marshal', 'array', 'email', 'sitecustomize', 'calendar', 'linecache', 'operator', 'posixpath', 'distutils', 'cycler', 'stringprep', 'enum', 'six', 'ntpath', 'charset_normalizer', 'fractions', 'pydevd_file_utils', 'pkgutil', 'ast', 'Cryptodome', 'datetime', 'contextlib', 'cProfile', 'importlib', 'bz2', 'stat', 'sys', 'termios', 'pydevconsole', 'ctypes', 'platform', 'decimal', 'shutil', 'ipaddress', 'mpl_toolkits', 'colorsys', 'idna', 'typing_extensions', 'bisect', 'plistlib', 'pickleshare', 'pdb', 'cloudpickle', 'weakref', 'html', 'google', 'shlex', 'inspect', 'base64', 'sysconfig', 'cmd', 'traitlets', 'pytz', 'cmath', 'cgi', 'site', 'wcwidth', 'profile', 'jmespath', 'urllib3', 'ruamel', 'os', 'PIL', 'pyexpat', 'cffi', 'keyword', 'dataclasses', 'cython_runtime', 'xmlrpc', 'signal', 'IPython', 'secrets', 'oscrypto', 'OpenSSL', 'encodings', 'pycparser', 'json', 'string', 'codeop', 'pydev_ipython', 'csv', 'hashlib', 'io', 'heapq', 'lzma', 'binascii', 'codecs', 'typing', 'ptyprocess', 'sre_compile', 'reprlib', 'threading', 'argparse', 'jedi', 'webbrowser', 'time', 'pydoc', 'mimetypes', 'urllib', 'math', 'pyparsing', 'pydev_console', 'asyncio', 'traceback', 'atexit', 'tokenize', 'numbers', 'prompt_toolkit', 'fcntl', 'selectors', 'uu', 'pwd', 'warnings', 'kiwisolver', 'pickle', 'random', 'copyreg', 'getpass', 'copy', 'decorator', 'textwrap', 'pydoc_data', 'defusedxml', 'grp', 'resource', 'pandas', 'gzip', 'runpy', 'difflib', 'code', 'snowflake', 'botocore', 'numpy', 'socket', 'unicodedata', 'timeit', 'queue', 'zipfile', 'colorama', 'gc', 'mmap', 'socketserver', 're', 'types', 'struct', 'sre_constants', 'xml', 'abc', 'locale', 'parso', 'pyarrow', 'sqlite3', 'pprint', 'bdb', 'ipython_genutils', 'requests', 'pydevd_tracing', 'console_thrift', 'glob', 'filelock', 'pydevd_plugins', 'gettext', 'dis', 'pygments', 'jwt', 'dateutil', 'pty', 'pexpect', 'getopt', 'cryptography', 'http', 'tempfile', 'fnmatch', 'pydev_umd', 'contextvars', 'boto3', 'filecmp', 'sre_parse', 'hmac', 'backcall', 'pkg_resources', 'itertools', 'logging', 'functools', 'genericpath', 'token', 'collections', 'concurrent', 'zlib'}"}, 'timestamp': '1668497182241'}]}.
DEBUG:snowflake.connector.network:Session status for SessionPool 'ACCOUNT.region.snowflakecomputing.com', SessionPool 1/1 active sessions
DEBUG:snowflake.connector.network:remaining request timeout: 5, retry cnt: 1
DEBUG:snowflake.connector.network:Request guid: GUID
DEBUG:snowflake.connector.network:socket timeout: 60
DEBUG:snowflake.connector.vendored.urllib3.connectionpool:https://ACCOUNT.us-east-1.snowflakecomputing.com:443 "POST /telemetry/send?request_guid=GUID HTTP/1.1" 200 86
DEBUG:snowflake.connector.network:SUCCESS
DEBUG:snowflake.connector.network:Session status for SessionPool 'ACCOUNT.region.snowflakecomputing.com', SessionPool 0/1 active sessions
DEBUG:snowflake.connector.network:ret[code] = None, after post request
DEBUG:snowflake.connector.telemetry:Successfully uploading metrics to telemetry.
INFO:snowflake.connector.connection:No async queries seem to be running, deleting session
DEBUG:snowflake.connector.network:Session status for SessionPool 'ACCOUNT.region.snowflakecomputing.com', SessionPool 1/1 active sessions
DEBUG:snowflake.connector.network:remaining request timeout: 5, retry cnt: 1
DEBUG:snowflake.connector.network:Request guid: bba20f58-4492-465f-bb3f-1866088b651d
DEBUG:snowflake.connector.network:socket timeout: 60
DEBUG:snowflake.connector.vendored.urllib3.connectionpool:https://ACCOUNT.region.snowflakecomputing.com:443 "POST /session?delete=true&request_guid=GUID HTTP/1.1" 200 76
DEBUG:snowflake.connector.network:SUCCESS
DEBUG:snowflake.connector.network:Session status for SessionPool 'ACCOUNT.region.snowflakecomputing.com', SessionPool 0/1 active sessions
DEBUG:snowflake.connector.network:ret[code] = None, after post request
DEBUG:snowflake.connector.connection:Session is closed
http://PROXY:PROXY_PORT http://PROXY:PROXY_PORT
gtoonstra commented 7 months ago

Reviving this. I recently had to look into proxies and stumbled across this as well.

sfc-gh-dszmolka commented 6 months ago

thank you for raising this issue - and especially for the PR ! hopefully the team can take a look soon.