search

snowflakedb / snowflake-connector-python

Snowflake Connector for Python
https://pypi.python.org/pypi/snowflake-connector-python/
Apache License 2.0
595 stars 473 forks source link

SNOW-142124: fetch_pandas_batches() returns nothing #286

Closed gfalcone closed 3 years ago

gfalcone commented 4 years ago

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using (python --version)? 3.7.6

  2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? Darwin-19.2.0-x86_64-i386-64bit

  3. What are the component versions in the environment (pip list)?

Package Version

asn1crypto 1.3.0
azure-common 1.1.24
azure-storage-blob 2.1.0
azure-storage-common 2.1.0
boto3 1.11.17
botocore 1.14.17
certifi 2019.11.28 cffi 1.13.2
chardet 3.0.4
cryptography 2.8
docutils 0.15.2
idna 2.8
ijson 2.6.1
jmespath 0.9.5
numpy 1.18.1
oscrypto 1.2.0
pandas 0.25.3
pip 20.0.2
pyarrow 0.15.1
pycparser 2.19
pycryptodomex 3.9.7
PyJWT 1.7.1
pyOpenSSL 19.1.0
python-dateutil 2.8.1
pytz 2019.3
requests 2.22.0
s3transfer 0.3.3
setuptools 45.2.0
six 1.14.0
snowflake-connector-python 2.2.1
urllib3 1.25.8
wheel 0.34.2

  1. What did you do?

    • Connect to Snowflake
    • Try fetch_pandas_batches()

  2. What did you expect to see? I expected to see some items printed on screen

  3. What did you see instead?

I saw nothing

  1. Can you set logging to DEBUG and collect the logs?

2020-03-02 13:58:54,796 - MainThread connection.py:178 - init() - INFO - Snowflake Connector for Python Version: 2.2.1, Python Version: 3.7.6, Platform: Darwin-19.2.0-x86_64-i386-64bit 2020-03-02 13:58:54,796 - MainThread connection.py:478 - connect() - DEBUG - connect 2020-03-02 13:58:54,797 - MainThread connection.py:714 - config() - DEBUG - config 2020-03-02 13:58:54,797 - MainThread connection.py:809 - config() - INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity. 2020-03-02 13:58:54,797 - MainThread converter.py:138 - init__() - DEBUG - use_numpy: False 2020-03-02 13:58:54,797 - MainThread connection.py:640 - __open_connection() - DEBUG - REST API object was created: hidden:443 2020-03-02 13:58:54,797 - MainThread auth.py:120 - authenticate() - DEBUG - authenticate 2020-03-02 13:58:54,797 - MainThread auth.py:144 - authenticate() - DEBUG - assertion content: ** 2020-03-02 13:58:54,797 - MainThread auth.py:156 - authenticate() - DEBUG - account=hidden, user=hidden, database=PRODUCTION, schema=None, warehouse=ANALYSTS, role=None, request_id=31fa637f-794a-486a-9847-e056b19c374e 2020-03-02 13:58:54,797 - MainThread auth.py:182 - authenticate() - DEBUG - body['data']: {'CLIENT_APP_ID': 'PythonConnector', 'CLIENT_APP_VERSION': '2.2.1', 'SVN_REVISION': None, 'ACCOUNT_NAME': 'hidden', 'LOGIN_NAME': 'hidden', 'CLIENT_ENVIRONMENT': {'APPLICATION': 'PythonConnector', 'OS': 'Darwin', 'OS_VERSION': 'Darwin-19.2.0-x86_64-i386-64bit', 'PYTHON_VERSION': '3.7.6', 'PYTHON_RUNTIME': 'CPython', 'PYTHON_COMPILER': 'Clang 11.0.0 (clang-1100.0.33.16)', 'OCSP_MODE': 'FAIL_OPEN'}, 'SESSION_PARAMETERS': {'CLIENT_SESSION_KEEP_ALIVE_HEARTBEAT_FREQUENCY': 900, 'CLIENT_PREFETCH_THREADS': 4}} 2020-03-02 13:58:54,797 - MainThread network.py:931 - _use_requests_session() - DEBUG - Active requests sessions: 1, idle: 0 2020-03-02 13:58:54,797 - MainThread network.py:636 - _request_exec_wrapper() - DEBUG - remaining request timeout: 120, retry cnt: 1 2020-03-02 13:58:54,798 - MainThread network.py:775 - _request_exec() - DEBUG - socket timeout: 60 2020-03-02 13:58:55,105 - MainThread ocsp_snowflake.py:382 - reset_cache_dir() - DEBUG - cache directory: /Users/paologenissel/Library/Caches/Snowflake 2020-03-02 13:58:55,211 - MainThread ssl_wrap_socket.py:406 - ssl_wrap_socket_with_ocsp() - DEBUG - OCSP Mode: FAIL_OPEN, OCSP response cache file name: None 2020-03-02 13:58:55,211 - MainThread ocsp_snowflake.py:421 - reset_ocsp_response_cache_uri() - DEBUG - ocsp_response_cache_uri: file:///Users/paologenissel/Library/Caches/Snowflake/ocsp_response_cache.json 2020-03-02 13:58:55,211 - MainThread ocsp_snowflake.py:423 - reset_ocsp_response_cache_uri() - DEBUG - OCSP_VALIDATION_CACHE size: 0 2020-03-02 13:58:55,211 - MainThread ocsp_snowflake.py:217 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP response cache server is enabled: http://ocsp.snowflakecomputing.com/ocsp_response_cache.json 2020-03-02 13:58:55,211 - MainThread ocsp_snowflake.py:251 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP dynamic cache server RETRY URL: None 2020-03-02 13:58:55,216 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:55,217 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:13:22+00:00 2020-03-02 13:58:55,218 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:16:05+00:00 2020-03-02 13:58:55,219 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 2020-03-02 13:58:55,221 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:55,225 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:55,227 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:55,231 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:46:27+00:00 2020-03-02 13:58:55,235 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:55,236 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:57:55+00:00 2020-03-02 13:58:55,237 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:57:55+00:00 2020-03-02 13:58:55,239 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:55,240 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:55,241 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:55,243 - MainThread ocsp_snowflake.py:459 - read_ocsp_response_cache_file() - DEBUG - Read OCSP response cache file: /Users/paologenissel/Library/Caches/Snowflake/ocsp_response_cache.json, count=77 2020-03-02 13:58:55,243 - MainThread ocsp_snowflake.py:971 - validate() - DEBUG - validating certificate: hidden.snowflakecomputing.com 2020-03-02 13:58:55,243 - MainThread ocsp_asn1crypto.py:347 - extract_certificate_chain() - DEBUG - # of certificates: 4 2020-03-02 13:58:55,244 - MainThread ocsp_asn1crypto.py:354 - extract_certificate_chain() - DEBUG - subject: OrderedDict([('common_name', '.us-east-1.snowflakecomputing.com')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]) 2020-03-02 13:58:55,244 - MainThread ocsp_asn1crypto.py:354 - extract_certificate_chain() - DEBUG - subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]) 2020-03-02 13:58:55,244 - MainThread ocsp_asn1crypto.py:354 - extract_certificate_chain() - DEBUG - subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]), issuer: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]) 2020-03-02 13:58:55,245 - MainThread ocsp_asn1crypto.py:354 - extract_certificate_chain() - DEBUG - subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Starfield Technologies, Inc.'), ('organizational_unit_name', 'Starfield Class 2 Certification Authority')]) 2020-03-02 13:58:55,246 - MainThread ocsp_asn1crypto.py:90 - read_cert_bundle() - DEBUG - reading certificate bundle: /Users/paologenissel/venv/snowflake/lib/python3.7/site-packages/certifi/cacert.pem 2020-03-02 13:58:55,259 - MainThread ocsp_asn1crypto.py:376 - create_pair_issuer_subject() - DEBUG - not found issuer_der: OrderedDict([('country_name', 'US'), ('organization_name', 'Starfield Technologies, Inc.'), ('organizational_unit_name', 'Starfield Class 2 Certification Authority')]) 2020-03-02 13:58:55,260 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('common_name', '.us-east-1.snowflakecomputing.com')]) 2020-03-02 13:58:55,261 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:13:22+00:00 2020-03-02 13:58:55,261 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]) 2020-03-02 13:58:55,263 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:16:05+00:00 2020-03-02 13:58:55,263 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]) 2020-03-02 13:58:55,265 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 2020-03-02 13:58:55,265 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]) 2020-03-02 13:58:55,266 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('common_name', '.us-east-1.snowflakecomputing.com')]) 2020-03-02 13:58:55,267 - MainThread ocsp_snowflake.py:1123 - validate_by_direct_connection() - DEBUG - using OCSP response cache 2020-03-02 13:58:55,268 - MainThread ocsp_asn1crypto.py:276 - process_ocsp_response() - DEBUG - Certificate is NOT attached in Basic OCSP Response. Using issuer's certificate 2020-03-02 13:58:55,268 - MainThread ocsp_asn1crypto.py:282 - process_ocsp_response() - DEBUG - Verifying the OCSP response is signed by the issuer. 2020-03-02 13:58:55,280 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:13:22+00:00 2020-03-02 13:58:55,280 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]) 2020-03-02 13:58:55,282 - MainThread ocsp_snowflake.py:1123 - validate_by_direct_connection() - DEBUG - using OCSP response cache 2020-03-02 13:58:55,283 - MainThread ocsp_asn1crypto.py:248 - process_ocsp_response() - DEBUG - Certificate is attached in Basic OCSP Response 2020-03-02 13:58:55,283 - MainThread ocsp_asn1crypto.py:250 - process_ocsp_response() - DEBUG - Verifying the attached certificate is signed by the issuer 2020-03-02 13:58:55,283 - MainThread ocsp_asn1crypto.py:254 - process_ocsp_response() - DEBUG - Valid Not After: 2022-10-24 20:13:22+00:00 2020-03-02 13:58:55,284 - MainThread ocsp_asn1crypto.py:282 - process_ocsp_response() - DEBUG - Verifying the OCSP response is signed by the issuer. 2020-03-02 13:58:55,288 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:16:05+00:00 2020-03-02 13:58:55,288 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]) 2020-03-02 13:58:55,289 - MainThread ocsp_snowflake.py:1123 - validate_by_direct_connection() - DEBUG - using OCSP response cache 2020-03-02 13:58:55,290 - MainThread ocsp_asn1crypto.py:248 - process_ocsp_response() - DEBUG - Certificate is attached in Basic OCSP Response 2020-03-02 13:58:55,291 - MainThread ocsp_asn1crypto.py:250 - process_ocsp_response() - DEBUG - Verifying the attached certificate is signed by the issuer 2020-03-02 13:58:55,291 - MainThread ocsp_asn1crypto.py:254 - process_ocsp_response() - DEBUG - Valid Not After: 2022-10-24 20:16:05+00:00 2020-03-02 13:58:55,292 - MainThread ocsp_asn1crypto.py:282 - process_ocsp_response() - DEBUG - Verifying the OCSP response is signed by the issuer. 2020-03-02 13:58:55,295 - MainThread ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 2020-03-02 13:58:55,295 - MainThread ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]) 2020-03-02 13:58:55,296 - MainThread ocsp_snowflake.py:1123 - validate_by_direct_connection() - DEBUG - using OCSP response cache 2020-03-02 13:58:55,297 - MainThread ocsp_asn1crypto.py:248 - process_ocsp_response() - DEBUG - Certificate is attached in Basic OCSP Response 2020-03-02 13:58:55,297 - MainThread ocsp_asn1crypto.py:250 - process_ocsp_response() - DEBUG - Verifying the attached certificate is signed by the issuer 2020-03-02 13:58:55,297 - MainThread ocsp_asn1crypto.py:254 - process_ocsp_response() - DEBUG - Valid Not After: 2021-12-14 07:00:00+00:00 2020-03-02 13:58:55,299 - MainThread ocsp_asn1crypto.py:282 - process_ocsp_response() - DEBUG - Verifying the OCSP response is signed by the issuer. 2020-03-02 13:58:55,301 - MainThread ocsp_snowflake.py:521 - write_ocsp_response_cache_file() - DEBUG - writing OCSP response cache file 2020-03-02 13:58:55,301 - MainThread ocsp_snowflake.py:1516 - encode_ocsp_response_cache() - DEBUG - encoding OCSP response cache to JSON 2020-03-02 13:58:55,311 - MainThread ocsp_snowflake.py:1014 - _validate() - DEBUG - ok 2020-03-02 13:58:55,630 - MainThread network.py:805 - _request_exec() - DEBUG - SUCCESS 2020-03-02 13:58:55,631 - MainThread network.py:944 - _use_requests_session() - DEBUG - Active requests sessions: 0, idle: 1 2020-03-02 13:58:55,631 - MainThread network.py:533 - _post_request() - DEBUG - ret[code] = None, after post request 2020-03-02 13:58:55,631 - MainThread auth.py:279 - authenticate() - DEBUG - completed authentication 2020-03-02 13:58:55,631 - MainThread auth.py:304 - authenticate() - DEBUG - token = 2020-03-02 13:58:55,631 - MainThread auth.py:308 - authenticate() - DEBUG - master_token = ** 2020-03-02 13:58:55,631 - MainThread auth.py:312 - authenticate() - DEBUG - id_token = NULL 2020-03-02 13:58:55,632 - MainThread connection.py:561 - cursor() - DEBUG - cursor 2020-03-02 13:58:55,632 - MainThread cursor.py:464 - execute() - DEBUG - executing SQL/command 2020-03-02 13:58:55,632 - MainThread cursor.py:487 - execute() - DEBUG - binding: [select from public.order_versions LIMIT 1000] with input=[None], processed=[{}] 2020-03-02 13:58:55,632 - MainThread cursor.py:519 - execute() - INFO - query: [select from public.order_versions LIMIT 1000] 2020-03-02 13:58:55,632 - MainThread connection.py:1075 - _next_sequence_counter() - DEBUG - sequence counter: 1 2020-03-02 13:58:55,632 - MainThread cursor.py:344 - _execute_helper() - DEBUG - running query [select from public.order_versions LIMIT 1000] 2020-03-02 13:58:55,632 - MainThread cursor.py:353 - _execute_helper() - DEBUG - is_file_transfer: False 2020-03-02 13:58:55,632 - MainThread connection.py:831 - cmd_query() - DEBUG - _cmd_query 2020-03-02 13:58:55,632 - MainThread connection.py:853 - cmd_query() - DEBUG - sql=[select from public.order_versions LIMIT 1000], sequence_id=[1], is_file_transfer=[None] 2020-03-02 13:58:55,633 - MainThread network.py:931 - _use_requests_session() - DEBUG - Active requests sessions: 1, idle: 0 2020-03-02 13:58:55,633 - MainThread network.py:636 - _request_exec_wrapper() - DEBUG - remaining request timeout: None, retry cnt: 1 2020-03-02 13:58:55,633 - MainThread network.py:775 - _request_exec() - DEBUG - socket timeout: 60 2020-03-02 13:58:55,930 - MainThread network.py:805 - _request_exec() - DEBUG - SUCCESS 2020-03-02 13:58:55,930 - MainThread network.py:944 - _use_requests_session() - DEBUG - Active requests sessions: 0, idle: 1 2020-03-02 13:58:55,931 - MainThread network.py:533 - _post_request() - DEBUG - ret[code] = None, after post request 2020-03-02 13:58:55,931 - MainThread cursor.py:543 - execute() - DEBUG - sfqid: 01929dea-0268-4bbc-0000-1af90488bede 2020-03-02 13:58:55,931 - MainThread cursor.py:545 - execute() - INFO - query execution done 2020-03-02 13:58:55,931 - MainThread cursor.py:547 - execute() - DEBUG - SUCCESS 2020-03-02 13:58:55,931 - MainThread cursor.py:551 - execute() - DEBUG - PUT OR GET: None 2020-03-02 13:58:55,931 - MainThread cursor.py:611 - _init_result_and_meta() - DEBUG - Query result format: arrow 2020-03-02 13:58:55,931 - MainThread cursor.py:631 - _init_result_and_meta() - DEBUG - Data from first gs response is empty 2020-03-02 13:58:55,932 - MainThread cursor.py:631 - _init_result_and_meta() - DEBUG - chunk size=1 2020-03-02 13:58:55,932 - MainThread cursor.py:631 - _init_result_and_meta() - DEBUG - added chunk header: key=x-amz-server-side-encryption-customer-key, value=9XXcx5Osh4XQK67y5QySK+Y2BFclz7A0a7SzSz+21to= 2020-03-02 13:58:55,932 - MainThread cursor.py:631 - _init_result_and_meta() - DEBUG - added chunk header: key=x-amz-server-side-encryption-customer-key-md5, value=ZVvaaj2wSsGUyxyjZXr7VA== 2020-03-02 13:58:55,932 - MainThread cursor.py:631 - _init_result_and_meta() - DEBUG - qrmk=9XXcx5Osh4XQK67y5QySK+Y2BFclz7A0a7SzSz+21to= 2020-03-02 13:58:55,932 - MainThread chunk_downloader.py:75 - _pre_init() - DEBUG - queued chunk 0: rowCount=1000 2020-03-02 13:58:55,932 - MainThread chunk_downloader.py:87 - _pre_init() - DEBUG - prefetch threads: 4, number of chunks: 1, effective threads: 1 2020-03-02 13:58:55,934 - MainThread chunk_downloader.py:105 - init() - DEBUG - Chunk Downloader in memory 2020-03-02 13:58:55,934 - MainThread cursor.py:742 - fetch_pandas_all() - DEBUG - Init table iterator successfully, current chunk index: 0, chunk count: 1 2020-03-02 13:58:55,934 - MainThread cursor.py:742 - fetch_pandas_all() - DEBUG - chunk index: 0, chunk_count: 1 2020-03-02 13:58:55,935 - MainThread chunk_downloader.py:162 - next_chunk() - DEBUG - next_chunk_to_consume=1, next_chunk_to_download=2, total_chunks=1 2020-03-02 13:58:55,935 - MainThread chunk_downloader.py:183 - next_chunk() - DEBUG - waiting for chunk 1/1 in 1/10 download attempt 2020-03-02 13:58:55,935 - MainThread chunk_downloader.py:197 - next_chunk() - DEBUG - chunk 1/1 is NOT ready to consume in 10/3600(s) 2020-03-02 13:58:55,935 - Thread-1 chunk_downloader.py:114 - _download_chunk() - DEBUG - downloading chunk 1/1 2020-03-02 13:58:55,935 - Thread-1 chunk_downloader.py:119 - _download_chunk() - DEBUG - use chunk headers from result 2020-03-02 13:58:55,935 - Thread-1 chunk_downloader.py:125 - _download_chunk() - DEBUG - started getting the result set 1: https://sfc-va-ds1-customer-stage.s3.amazonaws.com/0dnk-s-vass1701/results/01929de4-029a-bbaa-0000-1af90488d3a6_0/main/data_0_0_0?x-amz-server-side-encryption-customer-algorithm=AES256&response-content-encoding=gzip&AWSAccessKeyId=AKIAJW226MB3QZ4QY44A&Expires=1583240335&Signature=fSBnyf6RluYnGmJVVgoMlDM6SsM%3D 2020-03-02 13:58:55,935 - Thread-1 network.py:931 - _use_requests_session() - DEBUG - Active requests sessions: 1, idle: 0 2020-03-02 13:58:55,935 - Thread-1 network.py:636 - _request_exec_wrapper() - DEBUG - remaining request timeout: 3600, retry cnt: 1 2020-03-02 13:58:55,935 - Thread-1 network.py:775 - _request_exec() - DEBUG - socket timeout: 60 2020-03-02 13:58:56,203 - Thread-1 ssl_wrap_socket.py:406 - ssl_wrap_socket_with_ocsp() - DEBUG - OCSP Mode: FAIL_OPEN, OCSP response cache file name: None 2020-03-02 13:58:56,203 - Thread-1 ocsp_snowflake.py:421 - reset_ocsp_response_cache_uri() - DEBUG - ocsp_response_cache_uri: file:///Users/paologenissel/Library/Caches/Snowflake/ocsp_response_cache.json 2020-03-02 13:58:56,204 - Thread-1 ocsp_snowflake.py:423 - reset_ocsp_response_cache_uri() - DEBUG - OCSP_VALIDATION_CACHE size: 77 2020-03-02 13:58:56,204 - Thread-1 ocsp_snowflake.py:217 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP response cache server is enabled: http://ocsp.snowflakecomputing.com/ocsp_response_cache.json 2020-03-02 13:58:56,204 - Thread-1 ocsp_snowflake.py:251 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP dynamic cache server RETRY URL: None 2020-03-02 13:58:56,215 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,216 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,218 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:13:22+00:00 2020-03-02 13:58:56,219 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:13:22+00:00 2020-03-02 13:58:56,220 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:16:05+00:00 2020-03-02 13:58:56,222 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2022-10-24 20:16:05+00:00 2020-03-02 13:58:56,223 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 2020-03-02 13:58:56,225 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 2020-03-02 13:58:56,228 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,229 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,234 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,235 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,236 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,237 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,243 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:46:27+00:00 2020-03-02 13:58:56,244 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:46:27+00:00 2020-03-02 13:58:56,250 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,251 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,252 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:57:55+00:00 2020-03-02 13:58:56,253 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:57:55+00:00 2020-03-02 13:58:56,254 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:57:55+00:00 2020-03-02 13:58:56,255 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 18:57:55+00:00 2020-03-02 13:58:56,257 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,258 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,260 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,262 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,263 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,264 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-01-13 19:01:13+00:00 2020-03-02 13:58:56,267 - Thread-1 ocsp_snowflake.py:459 - read_ocsp_response_cache_file() - DEBUG - Read OCSP response cache file: /Users/paologenissel/Library/Caches/Snowflake/ocsp_response_cache.json, count=77 2020-03-02 13:58:56,267 - Thread-1 ocsp_snowflake.py:971 - validate() - DEBUG - validating certificate: sfc-va-ds1-customer-stage.s3.amazonaws.com 2020-03-02 13:58:56,267 - Thread-1 ocsp_asn1crypto.py:347 - extract_certificate_chain() - DEBUG - # of certificates: 2 2020-03-02 13:58:56,268 - Thread-1 ocsp_asn1crypto.py:354 - extract_certificate_chain() - DEBUG - subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Washington'), ('locality_name', 'Seattle'), ('organization_name', 'Amazon.com, Inc.'), ('common_name', '.s3.amazonaws.com')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'DigiCert Inc'), ('organizational_unit_name', 'www.digicert.com'), ('common_name', 'DigiCert Baltimore CA-2 G2')]) 2020-03-02 13:58:56,268 - Thread-1 ocsp_asn1crypto.py:354 - extract_certificate_chain() - DEBUG - subject: OrderedDict([('country_name', 'US'), ('organization_name', 'DigiCert Inc'), ('organizational_unit_name', 'www.digicert.com'), ('common_name', 'DigiCert Baltimore CA-2 G2')]), issuer: OrderedDict([('country_name', 'IE'), ('organization_name', 'Baltimore'), ('organizational_unit_name', 'CyberTrust'), ('common_name', 'Baltimore CyberTrust Root')]) 2020-03-02 13:58:56,269 - Thread-1 ocsp_asn1crypto.py:376 - create_pair_issuer_subject() - DEBUG - not found issuer_der: OrderedDict([('country_name', 'IE'), ('organization_name', 'Baltimore'), ('organizational_unit_name', 'CyberTrust'), ('common_name', 'Baltimore CyberTrust Root')]) 2020-03-02 13:58:56,270 - Thread-1 ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Washington'), ('locality_name', 'Seattle'), ('organization_name', 'Amazon.com, Inc.'), ('common_name', '.s3.amazonaws.com')]) 2020-03-02 13:58:56,272 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,272 - Thread-1 ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'DigiCert Inc'), ('organizational_unit_name', 'www.digicert.com'), ('common_name', 'DigiCert Baltimore CA-2 G2')]) 2020-03-02 13:58:56,272 - Thread-1 ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Washington'), ('locality_name', 'Seattle'), ('organization_name', 'Amazon.com, Inc.'), ('common_name', '*.s3.amazonaws.com')]) 2020-03-02 13:58:56,274 - Thread-1 ocsp_snowflake.py:1123 - validate_by_direct_connection() - DEBUG - using OCSP response cache 2020-03-02 13:58:56,274 - Thread-1 ocsp_asn1crypto.py:276 - process_ocsp_response() - DEBUG - Certificate is NOT attached in Basic OCSP Response. Using issuer's certificate 2020-03-02 13:58:56,274 - Thread-1 ocsp_asn1crypto.py:282 - process_ocsp_response() - DEBUG - Verifying the OCSP response is signed by the issuer. 2020-03-02 13:58:56,277 - Thread-1 ocsp_asn1crypto.py:194 - is_valid_time() - DEBUG - Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,277 - Thread-1 ocsp_snowflake.py:587 - find_cache() - DEBUG - hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'DigiCert Inc'), ('organizational_unit_name', 'www.digicert.com'), ('common_name', 'DigiCert Baltimore CA-2 G2')]) 2020-03-02 13:58:56,278 - Thread-1 ocsp_snowflake.py:1123 - validate_by_direct_connection() - DEBUG - using OCSP response cache 2020-03-02 13:58:56,279 - Thread-1 ocsp_asn1crypto.py:248 - process_ocsp_response() - DEBUG - Certificate is attached in Basic OCSP Response 2020-03-02 13:58:56,279 - Thread-1 ocsp_asn1crypto.py:250 - process_ocsp_response() - DEBUG - Verifying the attached certificate is signed by the issuer 2020-03-02 13:58:56,279 - Thread-1 ocsp_asn1crypto.py:254 - process_ocsp_response() - DEBUG - Valid Not After: 2025-05-12 23:58:59+00:00 2020-03-02 13:58:56,280 - Thread-1 ocsp_asn1crypto.py:282 - process_ocsp_response() - DEBUG - Verifying the OCSP response is signed by the issuer. 2020-03-02 13:58:56,282 - Thread-1 ocsp_snowflake.py:521 - write_ocsp_response_cache_file() - DEBUG - writing OCSP response cache file 2020-03-02 13:58:56,282 - Thread-1 ocsp_snowflake.py:1516 - encode_ocsp_response_cache() - DEBUG - encoding OCSP response cache to JSON 2020-03-02 13:58:56,291 - Thread-1 ocsp_snowflake.py:1014 - _validate() - DEBUG - ok 2020-03-02 13:58:56,394 - Thread-1 network.py:805 - _request_exec() - DEBUG - SUCCESS 2020-03-02 13:58:56,559 - Thread-1 chunk_downloader.py:331 - to_iterator() - DEBUG - Batches read: 1 2020-03-02 13:58:56,559 - Thread-1 network.py:944 - _use_requests_session() - DEBUG - Active requests sessions: 0, idle: 1 2020-03-02 13:58:56,559 - Thread-1 chunk_downloader.py:128 - _download_chunk() - DEBUG - finished getting the result set 1: https://sfc-va-ds1-customer-stage.s3.amazonaws.com/0dnk-s-vass1701/results/01929de4-029a-bbaa-0000-1af90488d3a6_0/main/data_0_0_0?x-amz-server-side-encryption-customer-algorithm=AES256&response-content-encoding=gzip&AWSAccessKeyId=AKIAJW226MB3QZ4QY44A&Expires=1583240335&Signature=fSBnyf6RluYnGmJVVgoMlDM6SsM%3D 2020-03-02 13:58:56,560 - Thread-1 chunk_downloader.py:145 - _download_chunk() - DEBUG - added chunk 1/1 to a chunk list. 2020-03-02 13:58:56,560 - MainThread chunk_downloader.py:224 - next_chunk() - DEBUG - chunk 1/1 is ready to consume 2020-03-02 13:58:56,560 - MainThread cursor.py:742 - fetch_pandas_all() - DEBUG - Arrow BatchSize: 1 2020-03-02 13:58:56,575 - MainThread cursor.py:742 - fetch_pandas_all() - INFO - fetching data into pandas dataframe done 2020-03-02 13:58:56,583 - MainThread cursor.py:731 - fetch_pandas_batches() - DEBUG - Init table iterator successfully, current chunk index: 2, chunk count: 0 2020-03-02 13:58:56,583 - MainThread cursor.py:731 - fetch_pandas_batches() - INFO - fetching data into pandas dataframe done 2020-03-02 13:58:56,628 - MainThread connection.py:501 - close() - INFO - closed 2020-03-02 13:58:56,628 - MainThread telemetry.py:113 - close() - DEBUG - Closing telemetry client. 2020-03-02 13:58:56,628 - MainThread network.py:931 - _use_requests_session() - DEBUG - Active requests sessions: 1, idle: 0 2020-03-02 13:58:56,628 - MainThread network.py:636 - _request_exec_wrapper() - DEBUG - remaining request timeout: 5, retry cnt: 1 2020-03-02 13:58:56,628 - MainThread network.py:775 - _request_exec() - DEBUG - socket timeout: 60 2020-03-02 13:58:56,629 - MainThread network.py:891 - _request_exec() - DEBUG - Hit retryable client error. Retrying... Ignore the following error stack: could not find io module state (interpreter shutdown?) Traceback (most recent call last): File "/Users/paologenissel/venv/snowflake/lib/python3.7/site-packages/snowflake/connector/network.py", line 799, in _request_exec auth=SnowflakeAuth(token), File "/Users/paologenissel/venv/snowflake/lib/python3.7/site-packages/requests/sessions.py", line 519, in request prep = self.prepare_request(req) File "/Users/paologenissel/venv/snowflake/lib/python3.7/site-packages/requests/sessions.py", line 462, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File "/Users/paologenissel/venv/snowflake/lib/python3.7/site-packages/requests/models.py", line 316, in prepare self.prepare_body(data, files, json) File "/Users/paologenissel/venv/snowflake/lib/python3.7/site-packages/requests/models.py", line 476, in prepare_body length = super_len(data) File "/Users/paologenissel/venv/snowflake/lib/python3.7/site-packages/requests/utils.py", line 119, in super_len fileno = o.fileno() RuntimeError: could not find io module state (interpreter shutdown?) 2020-03-02 13:58:56,634 - MainThread network.py:944 - _use_requests_session() - DEBUG - Active requests sessions: 0, idle: 1 2020-03-02 13:58:56,635 - MainThread network.py:533 - _post_request() - DEBUG - ret[code] = N/A, after post request 2020-03-02 13:58:56,639 - MainThread connection.py:507 - close() - DEBUG - Session is closed

Hello,

I tried using the python connector and use the fetch_pandas_batches() function, and got nothing (it should have printed my dataset on screen). I tried the same program but with fetch_pandas_all and it worked like a charm. Here is my code : 

import snowflake.connector

import logging
import os

for logger_name in ['snowflake.sqlalchemy', 'snowflake.connector', 'botocore']: 
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)

host = 'hidden'
user = 'hidden'
password = 'hidden'
account = 'hidden'
warehouse = 'ANALYSTS'
database = 'PRODUCTION'

ctx = snowflake.connector.connect(
          host=host,
          user=user,
          password=password,
          account=account,
          warehouse=warehouse,
          database=database,
          protocol='https')

# Create a cursor object.
cur = ctx.cursor()

# Execute a statement that will generate a result set.
sql = "select * from public.order_versions LIMIT 1000"
cur.execute(sql)

# Fetch the result set from the cursor and deliver it as the Pandas DataFrame.
df = cur.fetch_pandas_all()

for df in cur.fetch_pandas_batches():
    print(df)

Thank you in advance for your help !

chetansurwade commented 4 years ago

You should probably use this https://github.com/snowflakedb/snowflake-sqlalchemy It provides sqlalchemy dialect for snowflake and you could just use df = pandas.read_sql_query(query, engine) to load the data into pandas dataframe. Hope this helps :-D

xhochy commented 4 years ago

You should probably use this https://github.com/snowflakedb/snowflake-sqlalchemy It provides sqlalchemy dialect for snowflake and you could just use df = pandas.read_sql_query(query, engine) to load the data into pandas dataframe. Hope this helps :-D

Note that using pandas.read_sql_query(query, engine) is at least a magnitude slower than using fetch_pandas_all.

chetansurwade commented 4 years ago

You should probably use this https://github.com/snowflakedb/snowflake-sqlalchemy It provides sqlalchemy dialect for snowflake and you could just use df = pandas.read_sql_query(query, engine) to load the data into pandas dataframe. Hope this helps :-D

Note that using pandas.read_sql_query(query, engine) is at least a magnitude slower than using fetch_pandas_all.

Haven't profiled the comparison yet. But if you are saying so, it must be faster. Any benchmarks/profiler results to share?

xhochy commented 4 years ago

Have a look at figure 3 at https://www.snowflake.com/blog/fetching-query-results-from-snowflake-just-got-a-lot-faster-with-apache-arrow/

I'm seeing roughly similar results with real data.

chetansurwade commented 4 years ago

Have a look at figure 3 at https://www.snowflake.com/blog/fetching-query-results-from-snowflake-just-got-a-lot-faster-with-apache-arrow/

I'm seeing roughly similar results with real data.

Thanks, that was informational.

gfalcone commented 4 years ago

This is precisely why I opened this issue. This function was marketed by Snowflake and it was mentioned that it would be faster than the current functions available.

sfc-gh-hkapre commented 3 years ago

Is this still an issue? fetch_pandas_batches() should work and should be more performant.

sfc-gh-hkapre commented 3 years ago

Closing this issue.