search

snowflakedb / snowflake-connector-python

Snowflake Connector for Python
https://pypi.python.org/pypi/snowflake-connector-python/
Apache License 2.0
594 stars 472 forks source link

SNOW-197651: fetch_pandas_batches loads all lines and does not seem to adher to cursor.arraysize #414

Closed thondeboer closed 1 year ago

thondeboer commented 4 years ago

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using (python --version)? Python 3.7.6

  2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? Linux-4.19.102+-x86_64-with-Ubuntu-18.04-bionic

  3. What are the component versions in the environment (pip freeze)?

    Click to expand
    aiohttp==3.6.2 aiohttp-session==2.7.0 ansi2html==1.5.2 argcomplete==1.12.0 argh==0.26.2 argon2-cffi==20.1.0 asn1crypto==1.4.0 async-generator==1.10 async-timeout==3.0.1 asyncinit==0.2.4 attrs==19.3.0 azure-common==1.1.25 azure-core==1.8.1 azure-storage-blob==12.5.0 azure-storage-common==2.1.0 backcall==0.2.0 bash-kernel==0.7.2 bcbio-gff==0.6.6 biopython==1.77 bleach==3.1.5 blist==1.3.6 bokeh==1.2.0 boltons==19.3.0 boto3==1.11.17 botocore==1.14.17 bqplot==0.12.15 Brotli==1.0.7 cachetools==4.1.1 certifi==2020.6.20 cffi==1.13.2 chardet==3.0.4 click==7.1.2 cryptography==2.9.2 cycler==0.10.0 Cython==0.29.21 dash==1.14.0 dash-core-components==1.10.2 dash-html-components==1.0.3 dash-renderer==1.6.0 dash-table==4.9.0 dask==2.23.0 dataclasses==0.6 decorator==4.4.2 defusedxml==0.6.0 Deprecated==1.2.10 dill==0.3.2 dnspython==1.16.0 docutils==0.15.2 entrypoints==0.3 et-xmlfile==1.0.1 eventlet==0.26.1 fastcore==1.0.0 fastscript==1.0.0 fire==0.3.1 Flask==1.1.2 Flask-Compress==1.5.0 Flask-Login==0.5.0 fsspec==0.8.0 future==0.18.2 gcsfs==0.2.1 gevent==1.5.0 gffutils==0.10.1 google-api-core==1.22.1 google-auth==1.20.1 google-auth-oauthlib==0.4.1 google-cloud-core==1.4.1 google-cloud-storage==1.25.0 google-resumable-media==0.5.1 googleapis-common-protos==1.52.0 greenlet==0.4.16 gunicorn==19.9.0 hail==0.2.54 humanize==1.0.0 hurry.filesize==0.9 idna==2.8 ijson==2.6.1 importlib-metadata==1.7.0 iniconfig==1.0.1 intervaltree==3.1.0 ipydatawidgets==4.0.1 ipykernel==5.3.4 ipympl==0.5.7 ipython==7.17.0 ipython-genutils==0.2.0 ipyvolume==0.5.2 ipyvue==1.4.0 ipyvuetify==1.5.0 ipywebrtc==0.5.0 ipywidgets==7.5.1 isodate==0.6.0 itsdangerous==1.1.0 jdcal==1.4.1 jedi==0.17.2 Jinja2==2.11.2 jmespath==0.10.0 json5==0.9.5 jsonschema==3.2.0 jupyter-client==6.1.6 jupyter-core==4.6.3 jupyter-dash==0.3.1 jupyter-server==0.1.1 jupyterlab==2.2.4 jupyterlab-pygments==0.1.1 jupyterlab-server==1.2.0 kiwisolver==1.2.0 liftover==1.1.4 locket==0.2.0 lxml==4.5.2 marisa-trie==0.7.5 MarkupSafe==1.1.1 matplotlib==3.3.1 mgzip==0.2.1 mistune==0.8.4 monotonic==1.5 more-itertools==8.4.0 msrest==0.6.19 multidict==4.7.6 nbconvert==5.6.1 nbdev==1.0.10 nbformat==5.0.7 nest-asyncio==1.4.0 notebook==6.1.3 numpy==1.19.1 oauthlib==3.1.0 openpyxl==2.6.4 oscrypto==1.2.1 packaging==20.4 pandas==0.25.3 pandas-plink==2.0.5 pandocfilters==1.4.2 parsimonious==0.8.1 parso==0.7.1 partd==1.1.0 pexpect==4.8.0 PheWeb==1.1.19 pickleshare==0.7.5 Pillow==7.2.0 plotly==4.9.0 pluggy==0.13.1 prometheus-client==0.8.0 prompt-toolkit==3.0.6 protobuf==3.13.0 ptyprocess==0.6.0 py==1.9.0 py4j==0.10.7 pyarrow==0.17.1 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.20 pycrypto==2.6.1 pycryptodomex==3.9.8 pyfaidx==0.5.9.1 Pygments==2.6.1 PyJWT==1.7.1 pyliftover==0.4 pyOpenSSL==19.1.0 pyparsing==2.4.7 pyrsistent==0.16.0 pysam==0.15.4 pyspark==2.4.1 pytest==6.0.1 python-dateutil==2.8.1 python-json-logger==0.1.11 pythreejs==2.2.0 pytz==2020.1 PyYAML==5.3.1 pyzmq==19.0.2 rauth==0.7.3 requests==2.21.0 requests-oauthlib==1.3.0 retrying==1.3.3 rsa==4.6 s3transfer==0.3.3 scipy==1.3.3 seaborn==0.10.1 Send2Trash==1.5.0 simplejson==3.17.2 six==1.15.0 snowflake-connector-python==2.3.2 sortedcontainers==2.2.2 tabulate==0.8.3 termcolor==1.1.0 terminado==0.8.3 testpath==0.4.4 toml==0.10.1 toolz==0.10.0 tornado==6.0.4 tqdm==4.42.1 traitlets==4.3.3 traittypes==0.2.1 typing-extensions==3.7.4.2 urllib3==1.24.3 uWSGI==2.0.19.1 voila==0.1.21 voila-gridstack==0.0.9 voila-vuetify==0.4.0 wcwidth==0.2.5 webencodings==0.5.1 Werkzeug==1.0.1 wget==3.2 widgetsnbextension==3.5.1 wrapt==1.12.1 xarray==0.16.0 xlrd==1.2.0 yarl==1.5.1 zipp==3.1.0 zstandard==0.14.0

  4. What did you do? If possible, provide a recipe for reproducing the error. A complete runnable program is good.

    
import snowflake.connector
import pandas as pd
con = snowflake.connector.connect(
        user=SNOWFLAKE_USER,
        role=SNOWFLAKE_ROLE,
        password=SNOWSQL_PWD,
        account=SNOWFLAKE_ACCOUNT,
        warehouse=SNOWFLAKE_WAREHOUSE,
        database=DATABASE,
        schema=SCHEMA
        )
TABLE = '"GRE_GWAS"'
MAXLINES = 10
cur = con.cursor()
cur.arraysize = MAXLINES
q = f'SELECT * FROM {DATABASE}.{SCHEMA}.{TABLE} LIMIT 1000'
cur.execute(q)

for df in cur.fetch_pandas_batches(): logging.warning(f'Getting the GRE_GWAS data in batches of {MAXLINES}. Got {df.shape[0]} lines')

5. What did you expect to see?

18Sep2020 19:50:04-WARNING Getting the GRE_GWAS data in batches of 10. Got 10 lines 18Sep2020 19:50:04-WARNING Getting the GRE_GWAS data in batches of 10. Got 10 lines . . . 18Sep2020 19:50:04-WARNING Getting the GRE_GWAS data in batches of 10. Got 10 lines

6. What did you see instead?
18Sep2020 19:50:04-WARNING  Getting the GRE_GWAS data in batches of 10. Got 1000 lines

7. Can you set logging to DEBUG and collect the logs?

18Sep2020 19:58:57-INFO Snowflake Connector for Python Version: 2.3.2, Python Version: 3.7.6, Platform: Linux-4.19.102+-x86_64-with-debian-buster-sid 18Sep2020 19:58:57-DEBUG connect 18Sep2020 19:58:57-DEBUG __config 18Sep2020 19:58:57-INFO This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity. 18Sep2020 19:58:57-DEBUG use_numpy: False 18Sep2020 19:58:57-DEBUG REST API object was created: bs10304.eu-central-1.snowflakecomputing.com:443 18Sep2020 19:58:57-DEBUG authenticate 18Sep2020 19:58:57-DEBUG assertion content: ** 18Sep2020 19:58:57-DEBUG account=bs10304, user=deerfield_db1, database="DEERFIELD_DB1", schema="PUBLIC", warehouse=DEERFIELD_TEST, role=deerfield_db1, request_id=90482f53-c0b4-40ab-a784-114020d9177a 18Sep2020 19:58:57-DEBUG body['data']: {'CLIENT_APP_ID': 'PythonConnector', 'CLIENT_APP_VERSION': '2.3.2', 'SVN_REVISION': None, 'ACCOUNT_NAME': 'bs10304', 'LOGIN_NAME': 'deerfield_db1', 'CLIENT_ENVIRONMENT': {'APPLICATION': 'PythonConnector', 'OS': 'Linux', 'OS_VERSION': 'Linux-4.19.102+-x86_64-with-debian-buster-sid', 'PYTHON_VERSION': '3.7.6', 'PYTHON_RUNTIME': 'CPython', 'PYTHON_COMPILER': 'GCC 7.5.0', 'OCSP_MODE': 'FAIL_OPEN', 'TRACING': 10, 'LOGIN_TIMEOUT': 120, 'NETWORK_TIMEOUT': None, 'CLIENT_STORE_TEMPORARY_CREDENTIAL': None}, 'SESSION_PARAMETERS': {'CLIENT_SESSION_KEEP_ALIVE_HEARTBEAT_FREQUENCY': 900, 'CLIENT_PREFETCH_THREADS': 4}} 18Sep2020 19:58:57-DEBUG Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) 18Sep2020 19:58:57-DEBUG Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) 18Sep2020 19:58:57-DEBUG Active requests sessions: 1, idle: 0 18Sep2020 19:58:57-DEBUG remaining request timeout: 120, retry cnt: 1 18Sep2020 19:58:57-DEBUG socket timeout: 60 18Sep2020 19:58:57-DEBUG Starting new HTTPS connection (1): bs10304.eu-central-1.snowflakecomputing.com:443 18Sep2020 19:58:58-DEBUG OCSP Mode: FAIL_OPEN, OCSP response cache file name: None 18Sep2020 19:58:58-DEBUG ocsp_response_cache_uri: file:///data/.cache/snowflake/ocsp_response_cache.json 18Sep2020 19:58:58-DEBUG OCSP_VALIDATION_CACHE size: 140 18Sep2020 19:58:58-DEBUG OCSP response cache server is enabled: http://ocsp.snowflakecomputing.com/ocsp_response_cache.json 18Sep2020 19:58:58-DEBUG OCSP dynamic cache server RETRY URL: None 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:40:26+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:40:26+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:44:21+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:44:21+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2025-05-12 23:58:59+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 16:58:09+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 16:58:09+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:08:26+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:08:26+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 16:58:09+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 16:58:09+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:03:51+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:08:26+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:08:26+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-04-02 17:13:33+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 18Sep2020 19:58:58-DEBUG Read OCSP response cache file: /data/.cache/snowflake/ocsp_response_cache.json, count=140 18Sep2020 19:58:58-DEBUG validating certificate: bs10304.eu-central-1.snowflakecomputing.com 18Sep2020 19:58:58-DEBUG # of certificates: 4 18Sep2020 19:58:58-DEBUG subject: OrderedDict([('common_name', '.eu-central-1.snowflakecomputing.com')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]) 18Sep2020 19:58:58-DEBUG subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]) 18Sep2020 19:58:58-DEBUG subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]), issuer: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]) 18Sep2020 19:58:58-DEBUG subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]), issuer: OrderedDict([('country_name', 'US'), ('organization_name', 'Starfield Technologies, Inc.'), ('organizational_unit_name', 'Starfield Class 2 Certification Authority')]) 18Sep2020 19:58:58-DEBUG not found issuer_der: OrderedDict([('country_name', 'US'), ('organization_name', 'Starfield Technologies, Inc.'), ('organizational_unit_name', 'Starfield Class 2 Certification Authority')]) 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('common_name', '.eu-central-1.snowflakecomputing.com')]) 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:40:26+00:00 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]) 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:44:21+00:00 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]) 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]) 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('common_name', '.eu-central-1.snowflakecomputing.com')]) 18Sep2020 19:58:58-DEBUG using OCSP response cache 18Sep2020 19:58:58-DEBUG Certificate is NOT attached in Basic OCSP Response. Using issuer's certificate 18Sep2020 19:58:58-DEBUG Verifying the OCSP response is signed by the issuer. 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:40:26+00:00 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('organizational_unit_name', 'Server CA 1B'), ('common_name', 'Amazon')]) 18Sep2020 19:58:58-DEBUG using OCSP response cache 18Sep2020 19:58:58-DEBUG Certificate is attached in Basic OCSP Response 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer 18Sep2020 19:58:58-DEBUG Valid Not After: 2023-06-11 20:40:26+00:00 18Sep2020 19:58:58-DEBUG Verifying the OCSP response is signed by the issuer. 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2023-06-11 20:44:21+00:00 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('country_name', 'US'), ('organization_name', 'Amazon'), ('common_name', 'Amazon Root CA 1')]) 18Sep2020 19:58:58-DEBUG using OCSP response cache 18Sep2020 19:58:58-DEBUG Certificate is attached in Basic OCSP Response 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer 18Sep2020 19:58:58-DEBUG Valid Not After: 2023-06-11 20:44:21+00:00 18Sep2020 19:58:58-DEBUG Verifying the OCSP response is signed by the issuer. 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer. Valid Not After: 2021-12-14 07:00:00+00:00 18Sep2020 19:58:58-DEBUG hit cache for subject: OrderedDict([('country_name', 'US'), ('state_or_province_name', 'Arizona'), ('locality_name', 'Scottsdale'), ('organization_name', 'Starfield Technologies, Inc.'), ('common_name', 'Starfield Services Root Certificate Authority - G2')]) 18Sep2020 19:58:58-DEBUG using OCSP response cache 18Sep2020 19:58:58-DEBUG Certificate is attached in Basic OCSP Response 18Sep2020 19:58:58-DEBUG Verifying the attached certificate is signed by the issuer 18Sep2020 19:58:58-DEBUG Valid Not After: 2021-12-14 07:00:00+00:00 18Sep2020 19:58:58-DEBUG Verifying the OCSP response is signed by the issuer. 18Sep2020 19:58:58-DEBUG writing OCSP response cache file 18Sep2020 19:58:58-DEBUG encoding OCSP response cache to JSON 18Sep2020 19:58:58-DEBUG ok 18Sep2020 19:58:59-DEBUG https://bs10304.eu-central-1.snowflakecomputing.com:443 "POST /session/v1/login-request?request_id=90482f53-c0b4-40ab-a784-114020d9177a&databaseName=%22DEERFIELD_DB1%22&schemaName=%22PUBLIC%22&warehouse=DEERFIELD_TEST&roleName=deerfield_db1&request_guid=79719509-a693-4643-a52c-eb257bf458ac HTTP/1.1" 200 1373 18Sep2020 19:58:59-DEBUG SUCCESS 18Sep2020 19:58:59-DEBUG Active requests sessions: 0, idle: 1 18Sep2020 19:58:59-DEBUG ret[code] = None, after post request 18Sep2020 19:58:59-DEBUG completed authentication 18Sep2020 19:58:59-DEBUG token = 18Sep2020 19:58:59-DEBUG master_token = ** 18Sep2020 19:58:59-DEBUG id_token = NULL 18Sep2020 19:58:59-DEBUG no ID token is given when try to store temporary credential 18Sep2020 19:58:59-DEBUG cursor 18Sep2020 19:58:59-DEBUG executing SQL/command 18Sep2020 19:58:59-DEBUG binding: [SELECT FROM "DEERFIELD_DB1"."PUBLIC"."GRE_GWAS" LIMIT 1000] with input=[None], processed=[{}] 18Sep2020 19:58:59-INFO query: [SELECT FROM "DEERFIELD_DB1"."PUBLIC"."GRE_GWAS" LIMIT 1000] 18Sep2020 19:58:59-DEBUG sequence counter: 1 18Sep2020 19:58:59-DEBUG running query [SELECT FROM "DEERFIELD_DB1"."PUBLIC"."GRE_GWAS" LIMIT 1000] 18Sep2020 19:58:59-DEBUG is_file_transfer: False 18Sep2020 19:58:59-DEBUG _cmd_query 18Sep2020 19:58:59-DEBUG sql=[SELECT FROM "DEERFIELD_DB1"."PUBLIC"."GRE_GWAS" LIMIT 1000], sequence_id=[1], is_file_transfer=[None] 18Sep2020 19:58:59-DEBUG Active requests sessions: 1, idle: 0 18Sep2020 19:58:59-DEBUG remaining request timeout: None, retry cnt: 1 18Sep2020 19:58:59-DEBUG socket timeout: 60 18Sep2020 19:58:59-DEBUG https://bs10304.eu-central-1.snowflakecomputing.com:443 "POST /queries/v1/query-request?requestId=ed6ff317-f56e-48ad-8a6c-586ba40a80bd&request_guid=a0e6217b-7164-45bc-940d-1f70dc7f3abc HTTP/1.1" 200 None 18Sep2020 19:58:59-DEBUG SUCCESS 18Sep2020 19:58:59-DEBUG Active requests sessions: 0, idle: 1 18Sep2020 19:58:59-DEBUG ret[code] = None, after post request 18Sep2020 19:58:59-DEBUG sfqid: 0197048e-0091-7755-0000-3149006242ca 18Sep2020 19:58:59-INFO query execution done 18Sep2020 19:58:59-DEBUG SUCCESS 18Sep2020 19:58:59-DEBUG PUT OR GET: None 18Sep2020 19:58:59-DEBUG Query result format: arrow 18Sep2020 19:58:59-DEBUG Batches read: 1 18Sep2020 19:58:59-DEBUG Arrow BatchSize: 1 18Sep2020 19:58:59-DEBUG Init table iterator successfully, current chunk index: 0,chunk count: 0 18Sep2020 19:58:59-WARNING Getting the GRE_GWAS data in batches of 10. Got 1000 lines 18Sep2020 19:58:59-INFO fetching data into pandas dataframe done

thondeboer commented 4 years ago

Probably related to this is that cursor.fetchmany() only returns ONE line, instead of the expected cursor.arraysize.

It returns one line when used like an iterator:

for lines in cur.fetchmany():
    logging.warning(f'Getting the GRE_GWAS data in batches of {MAXLINES}. Got {len(lines)} lines')

len(lines) == [THE NUMBER OF COLUMNS] != [THE cursor.arraysize]

It DOES seem to work when used like this

cur.execute(q)
lines = cur.fetchmany()
len(lines) == cursor.arraysize
github-actions[bot] commented 1 year ago

To clean up and re-prioritize bugs and feature requests we are closing all issues older than 6 months as of March 1, 2023. If there are any issues or feature requests that you would like us to address, please re-create them. For urgent issues, opening a support case with this link Snowflake Community is the fastest way to get a response