search

snowflakedb / snowflake-connector-python

Snowflake Connector for Python
https://pypi.python.org/pypi/snowflake-connector-python/
Apache License 2.0
596 stars 473 forks source link

SNOW-750040: Failed to import ArrowResult. No Apache Arrow result set format can be used. #1452

Closed NiallRees closed 1 year ago

NiallRees commented 1 year ago

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using?

Python 3.11.2 (v3.11.2:878ead1ac1, Feb 7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)]

  1. What operating system and processor architecture are you using?

macOS-13.2.1-arm64-arm-64bit

  1. What are the component versions in the environment (pip freeze)?
aiohttp==3.8.4
aiosignal==1.3.1
alembic==1.9.4
anyio==3.6.2
asn1crypto==1.5.1
asttokens==2.2.1
async-timeout==4.0.2
attrs==22.2.0
bcrypt==4.0.1
black==22.12.0
bugsnag==4.4.0
cachetools==5.3.0
certifi==2022.12.7
cffi==1.15.1
cfgv==3.3.1
charset-normalizer==2.1.1
click==8.1.3
commonmark==0.9.1
cryptography==39.0.1
devtools==0.9.0
distlib==0.3.6
ecdsa==0.18.0
executing==0.10.0
fastapi==0.89.1
filelock==3.9.0
frozenlist==1.3.3
google-api-core==2.11.0
google-auth==2.16.1
google-cloud-core==2.3.2
google-cloud-secret-manager==2.15.1
google-cloud-storage==2.7.0
google-crc32c==1.5.0
google-resumable-media==2.4.1
googleapis-common-protos==1.58.0
grpc-google-iam-v1==0.12.6
grpcio==1.51.3
grpcio-status==1.51.3
h11==0.14.0
identify==2.5.18
idna==3.4
Mako==1.2.4
MarkupSafe==2.1.2
more-itertools==9.0.0
multidict==6.0.4
mypy-extensions==1.0.0
nodeenv==1.7.0
oscrypto==1.3.0
packaging==23.0
passlib==1.7.4
pathspec==0.11.0
platformdirs==3.0.0
pluggy==0.13.1
pre-commit==2.21.0
proto-plus==1.22.2
protobuf==4.22.0
psycopg2==2.9.5
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pycryptodomex==3.17
pydantic==1.10.5
Pygments==2.14.0
PyJWT==2.6.0
pyOpenSSL==23.0.0
pytest==5.4.3
python-dotenv==0.20.0
python-jose==3.3.0
python-multipart==0.0.5
pytz==2022.7.1
PyYAML==6.0
requests==2.28.2
rich==12.6.0
rsa==4.9
six==1.16.0
slack-sdk==3.20.0
sniffio==1.3.0
snowflake-connector-python==3.0.0
SQLAlchemy==1.4.46
starlette==0.22.0
typing_extensions==4.5.0
urllib3==1.26.14
uvicorn==0.20.0
virtualenv==20.19.0
wcwidth==0.2.6
WebOb==1.8.7
yarl==1.8.2
  1. What did you do?
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import snowflake.connector
Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: dlopen(/Users/niallwoodward/Library/Caches/pypoetry/virtualenvs/backend-Sy87x2_N-py3.11/lib/python3.11/site-packages/snowflake/connector/arrow_iterator.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN5arrow12ArrayBuilder12AppendScalarERKNS_6ScalarEx'
  1. What did you expect to see?

No error

  1. Can you set logging to DEBUG and collect the logs?
>>> import snowflake.connector
2023-02-26 09:53:22,157 - MainThread ssl_wrap_socket.py:43 - inject_into_urllib3() - DEBUG - Injecting ssl_wrap_socket_with_ocsp
2023-02-26 09:53:22,157 - MainThread _auth.py:91 - <module>() - DEBUG - cache directory: /Users/niallwoodward/Library/Caches/Snowflake
2023-02-26 09:53:22,209 - MainThread cursor.py:81 - <module>() - DEBUG - Failed to import pyarrow. Cannot use pandas fetch API
2023-02-26 09:53:22,210 - MainThread cursor.py:89 - <module>() - WARNING - Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: dlopen(/Users/niallwoodward/Library/Caches/pypoetry/virtualenvs/backend-Sy87x2_N-py3.11/lib/python3.11/site-packages/snowflake/connector/arrow_iterator.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN5arrow12ArrayBuilder12AppendScalarERKNS_6ScalarEx'
sfc-gh-mkeller commented 1 year ago

@NiallRees would you be able to post your output of pip install --verbose 'snowflake-connector-python==3.0.0'? It looks like you have a custom compiled Python. If you could describe how you have compiled it that could help us debug this too. If I had to take a guess what's going wrong it's that pip is disregarding wheels (it probably builds everything from source) and that causes pyarrow to not compile (because pyarrow has a complicated build) and that causes our library to not compile our Arrow C-extension properly. Please try installing our currently supported pyarrow with pip install pyarrow==10.0.1 and if that works then our library should compile without any issues. If you get to install pyarrow manually then you might find pip's --no-build-isolation flag useful while installing snowflake-connector-python.

dmuraco commented 1 year ago

Another datapoint; I am having the same error with Python 3.10.8 (v3.10.8:aaaf517424, Oct 11 2022, 10:14:40) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin.

dmuraco commented 1 year ago

@NiallRees try to install the package like this (I took this from another users suggestion on an earlier reported bug). pip install git+https://github.com/snowflakedb/snowflake-connector-python@main

dmuraco commented 1 year ago

I was also able to get the package to install in poetry: poetry add git+https://github.com/snowflakedb/snowflake-connector-python@main

output -->

Updating dependencies Resolving dependencies... (0.3s)

Writing lock file

Package operations: 1 install, 1 update, 1 removal

• Removing setuptools (67.4.0) • Installing packaging (23.0) • Updating snowflake-connector-python (3.0.0 -> 3.0.1 7b00025)

NiallRees commented 1 year ago

Hopefully this helps @sfc-gh-mkeller:

➜  venvs python --version
Python 3.11.2
➜  venvs python -m venv venv
➜  venvs ls
venv
➜  venvs . ./venv/bin/activate
(venv) ➜  venvs pip install --verbose 'snowflake-connector-python==3.0.0'
Using pip 22.3.1 from /Users/niallwoodward/dev/venvs/venv/lib/python3.11/site-packages/pip (python 3.11)
Collecting snowflake-connector-python==3.0.0
  Downloading snowflake_connector_python-3.0.0-cp311-cp311-macosx_11_0_arm64.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 6.5 MB/s eta 0:00:00
Collecting asn1crypto<2.0.0,>0.24.0
  Using cached asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Collecting cffi<2.0.0,>=1.9
  Using cached cffi-1.15.1-cp311-cp311-macosx_11_0_arm64.whl (174 kB)
Collecting cryptography<41.0.0,>=3.1.0
  Downloading cryptography-39.0.1-cp36-abi3-macosx_10_12_universal2.whl (5.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 8.1 MB/s eta 0:00:00
Collecting oscrypto<2.0.0
  Using cached oscrypto-1.3.0-py2.py3-none-any.whl (194 kB)
Collecting pyOpenSSL<24.0.0,>=16.2.0
  Downloading pyOpenSSL-23.0.0-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.3/57.3 kB 4.3 MB/s eta 0:00:00
Collecting pycryptodomex!=3.5.0,<4.0.0,>=3.2
  Using cached pycryptodomex-3.17-cp35-abi3-macosx_10_9_universal2.whl (2.4 MB)
Collecting pyjwt<3.0.0
  Using cached PyJWT-2.6.0-py3-none-any.whl (20 kB)
Collecting pytz
  Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Collecting requests<3.0.0
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Requirement already satisfied: setuptools>34.0.0 in ./venv/lib/python3.11/site-packages (from snowflake-connector-python==3.0.0) (65.5.0)
Collecting charset-normalizer<3,>=2
  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting typing-extensions<5,>=4.3
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting filelock<4,>=3.5
  Using cached filelock-3.9.0-py3-none-any.whl (9.7 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pytz, asn1crypto, urllib3, typing-extensions, pyjwt, pycryptodomex, pycparser, oscrypto, idna, filelock, charset-normalizer, certifi, requests, cffi, cryptography, pyOpenSSL, snowflake-connector-python
  changing mode of /Users/niallwoodward/dev/venvs/venv/bin/normalizer to 755
  changing mode of /Users/niallwoodward/dev/venvs/venv/bin/snowflake-dump-certs to 755
  changing mode of /Users/niallwoodward/dev/venvs/venv/bin/snowflake-dump-ocsp-response to 755
  changing mode of /Users/niallwoodward/dev/venvs/venv/bin/snowflake-dump-ocsp-response-cache to 755
  changing mode of /Users/niallwoodward/dev/venvs/venv/bin/snowflake-export-certs to 755
Successfully installed asn1crypto-1.5.1 certifi-2022.12.7 cffi-1.15.1 charset-normalizer-2.1.1 cryptography-39.0.1 filelock-3.9.0 idna-3.4 oscrypto-1.3.0 pyOpenSSL-23.0.0 pycparser-2.21 pycryptodomex-3.17 pyjwt-2.6.0 pytz-2022.7.1 requests-2.28.2 snowflake-connector-python-3.0.0 typing-extensions-4.5.0 urllib3-1.26.14

[notice] A new release of pip available: 22.3.1 -> 23.0.1
[notice] To update, run: pip install --upgrade pip
(venv) ➜  venvs python
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import snowflake.connector
Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: dlopen(/Users/niallwoodward/dev/venvs/venv/lib/python3.11/site-packages/snowflake/connector/arrow_iterator.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN5arrow12ArrayBuilder12AppendScalarERKNS_6ScalarEx'
NiallRees commented 1 year ago

Installing from GitHub and therefore not using the snowflake-connector-python wheel removes this message for me too, but obviously not a solution:

➜  venvs rm -rdf venv
➜  venvs python -m venv venv
. ./venv/%
➜  venvs . ./venv/bin/activate
(venv) ➜  venvs pip install git+https://github.com/snowflakedb/snowflake-connector-python@main
Collecting git+https://github.com/snowflakedb/snowflake-connector-python@main
  Cloning https://github.com/snowflakedb/snowflake-connector-python (to revision main) to /private/var/folders/vw/56t0s0vd24d1c21c12gbylrm0000gn/T/pip-req-build-5100_txu
  Running command git clone --filter=blob:none --quiet https://github.com/snowflakedb/snowflake-connector-python /private/var/folders/vw/56t0s0vd24d1c21c12gbylrm0000gn/T/pip-req-build-5100_txu
  Resolved https://github.com/snowflakedb/snowflake-connector-python to commit be500a08735504971de59690d1c13eec9ec1f7d6
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting asn1crypto<2.0.0,>0.24.0
  Using cached asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Collecting cffi<2.0.0,>=1.9
  Using cached cffi-1.15.1-cp311-cp311-macosx_11_0_arm64.whl (174 kB)
Collecting cryptography<41.0.0,>=3.1.0
  Using cached cryptography-39.0.1-cp36-abi3-macosx_10_12_universal2.whl (5.4 MB)
Collecting oscrypto<2.0.0
  Using cached oscrypto-1.3.0-py2.py3-none-any.whl (194 kB)
Collecting pyOpenSSL<24.0.0,>=16.2.0
  Using cached pyOpenSSL-23.0.0-py3-none-any.whl (57 kB)
Collecting pycryptodomex!=3.5.0,<4.0.0,>=3.2
  Using cached pycryptodomex-3.17-cp35-abi3-macosx_10_9_universal2.whl (2.4 MB)
Collecting pyjwt<3.0.0
  Using cached PyJWT-2.6.0-py3-none-any.whl (20 kB)
Collecting pytz
  Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Collecting requests<3.0.0
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting charset-normalizer<3,>=2
  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting typing-extensions<5,>=4.3
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting filelock<4,>=3.5
  Using cached filelock-3.9.0-py3-none-any.whl (9.7 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: snowflake-connector-python
  Building wheel for snowflake-connector-python (pyproject.toml) ... done
  Created wheel for snowflake-connector-python: filename=snowflake_connector_python-3.0.1-cp311-cp311-macosx_11_0_universal2.whl size=13861510 sha256=a4f8c1ed2e1e5edff48ef68b13bd7f3bb4d3a75ba87bf3d4f47c09f06212d4db
  Stored in directory: /private/var/folders/vw/56t0s0vd24d1c21c12gbylrm0000gn/T/pip-ephem-wheel-cache-xbcffypd/wheels/35/00/9b/a40f5cf4b096e91e168317bf14f7fe4a4a83d6166df2e431b8
Successfully built snowflake-connector-python
Installing collected packages: pytz, asn1crypto, urllib3, typing-extensions, pyjwt, pycryptodomex, pycparser, packaging, oscrypto, idna, filelock, charset-normalizer, certifi, requests, cffi, cryptography, pyOpenSSL, snowflake-connector-python
Successfully installed asn1crypto-1.5.1 certifi-2022.12.7 cffi-1.15.1 charset-normalizer-2.1.1 cryptography-39.0.1 filelock-3.9.0 idna-3.4 oscrypto-1.3.0 packaging-23.0 pyOpenSSL-23.0.0 pycparser-2.21 pycryptodomex-3.17 pyjwt-2.6.0 pytz-2022.7.1 requests-2.28.2 snowflake-connector-python-3.0.1 typing-extensions-4.5.0 urllib3-1.26.14

[notice] A new release of pip available: 22.3.1 -> 23.0.1
[notice] To update, run: pip install --upgrade pip
(venv) ➜  venvs python
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import snowflake.connector
sfc-gh-mkeller commented 1 year ago

So our ARM wheels are built here on GitHub actions. The one thing that I've noticed in this job:

    ld: warning: ignoring file /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-x4rdfipo/overlay/lib/python3.9/site-packages/pyarrow/libarrow.1000.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    ld: warning: ignoring file /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-x4rdfipo/overlay/lib/python3.9/site-packages/pyarrow/libarrow_python.1000.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    ld: warning: ignoring file /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-x4rdfipo/overlay/lib/python3.9/site-packages/pyarrow/libparquet.1000.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    ld: warning: ignoring file /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-x4rdfipo/overlay/lib/python3.9/site-packages/pyarrow/libarrow_dataset.1000.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

Which led me to find: Downloading pyarrow-10.0.1-cp39-cp39-macosx_10_14_x86_64.whl (25.1 MB)

It appears that even though we are cross-compiling from x86_64 to arm64 cibuildwheel installs x86 build dependencies. I was not aware of this.

After some more digging https://github.com/snowflakedb/snowflake-connector-python/commit/bee5dff58ad88a6bdb58f9cf2f4f6dde8152b1bd had the same issue (which was before 3.0.0) , so this shouldn't be the cause of this issue.

mmaaartin commented 1 year ago

Seeing exactly the same issue with Python 3.9.12 MacOS 13.2.1 (22D68)

And below pip freeze

asn1crypto==1.5.1
boto3==1.26.81
botocore==1.29.81
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
cryptography==39.0.1
filelock==3.9.0
idna==3.4
Jinja2==3.1.2
jmespath==1.0.1
MarkupSafe==2.1.2
oscrypto==1.3.0
pycparser==2.21
pycryptodomex==3.17
PyJWT==2.6.0
pyOpenSSL==23.0.0
python-dateutil==2.8.2
pytz==2022.7.1
requests==2.28.2
s3transfer==0.6.0
six==1.16.0
snowflake-connector-python==3.0.0
snowflake-sqlalchemy==1.4.6
SQLAlchemy==1.4.46
typer==0.7.0
typing_extensions==4.5.0
urllib3==1.26.14
sfc-gh-aalam commented 1 year ago

looks like a similar issue was hit on x86 arch when packages are installed via conda and pypi. Although this is not our case, I think ABI flags is a lead.

https://github.com/snowflakedb/snowflake-connector-python/issues/1426#issuecomment-1421051347

sfc-gh-aalam commented 1 year ago

@NiallRees @mmaaartin @dmuraco we have updated .whl files on pypi. Can you check if things are working again for you

NiallRees commented 1 year ago

That's working for me!

dmuraco commented 1 year ago

Works for me as well.

sfc-gh-aalam commented 1 year ago

We might be blocked on this: https://github.com/github/roadmap/issues/528

CarolYing commented 1 year ago

Seeing the same error msg - Python 3.11.3. MacOS 13.3.1

sfc-gh-aalam commented 1 year ago

@CarolYing can you share the connector version you're using here

CarolYing commented 1 year ago

Yes, it is 3.2.0

sfc-gh-aalam commented 1 year ago

can you also shared how to installed it? can you also confirm the error message is similar to this

>>> import snowflake.connector
Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: dlopen(/Users/niallwoodward/dev/venvs/venv/lib/python3.11/site-packages/snowflake/connector/arrow_iterator.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN5arrow12ArrayBuilder12AppendScalarERKNS_6ScalarEx'
dataninjato commented 1 year ago

similar here with conda python 3.8 @sfc-gh-aalam snowflake-connector-python 3.2.0 py38h4ed8f06_1 snowflake-snowpark-python 1.9.0 py38haa95532_0

connector installed first with conda, later with pip each as v3.2.0 as per your instructions in this issue thread

Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: DLL load failed while importing arrow_iterator: The specified procedure could not be found.

same error with 3.3.1

solution: use the conda snowflake channel for a working env https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages#local-development-and-testing and for snowpark this as well: pip install pyarrow==10.0.1