search

conda-forge / snowflake-connector-python-feedstock

A conda-smithy repository for snowflake-connector-python.
BSD 3-Clause "New" or "Revised" License
5 stars 14 forks source link

snowflake-connector-python v3.0.0 #125

Closed regro-cf-autotick-bot closed 1 year ago

regro-cf-autotick-bot commented 1 year ago

It is very likely that the current package version for this feedstock is out of date.

Checklist before merging this PR:

Information about this PR:

  1. Feel free to push to the bot's branch to update this PR if needed.
  2. The bot will almost always only open one PR per version.
  3. The bot will stop issuing PRs if more than 3 version bump PRs generated by the bot are open. If you don't want to package a particular version please close the PR.
  4. If you want these PRs to be merged automatically, make an issue with code>@conda-forge-admin,</codeplease add bot automerge in the title and merge the resulting PR. This command will add our bot automerge feature to your feedstock.
  5. If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase code>@<space/conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

Closes: #124

Dependency Analysis

We couldn't run dependency analysis due to an internal error in the bot. :/ Help is very welcome!

This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/autotick-bot/actions/runs/4029676573, please use this URL for debugging.

conda-forge-webservices[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

hajapy commented 1 year ago

https://github.com/snowflakedb/snowflake-connector-python/commit/91ec8386d0bc7240b6d62ec31e18dd6b3020b259 seems to switch to using c++17

hajapy commented 1 year ago

Here's the error I saw from build logs before it gets deleted that prompted the revert of std::string_view 


  creating build/temp.linux-x86_64-cpython-38/src/snowflake/connector/cpp/ArrowIterator/Util
  creating build/temp.linux-x86_64-cpython-38/src/snowflake/connector/cpp/Logging
  /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_build_env/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fPIC -O2 -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/work=/usr/local/src/conda/snowflake-connector-python-3.0.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac=/usr/local/src/conda-prefix -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fPIC -Isrc/snowflake/connector -Isrc/snowflake/connector/cpp/ArrowIterator -Isrc/snowflake/connector/cpp/Logging -I/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include/python3.8 -c src/snowflake/connector/arrow_iterator.cpp -o build/temp.linux-x86_64-cpython-38/src/snowflake/connector/arrow_iterator.o -isystem/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/site-packages/pyarrow/include -isystem/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/site-packages/numpy/core/include -std=c++17 -D_GLIBCXX_USE_CXX11_ABI=0
  In file included from src/snowflake/connector/arrow_iterator.cpp:806:
  /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include/python3.8/datetime.h:189:25: warning: 'PyDateTimeAPI' defined but not used [-Wunused-variable]
    189 | static PyDateTime_CAPI *PyDateTimeAPI = NULL;
        |                         ^~~~~~~~~~~~~
  /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_build_env/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fPIC -O2 -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/work=/usr/local/src/conda/snowflake-connector-python-3.0.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac=/usr/local/src/conda-prefix -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fPIC -Isrc/snowflake/connector -Isrc/snowflake/connector/cpp/ArrowIterator -Isrc/snowflake/connector/cpp/Logging -I/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include/python3.8 -c src/snowflake/connector/cpp/ArrowIterator/BinaryConverter.cpp -o build/temp.linux-x86_64-cpython-38/src/snowflake/connector/cpp/ArrowIterator/BinaryConverter.o -isystem/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/site-packages/pyarrow/include -isystem/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/site-packages/numpy/core/include -std=c++17 -D_GLIBCXX_USE_CXX11_ABI=0
  src/snowflake/connector/cpp/ArrowIterator/BinaryConverter.cpp: In member function 'virtual PyObject* sf::BinaryConverter::toPyObject(int64_t) const':
  src/snowflake/connector/cpp/ArrowIterator/BinaryConverter.cpp:21:43: error: conversion from 'nonstd::sv_lite::string_view' {aka 'nonstd::sv_lite::basic_string_view<char>'} to non-scalar type 'std::string_view' {aka 'std::basic_string_view<char>'} requested
     21 |     std::string_view sv = m_array->GetView(rowIndex);
        |                           ~~~~~~~~~~~~~~~~^~~~~~~~~~
  error: command '/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_build_env/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
  error: subprocess-exited-with-error

  × Building wheel for snowflake-connector-python (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin/python /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpumw8r2di
  cwd: /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674903862936/work
  Building wheel for snowflake-connector-python (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for snowflake-connector-python
hajapy commented 1 year ago

A bunch of the failures are now:

    File "<string>", line 202, in _get_arrow_lib_as_linker_input
  AssertionError

Looks like we already use SF_ARROW_LIBDIR to tell it where to look, but still it cannot find it? Hmm.

Edit: added a debug to show what lib it couldn't find:

  AssertionError: unable to find: $/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674910167115/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libarrow_python.so
  error: subprocess-exited-with-error
hajapy commented 1 year ago

Hmm, I checked inside a published pyarrow archive and do not see a libarrow_python.so file.

❯ tar --use-compress-program=unzstd -xvf pkg-pyarrow-8.0.1-py38h539e002_15_cpu.tar.zst
x lib/python3.8/site-packages/pyarrow/vendored/__pycache__/__init__.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/fs.py
x lib/python3.8/site-packages/pyarrow/plasma-store-server
x lib/python3.8/site-packages/pyarrow/_dataset.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/lib.pyx
x lib/python3.8/site-packages/pyarrow/serialization.pxi
x lib/python3.8/site-packages/pyarrow/_plasma.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/error.pxi
x bin/plasma_store
x lib/python3.8/site-packages/pyarrow/__pycache__/pandas_compat.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/array.pxi
x lib/python3.8/site-packages/pyarrow/__pycache__/__init__.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/__pycache__/serialization.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_s3fs.pyx
x lib/python3.8/site-packages/pyarrow/includes/libarrow_fs.pxd
x lib/python3.8/site-packages/pyarrow/_dataset_parquet.pyx
x lib/python3.8/site-packages/pyarrow/ipc.pxi
x lib/python3.8/site-packages/pyarrow/vendored/docscrape.py
x lib/python3.8/site-packages/pyarrow/parquet/__init__.py
x lib/python3.8/site-packages/pyarrow/hdfs.py
x lib/python3.8/site-packages/pyarrow/_fs.pyx
x lib/python3.8/site-packages/pyarrow/__pycache__/types.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/parquet/encryption.py
x lib/python3.8/site-packages/pyarrow/__pycache__/compat.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow-8.0.1-py3.8.egg-info/PKG-INFO
x lib/python3.8/site-packages/pyarrow-8.0.1-py3.8.egg-info/entry_points.txt
x lib/python3.8/site-packages/pyarrow/compute.py
x lib/python3.8/site-packages/pyarrow/types.pxi
x lib/python3.8/site-packages/pyarrow/__pycache__/fs.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/memory.pxi
x lib/python3.8/site-packages/pyarrow/includes/__init__.pxd
x lib/python3.8/site-packages/pyarrow/_hdfs.pyx
x lib/python3.8/site-packages/pyarrow/_dataset.pxd
x lib/python3.8/site-packages/pyarrow/__pycache__/dataset.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/tensorflow/plasma_op.cc
x lib/python3.8/site-packages/pyarrow/_hdfsio.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/public-api.pxi
x lib/python3.8/site-packages/pyarrow/_parquet.pxd
x lib/python3.8/site-packages/pyarrow/_s3fs.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/_orc.pxd
x lib/python3.8/site-packages/pyarrow/includes/libarrow_flight.pxd
x lib/python3.8/site-packages/pyarrow/_orc.pyx
x lib/python3.8/site-packages/pyarrow/__pycache__/cffi.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_generated_version.py
x lib/python3.8/site-packages/pyarrow/_cuda.pyx
x lib/python3.8/site-packages/pyarrow-8.0.1-py3.8.egg-info/SOURCES.txt
x lib/python3.8/site-packages/pyarrow/builder.pxi
x lib/python3.8/site-packages/pyarrow/_csv.pyx
x lib/python3.8/site-packages/pyarrow/pandas-shim.pxi
x lib/python3.8/site-packages/pyarrow/_fs.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/lib.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/_exec_plan.pyx
x lib/python3.8/site-packages/pyarrow/includes/libarrow.pxd
x lib/python3.8/site-packages/pyarrow-8.0.1-py3.8.egg-info/requires.txt
x lib/python3.8/site-packages/pyarrow-8.0.1-py3.8.egg-info/top_level.txt
x lib/python3.8/site-packages/pyarrow/__pycache__/json.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow-8.0.1-py3.8.egg-info/not-zip-safe
x lib/python3.8/site-packages/pyarrow/config.pxi
x lib/python3.8/site-packages/pyarrow/pandas_compat.py
x lib/python3.8/site-packages/pyarrow/_dataset_orc.pyx
x lib/python3.8/site-packages/pyarrow/includes/common.pxd
x lib/python3.8/site-packages/pyarrow/scalar.pxi
x lib/python3.8/site-packages/pyarrow/_dataset.pyx
x lib/python3.8/site-packages/pyarrow/gandiva.pyx
x lib/python3.8/site-packages/pyarrow/_dataset_parquet.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/_parquet.pyx
x lib/python3.8/site-packages/pyarrow/__pycache__/cuda.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/__init__.pxd
x lib/python3.8/site-packages/pyarrow/table.pxi
x lib/python3.8/site-packages/pyarrow/includes/libplasma.pxd
x lib/python3.8/site-packages/pyarrow/compat.pxi
x lib/python3.8/site-packages/pyarrow/_fs.pxd
x lib/python3.8/site-packages/pyarrow/json.py
x lib/python3.8/site-packages/pyarrow/parquet/__pycache__/encryption.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/gandiva.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/io.pxi
x lib/python3.8/site-packages/pyarrow/_feather.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/csv.py
x lib/python3.8/site-packages/pyarrow/_csv.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/_exec_plan.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/_json.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/ipc.py
x lib/python3.8/site-packages/pyarrow/orc.py
x lib/python3.8/site-packages/pyarrow/__init__.py
x lib/python3.8/site-packages/pyarrow/vendored/__init__.py
x lib/python3.8/site-packages/pyarrow/__pycache__/filesystem.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/benchmark.pxi
x lib/python3.8/site-packages/pyarrow/_hdfsio.pyx
x lib/python3.8/site-packages/pyarrow/__pycache__/flight.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_dataset_orc.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/_compute.pxd
x lib/python3.8/site-packages/pyarrow/_orc.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/includes/libgandiva.pxd
x lib/python3.8/site-packages/pyarrow/dataset.py
x lib/python3.8/site-packages/pyarrow/jvm.py
x lib/python3.8/site-packages/pyarrow/includes/libarrow_cuda.pxd
x lib/python3.8/site-packages/pyarrow/__pycache__/benchmark.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/vendored/__pycache__/docscrape.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_flight.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/_parquet_encryption.pyx
x lib/python3.8/site-packages/pyarrow/_cuda.pxd
x lib/python3.8/site-packages/pyarrow/plasma.py
x lib/python3.8/site-packages/pyarrow/vendored/__pycache__/version.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/includes/libarrow_dataset_parquet.pxd
x info/licenses/LICENSE.txt
x lib/python3.8/site-packages/pyarrow/_parquet_encryption.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/compat.py
x lib/python3.8/site-packages/pyarrow/includes/libarrow_feather.pxd
x lib/python3.8/site-packages/pyarrow/__pycache__/orc.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_hdfs.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/__pycache__/_generated_version.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_parquet_encryption.pxd
x lib/python3.8/site-packages/pyarrow/lib.pxd
x lib/python3.8/site-packages/pyarrow/tensor.pxi
x lib/python3.8/site-packages/pyarrow/__pycache__/jvm.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/__pycache__/csv.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/cuda.py
x lib/python3.8/site-packages/pyarrow/_compute.pyx
x lib/python3.8/site-packages/pyarrow/feather.py
x lib/python3.8/site-packages/pyarrow/__pycache__/compute.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/__pycache__/ipc.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/includes/libarrow_dataset.pxd
x lib/python3.8/site-packages/pyarrow/_csv.pxd
x lib/python3.8/site-packages/pyarrow/__pycache__/feather.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/__pycache__/_compute_docstrings.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_flight.pyx
x lib/python3.8/site-packages/pyarrow/_compute.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/serialization.py
x lib/python3.8/site-packages/pyarrow/parquet/__pycache__/__init__.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/__pycache__/util.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/lib_api.h
x lib/python3.8/site-packages/pyarrow/benchmark.py
x lib/python3.8/site-packages/pyarrow/_feather.pyx
x lib/python3.8/site-packages/pyarrow/types.py
x lib/python3.8/site-packages/pyarrow/_plasma.pyx
x lib/python3.8/site-packages/pyarrow/filesystem.py
x lib/python3.8/site-packages/pyarrow/__pycache__/plasma.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/_json.pyx
x lib/python3.8/site-packages/pyarrow/_compute_docstrings.py
x lib/python3.8/site-packages/pyarrow/util.py
x lib/python3.8/site-packages/pyarrow/includes/libarrow_python.pxd
x lib/python3.8/site-packages/pyarrow/cffi.py
x lib/python3.8/site-packages/pyarrow/_parquet.cpython-38-aarch64-linux-gnu.so
x lib/python3.8/site-packages/pyarrow/__pycache__/hdfs.cpython-38.pyc
x lib/python3.8/site-packages/pyarrow/vendored/version.py
x lib/python3.8/site-packages/pyarrow/flight.py
x lib/python3.8/site-packages/pyarrow-8.0.1-py3.8.egg-info/dependency_links.txt
hajapy commented 1 year ago

Looks like we might need patches that depend on arrow version and also looks like the pyarrow build is missing so objects for arrow versions less than 10. I'll open an issue at https://github.com/conda-forge/arrow-cpp-feedstock

hajapy commented 1 year ago

It looks like the wrong variants of pyarrow are being installed for different architectures. This will also need a patch to switch between std::string_view and arrow::util::string_view for different arrow versions. I have to put this down for now, but I made some progress. If someone wants to pick it up, would be happy to have the assistance.

hajapy commented 1 year ago

It's compiling better now but running into issues finding the needed DSO (all arrow 10 builds) or segfaulting due to missing symbols (all linux builds except arrow 10).

Example from linux_64_arrow_cpp10.0.1numpy1.20python3.8.____cpython failure

   INFO (snowflake-connector-python,lib/python3.8/site-packages/snowflake/connector/arrow_iterator.cpython-38-x86_64-linux-gnu.so): Needed DSO lib/libarrow.so.1000 found in conda-forge::libarrow-10.0.1-h86614e7_4_cpu
WARNING (snowflake-connector-python,lib/python3.8/site-packages/snowflake/connector/arrow_iterator.cpython-38-x86_64-linux-gnu.so): $RPATH/libarrow_python.so.1000 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?

Then in the import test:

import: 'snowflake.connector'
Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: libarrow_python.so.1000: cannot open shared object file: No such file or directory
import: 'snowflake.connector.arrow_iterator'
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674962051950/test_tmp/run_test.py", line 8, in <module>
    import snowflake.connector.arrow_iterator
ImportError: libarrow_python.so.1000: cannot open shared object file: No such file or directory

linux_aarch64_arrow_cpp9.0.0numpy1.23python3.11.____cpython

   INFO (snowflake-connector-python,lib/python3.11/site-packages/snowflake/connector/arrow_iterator.cpython-311-aarch64-linux-gnu.so): Needed DSO lib/libarrow.so.900 found in conda-forge::arrow-cpp-9.0.0-py311hdef34c3_16_cpu
   INFO (snowflake-connector-python,lib/python3.11/site-packages/snowflake/connector/arrow_iterator.cpython-311-aarch64-linux-gnu.so): Needed DSO lib/libarrow_python.so.900 found in conda-forge::arrow-cpp-9.0.0-py311hdef34c3_16_cpu

import: 'snowflake.connector'
Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: /home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674962512871/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.11/site-packages/snowflake/connector/arrow_iterator.cpython-311-aarch64-linux-gnu.so: undefined symbol: _ZNK5arrow16KeyValueMetadata5valueEl
import: 'snowflake.connector.arrow_iterator'
/home/conda/feedstock_root/build_artifacts/snowflake-connector-python_1674962512871/test_tmp/conda_test_runner.sh: line 3:  1223 Segmentation fault      (core dumped)

demangled symbol name: arrow::KeyValueMetadata::value(long) const

hajapy commented 1 year ago

@conda-forge/arrow-cpp do you have any ideas how to proceed here or what might be wrong?

h-vetinari commented 1 year ago

It's compiling better now but running into issues finding the needed DSO (all arrow 10 builds) or segfaulting due to missing symbols (all linux builds except arrow 10).

This is because there were substantial changes in the library setup in arrow v10 (and all following versions) - the core libarrow library is now not dependent on python anymore (therefore only needs to be built once, and therefore lives in $PREFIX/lib rather than $SP_DIR/pyarrow), and the bits that do depend on it are now in libarrow_python.so. More details in https://github.com/conda-forge/arrow-cpp-feedstock/pull/875.

This new version was then migrated (I commented in the bot PR at the time), but then the migration was closed, meaning that conda-forge by default will now build arrow-cpp for 7,8,9,10.

It's possible that snowflake-connector-python is not yet ready for v10, in that case it could be skipped here, but should be tackled quickly with upstream (arrow v11 was released a few days ago). Otherwise, you might get there by adding:

rpaths:
  - lib/
  - $SP_DIR/pyarrow

to the build section.

hajapy commented 1 year ago

It's possible that snowflake-connector-python is not yet ready for v10, in that case it could be skipped here, but should be tackled quickly with upstream (arrow v11 was released a few days ago). Otherwise, you might get there by adding:

The connector only supports arrow 10 as of v3. Previously, it only supported arrow 8, though in CF we built it for multiple versions. Now we also have to patch the connector's cpp code to make it work for arrow 7, 8, 9, and 10 which is a new level of patching needed on this recipe and increases the support burden.

rpaths:
  - lib/
  - $SP_DIR/pyarrow

to the build section.

I will try this!

jordantshaw commented 1 year ago

@hajapy any luck on getting this build working? Really need v3.0.0

hajapy commented 1 year ago

I don't really have any more ideas how to fix it. Welcome to suggestions or contributions from others.