search

sfu-db / connector-x

Fastest library to load data from DB to DataFrames in Rust and Python
https://sfu-db.github.io/connector-x
MIT License
2k stars 162 forks source link

[ARM64] ERROR: No matching distribution found for connectorx==0.2.2 #186

Open Matthieu-Tinycoaching opened 2 years ago

Matthieu-Tinycoaching commented 2 years ago

Hello,

I would like to use connector-x on ARM64 architecture. However, it seems that this library has no ARM architecture version when trying to pip install it via docker buildx on this type of architecture.

Would it be possible to release an ARM64 version of the library?

Thanks!

wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching, unfortunately we do not have a way to build wheel for ARM architecture for now. And the only way to use connector-x on it is build the project manually.

To manually build connector-x, you need to:

  1. Configure the environment. You can find the steps to install tools and dependencies here.
  2. Build connector-x. You can directly with command: just setup-python under project directory.
  3. You can find the target folder at: connectorx-python/connectorx. You can also copy it to your local path for python libraries.
Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying thanks for your feedback.

If I build it manually for arm architecture, where do I specify that I want an arm package?

dovahcrow commented 2 years ago

@Matthieu-Tinycoaching no need to specify the arch. The script only builds the package for the host platform.

Basically this command just bootstrap-python ci-build-python-extention ci-build-python-wheel ci-rename-wheel from: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L56

The wheel will be under connectorx-python/dist/

Matthieu-Tinycoaching commented 2 years ago

@dovahcrow thanks for your feedback.

Does this mean I have to be on arm64 architecture to build a version of connectorx for this host?

dovahcrow commented 2 years ago

@dovahcrow thanks for your feedback.

Does this mean I have to be on arm64 architecture to build a version of connectorx for this host?

Correct. That's the main reason why we don't provide an arm64 pip package - because GitHub action does not have arm64 yet.

Matthieu-Tinycoaching commented 2 years ago

@dovahcrow @wangxiaoying when trying to install tools and dependencies on ARM64 AWs instance : just bootstrap-python

I got the following error: error: No justfile found

Do you have any idea on how to deal with?

Matthieu-Tinycoaching commented 2 years ago

@dovahcrow @wangxiaoying I manage to deal with above error by installing git and cloning the repo.

Now, when running the command : just bootstrap-python

cp README.md connectorx-python/README.md
cp LICENSE connectorx-python/LICENSE
cd connectorx-python && poetry install
Creating virtualenv connectorx-LhtZ5s62-py3.7 in /home/ec2-user/.cache/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 99 installs, 0 updates, 0 removals

  • Installing pycparser (2.20)
  • Installing cffi (1.15.0)
  • Installing pyparsing (2.4.7)
  • Installing typing-extensions (3.10.0.2)
  • Installing zipp (3.6.0)
  • Installing backports.zoneinfo (0.2.1): Failed

  EnvCommandError

  Command ['/home/ec2-user/.cache/pypoetry/virtualenvs/connectorx-LhtZ5s62-py3.7/bin/pip', 'install', '--no-deps', '/home/ec2-user/.cache/pypoetry/artifacts/a2/4e/47/7614879069f1b315f09a6b3734172eaf47b88375aeeb8f68985190ee86/backports.zoneinfo-0.2.1.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/ec2-user/.cache/pypoetry/artifacts/a2/4e/47/7614879069f1b315f09a6b3734172eaf47b88375aeeb8f68985190ee86/backports.zoneinfo-0.2.1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: backports.zoneinfo
    Building wheel for backports.zoneinfo (pyproject.toml): started
    Building wheel for backports.zoneinfo (pyproject.toml): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/ec2-user/.cache/pypoetry/virtualenvs/connectorx-LhtZ5s62-py3.7/bin/python /home/ec2-user/.cache/pypoetry/virtualenvs/connectorx-LhtZ5s62-py3.7/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpygu046zq
         cwd: /tmp/pip-req-build-ic470pko
    Complete output (30 lines):
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-aarch64-3.7
    creating build/lib.linux-aarch64-3.7/backports
    copying src/backports/__init__.py -> build/lib.linux-aarch64-3.7/backports
    creating build/lib.linux-aarch64-3.7/backports/zoneinfo
    copying src/backports/zoneinfo/__init__.py -> build/lib.linux-aarch64-3.7/backports/zoneinfo
    copying src/backports/zoneinfo/_common.py -> build/lib.linux-aarch64-3.7/backports/zoneinfo
    copying src/backports/zoneinfo/_tzpath.py -> build/lib.linux-aarch64-3.7/backports/zoneinfo
    copying src/backports/zoneinfo/_version.py -> build/lib.linux-aarch64-3.7/backports/zoneinfo
    copying src/backports/zoneinfo/_zoneinfo.py -> build/lib.linux-aarch64-3.7/backports/zoneinfo
    running egg_info
    warning: no files found matching '*.png' under directory 'docs'
    warning: no files found matching '*.svg' under directory 'docs'
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'docs/_output'
    writing manifest file 'src/backports.zoneinfo.egg-info/SOURCES.txt'
    copying src/backports/zoneinfo/__init__.pyi -> build/lib.linux-aarch64-3.7/backports/zoneinfo
    copying src/backports/zoneinfo/py.typed -> build/lib.linux-aarch64-3.7/backports/zoneinfo
    running build_ext
    creating build/temp.linux-aarch64-3.7
    creating build/temp.linux-aarch64-3.7/lib
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -moutline-atomics -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/ec2-user/.cache/pypoetry/virtualenvs/connectorx-LhtZ5s62-py3.7/include -I/usr/include/python3.7m -c lib/zoneinfo_module.c -o build/temp.linux-aarch64-3.7/lib/zoneinfo_module.o -std=c99
    lib/zoneinfo_module.c:1:10: erreur fatale: Python.h : No such file or directory
     #include "Python.h"
              ^~~~~~~~~~
    compilation terminée.
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
    ERROR: Failed building wheel for backports.zoneinfo
  Failed to build backports.zoneinfo
  ERROR: Could not build wheels for backports.zoneinfo, which is required to install pyproject.toml-based projects

  at ~/.local/lib/python3.7/site-packages/poetry/utils/env.py:1183 in _run
      1179│                 output = subprocess.check_output(
      1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1181│                 )
      1182│         except CalledProcessError as e:
    → 1183│             raise EnvCommandError(e, input=input_)
      1184│ 
      1185│         return decode(output)
      1186│ 
      1187│     def execute(self, bin, *args, **kwargs):

  • Installing certifi (2021.10.8)
  • Installing charset-normalizer (2.0.7)
  • Installing cryptography (35.0.0)
  • Installing heapdict (1.0.1)
  • Installing idna (3.3)
  • Installing importlib-metadata (4.8.1)
  • Installing jeepney (0.7.1)
  • Installing markupsafe (2.0.1)
  • Installing packaging (21.2)
  • Installing six (1.16.0)
  • Installing tzdata (2021.5)
  • Installing urllib3 (1.26.7)
  • Installing webencodings (0.5.1)
error: Recipe `bootstrap-python` failed on line 26 with exit code 1
Matthieu-Tinycoaching commented 2 years ago

I managed to pass the previous error by installing a pyenv environment with python3 instead of the python2 system version.

Now I have the following error:

cp README.md connectorx-python/README.md
cp LICENSE connectorx-python/LICENSE
cd connectorx-python && poetry install
Installing dependencies from lock file

Package operations: 77 installs, 11 updates, 0 removals

  • Updating pycparser (2.21 -> 2.20)
  • Updating pyparsing (3.0.6 -> 2.4.7)
  • Installing backports.zoneinfo (0.2.1)
  • Updating charset-normalizer (2.0.10 -> 2.0.7)
  • Updating cryptography (36.0.1 -> 35.0.0)
  • Installing heapdict (1.0.1)
  • Installing markupsafe (2.0.1)
  • Updating packaging (20.9 -> 21.2)
  • Installing tzdata (2021.5)
  • Updating urllib3 (1.26.8 -> 1.26.7)
  • Installing zipp (3.6.0)
  • Installing attrs (21.2.0)
  • Installing bleach (4.1.0)
  • Installing click (8.0.3)
  • Installing cloudpickle (2.0.0)
  • Installing dask (2021.12.0)
  • Installing docutils (0.18)
  • Installing greenlet (1.1.2)
  • Installing importlib-metadata (4.8.1)
  • Installing iniconfig (1.1.1)
  • Installing jinja2 (3.0.2)
  • Installing locket (0.2.1)
  • Updating msgpack (1.0.3 -> 1.0.2)
  • Installing numpy (1.21.5)
  • Installing parso (0.8.2)
  • Installing pluggy (1.0.0)
  • Installing psutil (5.8.0)
  • Installing py (1.11.0)
  • Installing pygments (2.10.0)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2021.3)
  • Installing pytz-deprecation-shim (0.1.0.post0)
  • Installing pyyaml (6.0)
  • Updating requests (2.27.1 -> 2.26.0)
  • Installing sortedcontainers (2.4.0)
  • Installing tblib (1.7.0)
  • Installing toml (0.10.2)
  • Installing toolz (0.11.1)
  • Installing tornado (6.1)
  • Installing traitlets (5.1.1)
  • Installing wcwidth (0.2.5)
  • Installing zict (2.0.0)
  • Installing backcall (0.2.0)
  • Installing colorama (0.4.4)
  • Installing decorator (5.1.0)
  • Installing distributed (2021.12.0)
  • Installing fsspec (2021.10.1)
  • Installing future (0.18.2)
  • Installing jedi (0.18.0)
  • Updating keyring (21.8.0 -> 23.2.1)
  • Installing matplotlib-inline (0.1.3)
  • Installing mypy-extensions (0.4.3)
  • Installing pandas (1.3.4)
  • Installing partd (1.2.0)
  • Installing pathspec (0.9.0)
  • Installing pickleshare (0.7.5)
  • Updating pkginfo (1.8.2 -> 1.7.1)
  • Updating platformdirs (2.4.1 -> 2.4.0)
  • Installing prompt-toolkit (3.0.22)
  • Installing py-cpuinfo (8.0.0)
  • Installing pytest (6.2.5)
  • Installing readme-renderer (30.0)
  • Installing rfc3986 (1.5.0)
  • Installing sqlalchemy (1.4.26)
  • Installing tomli (1.2.3)
  • Installing tqdm (4.62.3)
  • Installing typing-extensions (3.10.0.2)
  • Installing tzlocal (4.1)
  • Installing black (21.12b0)
  • Installing clickhouse-driver (0.2.2)
  • Installing contexttimer (0.3.3)
  • Installing cx-oracle (8.3.0): Installing...
  • Installing docopt (0.6.2)
  • Installing ipython (7.29.0)
  • Installing maturin (0.9.4): Installing...
  • Installing modin (0.11.2): Installing...
  • Installing mysqlclient (2.1.0): Failed

  EnvCommandError

  Command ['/home/ec2-user/.pyenv/versions/3.8.12/envs/connectorx/bin/pip', 'install', '--no-deps', '/home/ec2-user/.cache/pypoetry/artifacts/f5/5b/d2/f720445dea9b2f0649ebc8ae23dea8387859deb885e3dc763eb039759b/mysqlclient-2.1.0.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/ec2-user/.cache/pypoetry/artifacts/f5/5b/d2/f720445dea9b2f0649ebc8ae23dea8387859deb885e3dc763eb039759b/mysqlclient-2.1.0.tar.gz
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/ec2-user/.pyenv/versions/3.8.12/envs/connectorx/bin/python3.8 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-j56fzn01/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-j56fzn01/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-6ttraa__
         cwd: /tmp/pip-req-build-j56fzn01/
    Complete output (15 lines):
    /bin/sh: mysql_config : commande introuvable
  • Installing modin (0.11.2): Installing...
...

 • Installing pymssql (2.2.3): Failed

  EnvCommandError

  Command ['/home/ec2-user/.pyenv/versions/3.8.12/envs/connectorx/bin/pip', 'install', '--no-deps', '/home/ec2-user/.cache/pypoetry/artifacts/94/c0/71/28ede7c992d2764b5ae4afb5ac9af9d81d42e2aadd1872e9e48c4bb140/pymssql-2.2.3.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/ec2-user/.cache/pypoetry/artifacts/94/c0/71/28ede7c992d2764b5ae4afb5ac9af9d81d42e2aadd1872e9e48c4bb140/pymssql-2.2.3.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: pymssql
    Building wheel for pymssql (pyproject.toml): started
    Building wheel for pymssql (pyproject.toml): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/ec2-user/.pyenv/versions/3.8.12/envs/connectorx/bin/python3.8 /home/ec2-user/.pyenv/versions/3.8.12/envs/connectorx/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpc_m8vqak
         cwd: /tmp/pip-req-build-osig5f6t
    Complete output (22 lines):
    setup.py: platform.system() => Linux
    setup.py: platform.architecture() => ('64bit', 'ELF')
    setup.py: platform.libc_ver() => ('glibc', '2.26')
    setup.py: include_dirs => []
    setup.py: library_dirs => []
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-aarch64-3.8
    creating build/lib.linux-aarch64-3.8/pymssql
    • Installing presto-python-client (0.8.2)
  • Installing psycopg2-binary (2.9.3)
  • Installing pyarrow (6.0.0)
  • Installing pyhive (0.6.4)
wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching , pymssql is only used for benchmarking other methods. I think here you can just remove it by running poetry remove pymssql --dev under connectorx-python directory. The same applies to other packages under [tool.poetry.dev-dependencies] in connectorx-python/pyproject.toml

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying thanks it worked.

However, when running: just ci-build-python-extention ci-build-python-wheel ci-rename-wheel

I got the following error now:

cd connectorx-python && cargo build --release
  Downloaded async-stream-impl v0.2.1
  Downloaded async-stream v0.2.1
  Downloaded const_fn v0.4.8
  Downloaded bitfield v0.13.2
  Downloaded async-trait v0.1.51
  Downloaded cmake v0.1.46
  Downloaded instant v0.1.11
  Downloaded http-body v0.4.4
  Downloaded bb8 v0.7.1
  Downloaded bytes v1.1.0
  Downloaded bindgen v0.58.1
  Downloaded pin-utils v0.1.0
  Downloaded futures-macro v0.3.17
  Downloaded num_cpus v1.13.0
  Downloaded mysql v21.0.2
  Downloaded pkg-config v0.3.20
  Downloaded phf v0.10.0
  Downloaded parking_lot_core v0.8.5
  Downloaded num v0.4.0
  Downloaded polars-lazy v0.18.0
  Downloaded polars-core v0.18.0
  Downloaded prettytable-rs v0.8.0
  Downloaded polars-io v0.18.0
  Downloaded num-complex v0.4.0
  Downloaded phf_shared v0.10.0
  Downloaded native-tls v0.2.8
  Downloaded itertools v0.10.1
  Downloaded proc-macro-hack v0.5.19
  Downloaded owning_ref v0.4.1
  Downloaded pyo3-built v0.4.6
  Downloaded postgres-native-tls v0.5.0
  Downloaded pyo3-macros v0.14.5
  Downloaded dirs v1.0.5
  Downloaded libloading v0.7.1
  Downloaded lexical-core v0.8.2
  Downloaded nom v5.1.2
  Downloaded memmap2 v0.5.0
  Downloaded httpdate v1.0.2
  Downloaded nix v0.23.0
  Downloaded paste-impl v0.1.18
  Downloaded multiversion v0.6.1
  Downloaded pyo3 v0.14.5
  Downloaded lock_api v0.4.5
  Downloaded tracing-attributes v0.1.18
  Downloaded r2d2 v0.8.9
  Downloaded want v0.3.0
  Downloaded which v3.1.1
  Downloaded untrusted v0.7.1
  Downloaded tap v1.0.1
  Downloaded tokio-util v0.6.9
  Downloaded unicode-bidi v0.3.6
  Downloaded thiserror-impl v1.0.29
  Downloaded standback v0.2.17
  Downloaded unindent v0.1.7
  Downloaded wyz v0.4.0
  Downloaded pyo3-build-config v0.14.5
  Downloaded ndarray v0.15.3
  Downloaded dirs-sys v0.3.6
  Downloaded itoa v0.4.8
  Downloaded getrandom v0.2.3
  Downloaded digest v0.9.0
  Downloaded encoding_index_tests v0.1.4
  Downloaded encoding-index-tradchinese v1.20141219.5
  Downloaded proc-macro2 v1.0.29
  Downloaded libz-sys v1.1.3
  Downloaded scopeguard v1.1.0
  Downloaded serde v1.0.130
  Downloaded static_assertions v1.1.0
  Downloaded slab v0.4.4
  Downloaded rustc-hash v1.1.0
  Downloaded tokio-native-tls v0.3.0
  Downloaded time-macros v0.1.1
  Downloaded time v0.2.27
  Downloaded try-lock v0.2.3
  Downloaded tokio-macros v1.4.1
  Downloaded twox-hash v1.6.1
  Downloaded tracing-core v0.1.21
  Downloaded term v0.5.2
  Downloaded sha1 v0.6.0
  Downloaded time-macros-impl v0.1.2
  Downloaded tokio-postgres v0.7.3
  Downloaded tokio v1.12.0
  Downloaded tracing v0.1.29
  Downloaded pyo3-macros-backend v0.14.5
  Downloaded toml v0.5.8
  Downloaded rayon v1.5.1
  Downloaded time v0.1.44
  Downloaded unsafe_unwrap v0.1.0
  Downloaded serde_json v1.0.68
  Downloaded subprocess v0.2.8
  Downloaded tower-service v0.3.1
  Downloaded unicode-normalization v0.1.19
  Downloaded typenum v1.14.0
  Downloaded saturating v0.1.0
  Downloaded syn v1.0.78
  Downloaded urlencoding v2.1.0
  Downloaded url v2.2.2
  Downloaded lexical-write-float v0.8.2
  Downloaded tinyvec v1.5.0
  Downloaded libsqlite3-sys v0.22.2
  Downloaded serde_urlencoded v0.7.0
  Downloaded num-traits v0.2.14
  Downloaded version_check v0.9.3
  Downloaded mysql_common v0.27.5
  Downloaded ppv-lite86 v0.2.10
  Downloaded proc-macro-nested v0.1.7
  Downloaded pem v0.8.3
  Downloaded postgres-protocol v0.6.2
  Downloaded postgres-types v0.2.2
  Downloaded tiberius v0.5.16
  Downloaded sqlparser v0.11.0
  Downloaded paste v0.1.18
  Downloaded postgres v0.19.2
  Downloaded num-rational v0.4.0
  Downloaded sct v0.6.1
  Downloaded webpki-roots v0.21.1
  Downloaded webpki v0.21.4
  Downloaded ring v0.16.20
  Downloaded uuid v0.8.2
  Downloaded tokio-rustls v0.22.0
  Downloaded rustls v0.19.1
  Downloaded rand_chacha v0.3.1
  Downloaded lexical-core v0.7.6
  Downloaded csv v1.1.6
  Downloaded matches v0.1.9
  Downloaded num-bigint v0.4.2
  Downloaded pin-project-lite v0.2.7
  Downloaded crypto-mac v0.11.1
  Downloaded hex v0.4.3
  Downloaded ipnet v2.3.1
  Downloaded hmac v0.11.0
  Downloaded rand v0.8.4
  Downloaded ryu v1.0.5
  Downloaded rustls-native-certs v0.5.0
  Downloaded r2d2_sqlite v0.18.0
  Downloaded reqwest v0.11.6
  Downloaded bb8-tiberius v0.5.2
  Downloaded rand_core v0.6.3
  Downloaded r2d2_mysql v21.0.0
  Downloaded strength_reduce v0.2.3
  Downloaded socket2 v0.4.2
  Downloaded shlex v1.1.0
  Downloaded rusqlite v0.25.3
  Downloaded seahash v4.1.0
  Downloaded radium v0.6.2
  Downloaded smallvec v1.7.0
  Downloaded sha2 v0.9.8
  Downloaded streaming-iterator v0.1.5
  Downloaded pretty-hex v0.1.1
  Downloaded rust_decimal_macros v1.16.0
  Downloaded hyper v0.14.15
  Downloaded encoding-index-simpchinese v1.20141219.5
  Downloaded siphasher v0.3.7
  Downloaded rayon-core v1.9.1
  Downloaded clap v2.33.3
  Downloaded serde_derive v1.0.130
  Downloaded rawpointer v0.2.1
  Downloaded rust_decimal v1.16.0
  Downloaded r2d2_postgres v0.18.1
  Downloaded arrow2 v0.8.1
  Downloaded dict_derive v0.4.0
  Downloaded bufstream v0.1.4
  Downloaded thiserror v1.0.29
  Downloaded stringprep v0.1.2
  Downloaded lexical v6.0.1
  Downloaded encoding-index-japanese v1.20141219.5
  Downloaded subtle v2.4.1
  Downloaded r2d2-oracle v0.5.0
  Downloaded indoc v0.3.6
  Downloaded stable_deref_trait v1.2.0
  Downloaded lexical-parse-integer v0.8.0
  Downloaded idna v0.2.3
  Downloaded simdutf8 v0.1.3
  Downloaded memoffset v0.6.4
  Downloaded encoding v0.2.33
  Downloaded spin v0.5.2
  Downloaded semver v1.0.4
  Downloaded fehler v1.0.0
  Downloaded frunk_core v0.4.0
  Downloaded libc v0.2.103
  Downloaded scheduled-thread-pool v0.2.5
  Downloaded md5 v0.7.0
  Downloaded enumflags2_derive v0.7.0
  Downloaded tinyvec_macros v0.1.0
  Downloaded quote v1.0.10
  Downloaded fallible-streaming-iterator v0.1.9
  Downloaded num-integer v0.1.44
  Downloaded mime v0.3.16
  Downloaded env_logger v0.8.4
  Downloaded polars-arrow v0.18.0
  Downloaded oracle v0.5.3
  Downloaded foreign-types-shared v0.1.1
  Downloaded encoding_rs v0.8.29
  Downloaded frunk_proc_macro_helpers v0.1.0
  Downloaded lexical-util v0.8.1
  Downloaded frunk_proc_macros_impl v0.1.0
  Downloaded lru v0.6.6
  Downloaded futures-executor v0.3.17
  Downloaded frunk_proc_macros v0.1.0
  Downloaded connection-string v0.1.13
  Downloaded frunk_derives v0.4.0
  Downloaded fallible-iterator v0.2.0
  Downloaded crc32fast v1.2.1
  Downloaded futures-sink v0.3.17
  Downloaded enumflags2 v0.7.1
  Downloaded matrixmultiply v0.3.1
  Downloaded lexical-parse-float v0.8.2
  Downloaded asynchronous-codec v0.5.0
  Downloaded base64 v0.13.0
  Downloaded num-iter v0.1.42
  Downloaded futures-channel v0.3.17
  Downloaded futures v0.3.17
  Downloaded flate2 v1.0.22
  Downloaded flatbuffers v2.0.0
  Downloaded httparse v1.5.1
  Downloaded glob v0.3.0
  Downloaded arrow v5.5.0
  Downloaded dirs v4.0.0
  Downloaded crossbeam-channel v0.5.1
  Downloaded bitvec v0.22.3
  Downloaded num-bigint v0.3.3
  Downloaded md-5 v0.9.1
  Downloaded indexmap v1.7.0
  Downloaded funty v1.2.0
  Downloaded ct-logs v0.8.0
  Downloaded lexical-write-integer v0.8.0
  Downloaded frunk v0.4.0
  Downloaded fnv v1.0.7
  Downloaded fehler-macros v1.0.0
  Downloaded form_urlencoded v1.0.1
  Downloaded foreign-types v0.3.2
  Downloaded csv-core v0.1.10
  Downloaded hyper-rustls v0.22.1
  Downloaded concurrent-queue v1.2.2
  Downloaded cfg-if v0.1.10
  Downloaded cache-padded v1.1.1
  Downloaded block-buffer v0.9.0
  Downloaded polars v0.18.0
  Downloaded chrono v0.4.19
  Downloaded arrayvec v0.5.2
  Downloaded paste v1.0.5
  Downloaded opaque-debug v0.3.0
  Downloaded once_cell v1.8.0
  Downloaded mio v0.7.13
  Downloaded crossbeam-epoch v0.9.5
  Downloaded async-lock v2.4.0
  Downloaded ahash v0.7.4
  Downloaded generic-array v0.14.4
  Downloaded anyhow v1.0.44
  Downloaded parking_lot v0.11.2
  Downloaded lexical v5.2.2
  Downloaded hashbrown v0.11.2
  Downloaded io-enum v1.0.1
  Downloaded peeking_take_while v0.1.2
  Downloaded openssl-sys v0.9.67
  Downloaded openssl v0.10.36
  Downloaded gcp-bigquery-client v0.10.0
  Downloaded numpy v0.14.1
  Downloaded lazycell v1.3.0
  Downloaded hashlink v0.7.0
  Downloaded h2 v0.3.7
  Downloaded http v0.2.5
  Downloaded encoding-index-singlebyte v1.20141219.5
  Downloaded encoding-index-korean v1.20141219.5
  Downloaded encode_unicode v0.3.6
  Downloaded derive_utils v0.11.2
  Downloaded crossbeam-utils v0.8.5
  Downloaded crossbeam-deque v0.8.1
  Downloaded cc v1.0.70
  Downloaded multiversion-macros v0.6.1
  Downloaded hash_hasher v2.0.3
  Downloaded futures-util v0.3.17
  Downloaded cpufeatures v0.2.1
  Downloaded clang-sys v1.2.2
  Downloaded built v0.5.1
  Downloaded cargo-lock v7.0.1
  Downloaded percent-encoding v2.1.0
  Downloaded openssl-probe v0.1.4
  Downloaded futures-task v0.3.17
  Downloaded futures-io v0.3.17
  Downloaded futures-core v0.3.17
  Downloaded cexpr v0.4.0
  Downloaded bigdecimal v0.2.2
  Downloaded async-std v1.10.0
  Downloaded async-native-tls v0.3.3
  Downloaded async-channel v1.6.1
  Downloaded ansi_term v0.11.0
  Downloaded indoc-impl v0.3.6
  Downloaded event-listener v2.5.1
  Downloaded either v1.6.1
  Downloaded byteorder v1.4.3
  Downloaded yup-oauth2 v6.0.0
  Downloaded 292 crates (26.1 MB) in 1.55s (largest was `ring` at 5.1 MB)
   Compiling libc v0.2.103
   Compiling autocfg v1.0.1
   Compiling proc-macro2 v1.0.29
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.78
   Compiling version_check v0.9.3
   Compiling cfg-if v1.0.0
   Compiling memchr v2.4.1
   Compiling crossbeam-utils v0.8.5
   Compiling serde_derive v1.0.130
   Compiling serde v1.0.130
   Compiling log v0.4.14
   Compiling crossbeam-epoch v0.9.5
   Compiling lazy_static v1.4.0
   Compiling scopeguard v1.1.0
   Compiling cc v1.0.70
   Compiling rayon-core v1.9.1
   Compiling once_cell v1.8.0
   Compiling proc-macro-hack v0.5.19
   Compiling futures-core v0.3.17
   Compiling parking_lot_core v0.8.5
   Compiling ryu v1.0.5
   Compiling typenum v1.14.0
   Compiling futures-task v0.3.17
   Compiling futures-channel v0.3.17
   Compiling smallvec v1.7.0
   Compiling proc-macro-nested v0.1.7
   Compiling either v1.6.1
   Compiling itoa v0.4.8
   Compiling pin-project-lite v0.2.7
   Compiling bytes v1.1.0
   Compiling serde_json v1.0.68
   Compiling futures-io v0.3.17
   Compiling pkg-config v0.3.20
   Compiling futures-sink v0.3.17
   Compiling pin-utils v0.1.0
   Compiling slab v0.4.4
   Compiling tinyvec_macros v0.1.0
   Compiling static_assertions v1.1.0
   Compiling base64 v0.13.0
   Compiling unicode-bidi v0.3.6
   Compiling async-trait v0.1.51
   Compiling byteorder v1.4.3
   Compiling percent-encoding v2.1.0
   Compiling bitflags v1.3.2
   Compiling ppv-lite86 v0.2.10
   Compiling subtle v2.4.1
   Compiling fallible-iterator v0.2.0
   Compiling opaque-debug v0.3.0
   Compiling openssl-probe v0.1.4
   Compiling untrusted v0.7.1
   Compiling spin v0.5.2
   Compiling matches v0.1.9
   Compiling md5 v0.7.0
   Compiling siphasher v0.3.7
   Compiling regex-syntax v0.6.25
   Compiling foreign-types-shared v0.1.1
   Compiling regex-automata v0.1.10
   Compiling openssl v0.10.36
   Compiling glob v0.3.0
   Compiling arrayvec v0.5.2
   Compiling native-tls v0.2.8
   Compiling frunk_core v0.4.0
   Compiling fnv v1.0.7
   Compiling anyhow v1.0.44
   Compiling encoding_index_tests v0.1.4
   Compiling unicode-width v0.1.9
   Compiling pyo3-build-config v0.14.5
   Compiling humantime v2.1.0
   Compiling lexical-core v0.7.6
   Compiling strsim v0.8.0
   Compiling bindgen v0.58.1
   Compiling ansi_term v0.11.0
   Compiling vec_map v0.8.2
   Compiling termcolor v1.1.2
   Compiling httparse v1.5.1
   Compiling try-lock v0.2.3
   Compiling hash_hasher v2.0.3
   Compiling radium v0.6.2
   Compiling simdutf8 v0.1.3
   Compiling streaming-iterator v0.1.5
   Compiling event-listener v2.5.1
   Compiling peeking_take_while v0.1.2
   Compiling cache-padded v1.1.1
   Compiling rustc-hash v1.1.0
   Compiling strength_reduce v0.2.3
   Compiling shlex v1.1.0
   Compiling const_fn v0.4.8
   Compiling lazycell v1.3.0
   Compiling crc32fast v1.2.1
   Compiling tap v1.0.1
   Compiling semver v1.0.4
   Compiling httpdate v1.0.2
   Compiling tower-service v0.3.1
   Compiling encode_unicode v0.3.6
   Compiling encoding_rs v0.8.29
   Compiling funty v1.2.0
   Compiling unsafe_unwrap v0.1.0
   Compiling unindent v0.1.7
   Compiling saturating v0.1.0
   Compiling mysql v21.0.2
   Compiling sha1 v0.6.0
   Compiling ipnet v2.3.1
   Compiling rawpointer v0.2.1
   Compiling fallible-streaming-iterator v0.1.9
   Compiling seahash v4.1.0
   Compiling bufstream v0.1.4
   Compiling mime v0.3.16
   Compiling paste v1.0.5
   Compiling connection-string v0.1.13
   Compiling pretty-hex v0.1.1
   Compiling stable_deref_trait v1.2.0
   Compiling hex v0.4.3
   Compiling cfg-if v0.1.10
   Compiling urlencoding v2.1.0
   Compiling pyo3-built v0.4.6
   Compiling bitfield v0.13.2
   Compiling memoffset v0.6.4
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling rayon v1.5.1
   Compiling futures-macro v0.3.17
   Compiling futures-util v0.3.17
   Compiling indexmap v1.7.0
   Compiling tokio v1.12.0
   Compiling num-bigint v0.4.2
   Compiling num-iter v0.1.42
   Compiling num-rational v0.4.0
   Compiling num-bigint v0.3.3
   Compiling io-enum v1.0.1
   Compiling instant v0.1.11
   Compiling ahash v0.7.4
   Compiling generic-array v0.14.4
   Compiling standback v0.2.17
   Compiling nom v5.1.2
   Compiling time v0.2.27
   Compiling tracing-core v0.1.21
   Compiling lock_api v0.4.5
   Compiling cmake v0.1.46
   Compiling libloading v0.7.1
   Compiling ring v0.16.20
   Compiling oracle v0.5.3
   Compiling itertools v0.10.1
   Compiling openssl-sys v0.9.67
   Compiling libz-sys v1.1.3
   Compiling libsqlite3-sys v0.22.2
   Compiling tinyvec v1.5.0
   Compiling lexical-util v0.8.1
   Compiling form_urlencoded v1.0.1
   Compiling phf_shared v0.10.0
   Compiling foreign-types v0.3.2
   Compiling clang-sys v1.2.2
   Compiling http v0.2.5
   Compiling encoding-index-japanese v1.20141219.5
   Compiling encoding-index-singlebyte v1.20141219.5
   Compiling encoding-index-tradchinese v1.20141219.5
   Compiling encoding-index-korean v1.20141219.5
   Compiling encoding-index-simpchinese v1.20141219.5
   Compiling textwrap v0.11.0
   Compiling async-lock v2.4.0
   Compiling concurrent-queue v1.2.2
   Compiling wyz v0.4.0
   Compiling matrixmultiply v0.3.1
   Compiling owning_ref v0.4.1
   Compiling lexical-write-integer v0.8.0
   Compiling lexical-parse-integer v0.8.0
   Compiling unicode-normalization v0.1.19
   Compiling phf v0.10.0
   Compiling http-body v0.4.4
   Compiling encoding v0.2.33
   Compiling getrandom v0.2.3
   Compiling num_cpus v1.13.0
   Compiling time v0.1.44
   Compiling socket2 v0.4.2
   Compiling cpufeatures v0.2.1
   Compiling atty v0.2.14
   Compiling dirs v1.0.5
   Compiling dirs-sys v0.3.6
   Compiling memmap2 v0.5.0
   Compiling which v3.1.1
   Compiling subprocess v0.2.8
   Compiling quote v1.0.10
   Compiling aho-corasick v0.7.18
   Compiling csv-core v0.1.10
   Compiling crossbeam-channel v0.5.1
   Compiling mio v0.7.13
   Compiling want v0.3.0
   Compiling sqlparser v0.11.0
   Compiling async-channel v1.6.1
   Compiling paste-impl v0.1.18
   Compiling lexical-write-float v0.8.2
   Compiling lexical-parse-float v0.8.2
   Compiling stringprep v0.1.2
   Compiling idna v0.2.3
   Compiling pyo3 v0.14.5
   Compiling bitvec v0.22.3
   Compiling num-complex v0.4.0
   Compiling nix v0.23.0
   Compiling rand_core v0.6.3
   Compiling uuid v0.8.2
   Compiling parking_lot v0.11.2
   Compiling term v0.5.2
   Compiling dirs v4.0.0
   Compiling clap v2.33.3
   Compiling regex v1.5.4
   Compiling cexpr v0.4.0
   Compiling async-std v1.10.0
   Compiling paste v0.1.18
   Compiling sct v0.6.1
   Compiling webpki v0.21.4
   Compiling lexical v5.2.2
   Compiling digest v0.9.0
   Compiling block-buffer v0.9.0
   Compiling crypto-mac v0.11.1
   Compiling flate2 v1.0.22
   Compiling lexical-core v0.8.2
   Compiling url v2.2.2
   Compiling ndarray v0.15.3
   Compiling crossbeam-deque v0.8.1
   Compiling rand_chacha v0.3.1
   Compiling scheduled-thread-pool v0.2.5
   Compiling frunk_proc_macro_helpers v0.1.0
   Compiling derive_utils v0.11.2
   Compiling pyo3-macros-backend v0.14.5
   Compiling pem v0.8.3
   Compiling env_logger v0.9.0
   Compiling tokio-macros v1.4.1
   Compiling thiserror-impl v1.0.29
   Compiling tracing-attributes v0.1.18
   Compiling multiversion-macros v0.6.1
   Compiling time-macros-impl v0.1.2
   Compiling enumflags2_derive v0.7.0
   Compiling async-stream-impl v0.2.1
   Compiling indoc-impl v0.3.6
   Compiling fehler-macros v1.0.0
   Compiling dict_derive v0.4.0
   Compiling env_logger v0.8.4
   Compiling ct-logs v0.8.0
   Compiling rustls v0.19.1
   Compiling webpki-roots v0.21.1
   Compiling sha2 v0.9.8
   Compiling md-5 v0.9.1
   Compiling hmac v0.11.0
   Compiling lexical v6.0.1
   Compiling rand v0.8.4
   Compiling r2d2 v0.8.9
   Compiling frunk_proc_macros_impl v0.1.0
   Compiling frunk_derives v0.4.0
   Compiling pyo3-macros v0.14.5
   Compiling thiserror v1.0.29
   Compiling tracing v0.1.29
   Compiling time-macros v0.1.1
   Compiling multiversion v0.6.1
   Compiling async-stream v0.2.1
   Compiling enumflags2 v0.7.1
   Compiling indoc v0.3.6
   Compiling fehler v1.0.0
   Compiling rustls-native-certs v0.5.0
   Compiling num v0.4.0
   Compiling postgres-protocol v0.6.2
   Compiling twox-hash v1.6.1
   Compiling frunk_proc_macros v0.1.0
   Compiling futures-executor v0.3.17
   Compiling asynchronous-codec v0.5.0
   Compiling tokio-util v0.6.9
   Compiling tokio-rustls v0.22.0
   Compiling bb8 v0.7.1
   Compiling tokio-native-tls v0.3.0
   Compiling chrono v0.4.19
   Compiling bstr v0.2.17
   Compiling bigdecimal v0.2.2
   Compiling serde_urlencoded v0.7.0
   Compiling toml v0.5.8
   Compiling async-native-tls v0.3.3
   Compiling flatbuffers v2.0.0
   Compiling hashbrown v0.11.2
   Compiling mysql_common v0.27.5
   Compiling frunk v0.4.0
   Compiling futures v0.3.17
   Compiling csv v1.1.6
error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/batch.rs:206:56
    |
206 |                 dpiStmt_getBindNames(handle, &mut num, names.as_mut_ptr(), lengths.as_mut_ptr()),
    |                                                        ^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut *const u8`
               found raw pointer `*mut *const i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/batch.rs:213:45
    |
213 |                 bind_names.push(to_rust_str(names[i], lengths[i]));
    |                                             ^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_type/object.rs:199:45
    |
199 |                 dpiData_setBytes(&mut data, buf.as_mut_ptr(), buf.len() as u32);
    |                                             ^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_type/object.rs:402:45
    |
402 |                 dpiData_setBytes(&mut data, buf.as_mut_ptr(), buf.len() as u32);
    |                                             ^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_value.rs:532:63
    |
532 |                 dpiLob_readBytes(lob, offset, READ_CHAR_SIZE, bufptr, &mut read_len)
    |                                                               ^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_value.rs:555:17
    |
555 |                 result.as_mut_ptr() as *mut i8,
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_value.rs:582:58
    |
582 |                 dpiLob_readBytes(lob, offset, READ_SIZE, bufptr, &mut read_len)
    |                                                          ^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_value.rs:671:21
    |
671 |                     self.keep_bytes.as_mut_ptr() as *mut i8,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_value.rs:681:21
    |
681 |                     val.as_ptr() as *const i8,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_value.rs:750:56
    |
750 |         chkerr!(self.ctxt(), dpiLob_writeBytes(lob, 1, ptr, len));
    |                                                        ^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/sql_value.rs:762:56
    |
762 |         chkerr!(self.ctxt(), dpiLob_writeBytes(lob, 1, ptr, len));
    |                                                        ^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/statement.rs:477:56
    |
477 |                 dpiStmt_getBindNames(handle, &mut num, names.as_mut_ptr(), lengths.as_mut_ptr()),
    |                                                        ^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*mut *const u8`
               found raw pointer `*mut *const i8`

error[E0308]: mismatched types
   --> /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/oracle-0.5.3/src/statement.rs:484:45
    |
484 |                 bind_names.push(to_rust_str(names[i], lengths[i]));
    |                                             ^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`

   Compiling cargo-lock v7.0.1
For more information about this error, try `rustc --explain E0308`.
error: could not compile `oracle` due to 13 previous errors
warning: build failed, waiting for other jobs to finish...
error: build failed
error: Recipe `ci-build-python-extention` failed on line 84 with exit code 101
Matthieu-Tinycoaching commented 2 years ago

Solved by replacing i8 by u8 in the specified files.

But got a new error:

cd connectorx-python && cargo build --release
   Compiling oracle v0.5.3
   Compiling indexmap v1.7.0
   Compiling lru v0.6.6
   Compiling hashlink v0.7.0
   Compiling arrow2 v0.8.1
   Compiling prettytable-rs v0.8.0
   Compiling built v0.5.1
   Compiling serde_json v1.0.68
   Compiling h2 v0.3.7
   Compiling rusqlite v0.25.3
   Compiling r2d2-oracle v0.5.0
   Compiling polars-arrow v0.18.0
   Compiling connectorx-python v0.2.4-alpha.5 (/home/ec2-user/connector-x/connectorx-python)
   Compiling postgres-types v0.2.2
   Compiling arrow v5.5.0
error: could not compile `arrow2`

Caused by:
  process didn't exit successfully: `rustc --crate-name arrow2 --edition=2018 /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/arrow2-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C debuginfo=2 --cfg 'feature="compute_aggregate"' --cfg 'feature="compute_arithmetics"' --cfg 'feature="compute_boolean"' --cfg 'feature="compute_boolean_kleene"' --cfg 'feature="compute_cast"' --cfg 'feature="compute_comparison"' --cfg 'feature="compute_concatenate"' --cfg 'feature="compute_filter"' --cfg 'feature="compute_if_then_else"' --cfg 'feature="compute_take"' --cfg 'feature="csv"' --cfg 'feature="default"' --cfg 'feature="io_csv_write"' --cfg 'feature="lexical-core"' --cfg 'feature="multiversion"' --cfg 'feature="streaming-iterator"' --cfg 'feature="strength_reduce"' -C metadata=0ce4fc5666fa6848 -C extra-filename=-0ce4fc5666fa6848 --out-dir /home/ec2-user/connector-x/target/release/deps -L dependency=/home/ec2-user/connector-x/target/release/deps --extern chrono=/home/ec2-user/connector-x/target/release/deps/libchrono-0be9280ed612d29f.rmeta --extern csv=/home/ec2-user/connector-x/target/release/deps/libcsv-41640cd812f76c2f.rmeta --extern hash_hasher=/home/ec2-user/connector-x/target/release/deps/libhash_hasher-541cb7992cba6c29.rmeta --extern lexical_core=/home/ec2-user/connector-x/target/release/deps/liblexical_core-12e1efced66cc5c8.rmeta --extern multiversion=/home/ec2-user/connector-x/target/release/deps/libmultiversion-ee60dbaae3231ad9.rmeta --extern num_traits=/home/ec2-user/connector-x/target/release/deps/libnum_traits-02808b2bec604488.rmeta --extern simdutf8=/home/ec2-user/connector-x/target/release/deps/libsimdutf8-3241aa4ccefc32c1.rmeta --extern streaming_iterator=/home/ec2-user/connector-x/target/release/deps/libstreaming_iterator-24c9f3b3b1c826c9.rmeta --extern strength_reduce=/home/ec2-user/connector-x/target/release/deps/libstrength_reduce-196c1b0ce244914c.rmeta --cap-lints allow` (signal: 9, SIGKILL: kill)
warning: build failed, waiting for other jobs to finish...
error: build failed
error: Recipe `ci-build-python-extention` failed on line 84 with exit code 101
wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching , I'm not sure which rust version you are using. It should be the same with our release compilation environment: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L34, which is nightly-2021-10-21 for now.

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying I installed the nightly-2021-10-21 version of rust, as you can see below:

rustup show
Default host: aarch64-unknown-linux-gnu
rustup home:  /home/ec2-user/.rustup

installed toolchains
--------------------

stable-aarch64-unknown-linux-gnu (default)
nightly-2021-10-21-aarch64-unknown-linux-gnu
nightly-aarch64-unknown-linux-gnu

active toolchain
----------------

nightly-2021-10-21-aarch64-unknown-linux-gnu (directory override for '/home/ec2-user')
rustc 1.58.0-nightly (efd048394 2021-10-20)

However, when running just ci-build-python-extention ci-build-python-wheel ci-rename-wheel I still got an error for compiling arrow library:

cd connectorx-python && cargo build --release
   Compiling autocfg v1.0.1
   Compiling libc v0.2.103
   Compiling proc-macro2 v1.0.29
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.78
   Compiling version_check v0.9.3
   Compiling cfg-if v1.0.0
   Compiling memchr v2.4.1
   Compiling crossbeam-utils v0.8.5
   Compiling serde v1.0.130
   Compiling serde_derive v1.0.130
   Compiling log v0.4.14
   Compiling crossbeam-epoch v0.9.5
   Compiling lazy_static v1.4.0
   Compiling scopeguard v1.1.0
   Compiling rayon-core v1.9.1
   Compiling cc v1.0.70
   Compiling once_cell v1.8.0
   Compiling proc-macro-hack v0.5.19
   Compiling futures-core v0.3.17
   Compiling parking_lot_core v0.8.5
   Compiling ryu v1.0.5
   Compiling typenum v1.14.0
   Compiling futures-channel v0.3.17
   Compiling proc-macro-nested v0.1.7
   Compiling futures-task v0.3.17
   Compiling smallvec v1.7.0
   Compiling either v1.6.1
   Compiling itoa v0.4.8
   Compiling pin-project-lite v0.2.7
   Compiling bytes v1.1.0
   Compiling serde_json v1.0.68
   Compiling futures-io v0.3.17
   Compiling pkg-config v0.3.20
   Compiling slab v0.4.4
   Compiling pin-utils v0.1.0
   Compiling futures-sink v0.3.17
   Compiling tinyvec_macros v0.1.0
   Compiling base64 v0.13.0
   Compiling static_assertions v1.1.0
   Compiling async-trait v0.1.51
   Compiling unicode-bidi v0.3.6
   Compiling byteorder v1.4.3
   Compiling bitflags v1.3.2
   Compiling percent-encoding v2.1.0
   Compiling ppv-lite86 v0.2.10
   Compiling subtle v2.4.1
   Compiling fallible-iterator v0.2.0
   Compiling opaque-debug v0.3.0
   Compiling openssl-probe v0.1.4
   Compiling untrusted v0.7.1
   Compiling siphasher v0.3.7
   Compiling spin v0.5.2
   Compiling matches v0.1.9
   Compiling md5 v0.7.0
   Compiling regex-syntax v0.6.25
   Compiling foreign-types-shared v0.1.1
   Compiling regex-automata v0.1.10
   Compiling glob v0.3.0
   Compiling openssl v0.10.36
   Compiling arrayvec v0.5.2
   Compiling native-tls v0.2.8
   Compiling fnv v1.0.7
   Compiling anyhow v1.0.44
   Compiling encoding_index_tests v0.1.4
   Compiling frunk_core v0.4.0
   Compiling unicode-width v0.1.9
   Compiling strsim v0.8.0
   Compiling httparse v1.5.1
   Compiling lexical-core v0.7.6
   Compiling ansi_term v0.11.0
   Compiling termcolor v1.1.2
   Compiling humantime v2.1.0
   Compiling pyo3-build-config v0.14.5
   Compiling bindgen v0.58.1
   Compiling vec_map v0.8.2
   Compiling radium v0.6.2
   Compiling const_fn v0.4.8
   Compiling crc32fast v1.2.1
   Compiling shlex v1.1.0
   Compiling strength_reduce v0.2.3
   Compiling simdutf8 v0.1.3
   Compiling try-lock v0.2.3
   Compiling peeking_take_while v0.1.2
   Compiling cache-padded v1.1.1
   Compiling event-listener v2.5.1
   Compiling lazycell v1.3.0
   Compiling streaming-iterator v0.1.5
   Compiling hash_hasher v2.0.3
   Compiling rustc-hash v1.1.0
   Compiling encode_unicode v0.3.6
   Compiling tap v1.0.1
   Compiling httpdate v1.0.2
   Compiling semver v1.0.4
   Compiling tower-service v0.3.1
   Compiling unsafe_unwrap v0.1.0
   Compiling funty v1.2.0
   Compiling encoding_rs v0.8.29
   Compiling saturating v0.1.0
   Compiling mysql v21.0.2
   Compiling sha1 v0.6.0
   Compiling unindent v0.1.7
   Compiling fallible-streaming-iterator v0.1.9
   Compiling rawpointer v0.2.1
   Compiling mime v0.3.16
   Compiling paste v1.0.5
   Compiling seahash v4.1.0
   Compiling ipnet v2.3.1
   Compiling connection-string v0.1.13
   Compiling bufstream v0.1.4
   Compiling pretty-hex v0.1.1
   Compiling hex v0.4.3
   Compiling stable_deref_trait v1.2.0
   Compiling cfg-if v0.1.10
   Compiling urlencoding v2.1.0
   Compiling pyo3-built v0.4.6
   Compiling bitfield v0.13.2
   Compiling memoffset v0.6.4
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling rayon v1.5.1
   Compiling futures-util v0.3.17
   Compiling futures-macro v0.3.17
   Compiling indexmap v1.7.0
   Compiling tokio v1.12.0
   Compiling num-bigint v0.4.2
   Compiling num-rational v0.4.0
   Compiling num-iter v0.1.42
   Compiling num-bigint v0.3.3
   Compiling io-enum v1.0.1
   Compiling instant v0.1.11
   Compiling ahash v0.7.4
   Compiling generic-array v0.14.4
   Compiling standback v0.2.17
   Compiling nom v5.1.2
   Compiling time v0.2.27
   Compiling tracing-core v0.1.21
   Compiling lock_api v0.4.5
   Compiling cmake v0.1.46
   Compiling libloading v0.7.1
   Compiling ring v0.16.20
   Compiling oracle v0.5.3
   Compiling itertools v0.10.1
   Compiling openssl-sys v0.9.67
   Compiling libz-sys v1.1.3
   Compiling libsqlite3-sys v0.22.2
   Compiling tinyvec v1.5.0
   Compiling lexical-util v0.8.1
   Compiling phf_shared v0.10.0
   Compiling form_urlencoded v1.0.1
   Compiling foreign-types v0.3.2
   Compiling clang-sys v1.2.2
   Compiling http v0.2.5
   Compiling encoding-index-korean v1.20141219.5
   Compiling encoding-index-simpchinese v1.20141219.5
   Compiling encoding-index-japanese v1.20141219.5
   Compiling encoding-index-tradchinese v1.20141219.5
   Compiling encoding-index-singlebyte v1.20141219.5
   Compiling textwrap v0.11.0
   Compiling concurrent-queue v1.2.2
   Compiling async-lock v2.4.0
   Compiling wyz v0.4.0
   Compiling matrixmultiply v0.3.1
   Compiling owning_ref v0.4.1
   Compiling lexical-parse-integer v0.8.0
   Compiling lexical-write-integer v0.8.0
   Compiling unicode-normalization v0.1.19
   Compiling phf v0.10.0
   Compiling http-body v0.4.4
   Compiling encoding v0.2.33
   Compiling getrandom v0.2.3
   Compiling num_cpus v1.13.0
   Compiling time v0.1.44
   Compiling socket2 v0.4.2
   Compiling cpufeatures v0.2.1
   Compiling atty v0.2.14
   Compiling dirs v1.0.5
   Compiling dirs-sys v0.3.6
   Compiling memmap2 v0.5.0
   Compiling which v3.1.1
   Compiling subprocess v0.2.8
   Compiling quote v1.0.10
   Compiling aho-corasick v0.7.18
   Compiling csv-core v0.1.10
   Compiling crossbeam-channel v0.5.1
   Compiling mio v0.7.13
   Compiling want v0.3.0
   Compiling sqlparser v0.11.0
   Compiling async-channel v1.6.1
   Compiling paste-impl v0.1.18
   Compiling lexical-parse-float v0.8.2
   Compiling lexical-write-float v0.8.2
   Compiling stringprep v0.1.2
   Compiling idna v0.2.3
   Compiling pyo3 v0.14.5
   Compiling bitvec v0.22.3
   Compiling num-complex v0.4.0
   Compiling nix v0.23.0
   Compiling rand_core v0.6.3
   Compiling uuid v0.8.2
   Compiling parking_lot v0.11.2
   Compiling term v0.5.2
   Compiling dirs v4.0.0
   Compiling clap v2.33.3
   Compiling regex v1.5.4
   Compiling cexpr v0.4.0
   Compiling async-std v1.10.0
   Compiling paste v0.1.18
   Compiling webpki v0.21.4
   Compiling sct v0.6.1
   Compiling lexical v5.2.2
   Compiling digest v0.9.0
   Compiling block-buffer v0.9.0
   Compiling crypto-mac v0.11.1
   Compiling flate2 v1.0.22
   Compiling lexical-core v0.8.2
   Compiling url v2.2.2
   Compiling ndarray v0.15.3
   Compiling crossbeam-deque v0.8.1
   Compiling rand_chacha v0.3.1
   Compiling scheduled-thread-pool v0.2.5
   Compiling frunk_proc_macro_helpers v0.1.0
   Compiling derive_utils v0.11.2
   Compiling pyo3-macros-backend v0.14.5
   Compiling pem v0.8.3
   Compiling env_logger v0.9.0
   Compiling tokio-macros v1.4.1
   Compiling thiserror-impl v1.0.29
   Compiling tracing-attributes v0.1.18
   Compiling multiversion-macros v0.6.1
   Compiling time-macros-impl v0.1.2
   Compiling enumflags2_derive v0.7.0
   Compiling async-stream-impl v0.2.1
   Compiling indoc-impl v0.3.6
   Compiling fehler-macros v1.0.0
   Compiling dict_derive v0.4.0
   Compiling env_logger v0.8.4
   Compiling webpki-roots v0.21.1
   Compiling rustls v0.19.1
   Compiling ct-logs v0.8.0
   Compiling md-5 v0.9.1
   Compiling sha2 v0.9.8
   Compiling hmac v0.11.0
   Compiling lexical v6.0.1
   Compiling rand v0.8.4
   Compiling r2d2 v0.8.9
   Compiling frunk_proc_macros_impl v0.1.0
   Compiling frunk_derives v0.4.0
   Compiling pyo3-macros v0.14.5
   Compiling thiserror v1.0.29
   Compiling tracing v0.1.29
   Compiling time-macros v0.1.1
   Compiling multiversion v0.6.1
   Compiling async-stream v0.2.1
   Compiling enumflags2 v0.7.1
   Compiling indoc v0.3.6
   Compiling fehler v1.0.0
   Compiling rustls-native-certs v0.5.0
   Compiling num v0.4.0
   Compiling postgres-protocol v0.6.2
   Compiling twox-hash v1.6.1
   Compiling frunk_proc_macros v0.1.0
   Compiling futures-executor v0.3.17
   Compiling asynchronous-codec v0.5.0
   Compiling tokio-util v0.6.9
   Compiling tokio-rustls v0.22.0
   Compiling bb8 v0.7.1
   Compiling tokio-native-tls v0.3.0
   Compiling chrono v0.4.19
   Compiling bstr v0.2.17
   Compiling bigdecimal v0.2.2
   Compiling serde_urlencoded v0.7.0
   Compiling toml v0.5.8
   Compiling async-native-tls v0.3.3
   Compiling flatbuffers v2.0.0
   Compiling hashbrown v0.11.2
   Compiling mysql_common v0.27.5
   Compiling frunk v0.4.0
   Compiling futures v0.3.17
   Compiling csv v1.1.6
   Compiling cargo-lock v7.0.1
   Compiling numpy v0.14.1
   Compiling lru v0.6.6
   Compiling hashlink v0.7.0
   Compiling arrow2 v0.8.1
   Compiling prettytable-rs v0.8.0
   Compiling r2d2-oracle v0.5.0
   Compiling built v0.5.1
   Compiling h2 v0.3.7
   Compiling rusqlite v0.25.3
   Compiling polars-arrow v0.18.0
   Compiling connectorx-python v0.2.4-alpha.5 (/home/ec2-user/connector-x/connectorx-python)
   Compiling postgres-types v0.2.2
   Compiling arrow v5.5.0
error: could not compile `arrow2`

Caused by:
  process didn't exit successfully: `rustc --crate-name arrow2 --edition=2018 /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/arrow2-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C debuginfo=2 --cfg 'feature="compute_aggregate"' --cfg 'feature="compute_arithmetics"' --cfg 'feature="compute_boolean"' --cfg 'feature="compute_boolean_kleene"' --cfg 'feature="compute_cast"' --cfg 'feature="compute_comparison"' --cfg 'feature="compute_concatenate"' --cfg 'feature="compute_filter"' --cfg 'feature="compute_if_then_else"' --cfg 'feature="compute_take"' --cfg 'feature="csv"' --cfg 'feature="default"' --cfg 'feature="io_csv_write"' --cfg 'feature="lexical-core"' --cfg 'feature="multiversion"' --cfg 'feature="streaming-iterator"' --cfg 'feature="strength_reduce"' -C metadata=57dabe023757d5ca -C extra-filename=-57dabe023757d5ca --out-dir /home/ec2-user/connector-x/target/release/deps -L dependency=/home/ec2-user/connector-x/target/release/deps --extern chrono=/home/ec2-user/connector-x/target/release/deps/libchrono-28cd2ec861b5817f.rmeta --extern csv=/home/ec2-user/connector-x/target/release/deps/libcsv-b81a67da07cc86c3.rmeta --extern hash_hasher=/home/ec2-user/connector-x/target/release/deps/libhash_hasher-541cb7992cba6c29.rmeta --extern lexical_core=/home/ec2-user/connector-x/target/release/deps/liblexical_core-12e1efced66cc5c8.rmeta --extern multiversion=/home/ec2-user/connector-x/target/release/deps/libmultiversion-336c9be7db008d25.rmeta --extern num_traits=/home/ec2-user/connector-x/target/release/deps/libnum_traits-91ca3c5c425d8941.rmeta --extern simdutf8=/home/ec2-user/connector-x/target/release/deps/libsimdutf8-3241aa4ccefc32c1.rmeta --extern streaming_iterator=/home/ec2-user/connector-x/target/release/deps/libstreaming_iterator-24c9f3b3b1c826c9.rmeta --extern strength_reduce=/home/ec2-user/connector-x/target/release/deps/libstrength_reduce-196c1b0ce244914c.rmeta --cap-lints allow` (signal: 9, SIGKILL: kill)
Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying I managed to pass the error for the arrow library compilation by choosing a graviton EC2 instance with 2 Go of RAM instaed of one.

However, now I still got an error with the polars-core compilation:

cd connectorx-python && cargo build --release
   Compiling arrow2 v0.8.1
   Compiling connectorx-python v0.2.4-alpha.5 (/home/ec2-user/connector-x/connectorx-python)
   Compiling postgres-types v0.2.2
   Compiling arrow v5.5.0
   Compiling hyper v0.14.15
   Compiling r2d2_sqlite v0.18.0
   Compiling tokio-postgres v0.7.3
   Compiling polars-arrow v0.18.0
   Compiling hyper-rustls v0.22.1
   Compiling postgres v0.19.2
   Compiling postgres-native-tls v0.5.0
   Compiling polars-core v0.18.0
   Compiling yup-oauth2 v6.0.0
   Compiling reqwest v0.11.6
   Compiling rust_decimal v1.16.0
   Compiling r2d2_postgres v0.18.1
   Compiling polars-io v0.18.0
   Compiling gcp-bigquery-client v0.10.0
   Compiling mysql_common v0.27.5
   Compiling tiberius v0.5.16
   Compiling polars-lazy v0.18.0
   Compiling mysql v21.0.2
   Compiling bb8-tiberius v0.5.2
   Compiling rust_decimal_macros v1.16.0
   Compiling polars v0.18.0
   Compiling r2d2_mysql v21.0.0
error: could not compile `polars-core`

Caused by:
  process didn't exit successfully: `rustc --crate-name polars_core --edition=2018 /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/polars-core-0.18.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C debuginfo=2 --cfg 'feature="chrono"' --cfg 'feature="docs"' --cfg 'feature="dtype-date"' --cfg 'feature="dtype-datetime"' --cfg 'feature="lazy"' --cfg 'feature="performant"' --cfg 'feature="plain_fmt"' --cfg 'feature="prettytable-rs"' --cfg 'feature="private"' --cfg 'feature="regex"' --cfg 'feature="sort_multiple"' --cfg 'feature="temporal"' --cfg 'feature="zip_with"' -C metadata=9fbfc466be5e6aa5 -C extra-filename=-9fbfc466be5e6aa5 --out-dir /home/ec2-user/connector-x/target/release/deps -L dependency=/home/ec2-user/connector-x/target/release/deps --extern ahash=/home/ec2-user/connector-x/target/release/deps/libahash-3bb4d85c1351bac9.rmeta --extern anyhow=/home/ec2-user/connector-x/target/release/deps/libanyhow-b61f27d4e6c4d440.rmeta --extern arrow=/home/ec2-user/connector-x/target/release/deps/libarrow2-57dabe023757d5ca.rmeta --extern chrono=/home/ec2-user/connector-x/target/release/deps/libchrono-28cd2ec861b5817f.rmeta --extern hashbrown=/home/ec2-user/connector-x/target/release/deps/libhashbrown-f5cf62f626701e9d.rmeta --extern itertools=/home/ec2-user/connector-x/target/release/deps/libitertools-7335e749d4fb965c.rmeta --extern lazy_static=/home/ec2-user/connector-x/target/release/deps/liblazy_static-ea45d35ae4175dd8.rmeta --extern num=/home/ec2-user/connector-x/target/release/deps/libnum-ab93ffce627aa183.rmeta --extern num_cpus=/home/ec2-user/connector-x/target/release/deps/libnum_cpus-7dfc8331ed283a11.rmeta --extern polars_arrow=/home/ec2-user/connector-x/target/release/deps/libpolars_arrow-bc34d0dbf11dc3b2.rmeta --extern prettytable=/home/ec2-user/connector-x/target/release/deps/libprettytable-8aacd66bdb309414.rmeta --extern rayon=/home/ec2-user/connector-x/target/release/deps/librayon-bc3df8c9d0d9019b.rmeta --extern regex=/home/ec2-user/connector-x/target/release/deps/libregex-fc84cf4f56463aca.rmeta --extern thiserror=/home/ec2-user/connector-x/target/release/deps/libthiserror-79aaef3755e875ee.rmeta --extern unsafe_unwrap=/home/ec2-user/connector-x/target/release/deps/libunsafe_unwrap-e8ee60c5e32bc922.rmeta --cap-lints allow` (signal: 9, SIGKILL: kill)
error: Recipe `ci-build-python-extention` failed on line 94 with exit code 101

Could this still be due to a lack of RAM?

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying I managed to build it all with a 4Go RAM graviton AWs instance.

One strange thing when I tested it on arm64 instance was that loading a small DB table with ConnectorX was slower than pandas (360 vs. 70 ms, respectively). Could you help me understand why?

wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching , great that it successfully compiled.

One strange thing when I tested it on arm64 instance was that loading a small DB table with ConnectorX was slower than pandas (360 vs. 70 ms, respectively). Could you help me understand why?

May I ask which database are you using and what does the query look like? Also it would be great if you could show the log so we can see which step is consuming the time. You can set the RUST_LOG environment variable before importing the library like the following example:

import os
os.environ["RUST_LOG"]="connectorx=debug,connectorx_python=debug"
import connectorx as cx

df = cx.read_sql(conn_url, query, return_type="pandas")
lonestriker commented 2 years ago

I'm running into Rust compile errors when trying to compile the latest release branch. I'm using brew for most of the dependent packages and python.

Below are my settings. Any pointers would be appreciated. Thanks for the fantastic software!

brew packages

$ brew list
==> Formulae
autoconf    freetds     git-lfs     gmp     libevent    libpq       lz4     msodbcsql17 openssl@1.1 protobuf    python@3.10 six     xz
automake    gcc     gitlab-runner   icu4c       libffi      libtool     m4      mssql-tools openssl@3   putty       python@3.9  sqlite      zstd
ca-certificates gdbm        glib        isl     libidn2     libunistring    mpdecimal   mysql       pcre        pyenv       qt@5        unixodbc
cmake       gettext     glib-openssl    krb5        libmpc      llvm        mpfr        openblas    pkg-config  pyqt@5      readline    wget

git branch

$ git status
On branch release
Your branch is up to date with 'origin/release'.

nothing to commit, working tree clean

rust nightly

$ rustup show
Default host: aarch64-apple-darwin
rustup home:  /Users/user/.rustup

installed toolchains
--------------------

stable-aarch64-apple-darwin (default)
nightly-aarch64-apple-darwin

active toolchain
----------------

nightly-aarch64-apple-darwin (directory override for '/Users/user/work/connector-x')
rustc 1.60.0-nightly (ad46af247 2022-01-14)

Build log

Log for just the failed section. It compiles all steps fine up until the last connector-x component.

cd connectorx-python && cargo build --release
   Compiling connectorx-python v0.2.4-alpha.4 (/Users/user/work/connector-x/connectorx-python)
error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/bigquery.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = BigQueryPandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = BigQueryTypeSystem => PandasTypeSystem,
...  |
33 | |     }
34 | | );
   | |_^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/bigquery.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = BigQueryPandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = BigQueryTypeSystem => PandasTypeSystem,
...  |
33 | |     }
34 | | );
   | |_^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/bigquery.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = BigQueryPandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = BigQueryTypeSystem => PandasTypeSystem,
...  |
33 | |     }
34 | | );
   | |_^
   = note: expected `<BigQueryPandasTransport<'tp> as Transport>`
              found `<BigQueryPandasTransport<'_> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/bigquery.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = BigQueryPandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = BigQueryTypeSystem => PandasTypeSystem,
...  |
33 | |     }
34 | | );
   | |_^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/bigquery.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = BigQueryPandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = BigQueryTypeSystem => PandasTypeSystem,
...  |
33 | |     }
34 | | );
   | |_^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/mssql.rs:14:1
   |
14 | / impl_transport!(
15 | |     name = MsSQLPandasTransport<'tp>,
16 | |     error = ConnectorXPythonError,
17 | |     systems = MsSQLTypeSystem => PandasTypeSystem,
...  |
49 | |     }
50 | | );
   | |_^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/mssql.rs:14:1
   |
14 | / impl_transport!(
15 | |     name = MsSQLPandasTransport<'tp>,
16 | |     error = ConnectorXPythonError,
17 | |     systems = MsSQLTypeSystem => PandasTypeSystem,
...  |
49 | |     }
50 | | );
   | |_^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/mssql.rs:14:1
   |
14 | / impl_transport!(
15 | |     name = MsSQLPandasTransport<'tp>,
16 | |     error = ConnectorXPythonError,
17 | |     systems = MsSQLTypeSystem => PandasTypeSystem,
...  |
49 | |     }
50 | | );
   | |_^
   = note: expected `<MsSQLPandasTransport<'tp> as Transport>`
              found `<MsSQLPandasTransport<'_> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/mssql.rs:14:1
   |
14 | / impl_transport!(
15 | |     name = MsSQLPandasTransport<'tp>,
16 | |     error = ConnectorXPythonError,
17 | |     systems = MsSQLTypeSystem => PandasTypeSystem,
...  |
49 | |     }
50 | | );
   | |_^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/mssql.rs:14:1
   |
14 | / impl_transport!(
15 | |     name = MsSQLPandasTransport<'tp>,
16 | |     error = ConnectorXPythonError,
17 | |     systems = MsSQLTypeSystem => PandasTypeSystem,
...  |
49 | |     }
50 | | );
   | |_^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/mysql.rs:16:1
   |
16 | / impl_transport!(
17 | |     name = MysqlPandasTransport<'tp, BinaryProtocol>,
18 | |     error = ConnectorXPythonError,
19 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
48 | |     }
49 | | );
   | |_^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/mysql.rs:16:1
   |
16 | / impl_transport!(
17 | |     name = MysqlPandasTransport<'tp, BinaryProtocol>,
18 | |     error = ConnectorXPythonError,
19 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
48 | |     }
49 | | );
   | |_^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/mysql.rs:16:1
   |
16 | / impl_transport!(
17 | |     name = MysqlPandasTransport<'tp, BinaryProtocol>,
18 | |     error = ConnectorXPythonError,
19 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
48 | |     }
49 | | );
   | |_^
   = note: expected `<MysqlPandasTransport<'tp, connectorx::sources::mysql::BinaryProtocol> as Transport>`
              found `<MysqlPandasTransport<'_, connectorx::sources::mysql::BinaryProtocol> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/mysql.rs:16:1
   |
16 | / impl_transport!(
17 | |     name = MysqlPandasTransport<'tp, BinaryProtocol>,
18 | |     error = ConnectorXPythonError,
19 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
48 | |     }
49 | | );
   | |_^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/mysql.rs:16:1
   |
16 | / impl_transport!(
17 | |     name = MysqlPandasTransport<'tp, BinaryProtocol>,
18 | |     error = ConnectorXPythonError,
19 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
48 | |     }
49 | | );
   | |_^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/mysql.rs:51:1
   |
51 | / impl_transport!(
52 | |     name = MysqlPandasTransport<'tp, TextProtocol>,
53 | |     error = ConnectorXPythonError,
54 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
83 | |     }
84 | | );
   | |_^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/mysql.rs:51:1
   |
51 | / impl_transport!(
52 | |     name = MysqlPandasTransport<'tp, TextProtocol>,
53 | |     error = ConnectorXPythonError,
54 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
83 | |     }
84 | | );
   | |_^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/mysql.rs:51:1
   |
51 | / impl_transport!(
52 | |     name = MysqlPandasTransport<'tp, TextProtocol>,
53 | |     error = ConnectorXPythonError,
54 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
83 | |     }
84 | | );
   | |_^
   = note: expected `<MysqlPandasTransport<'tp, TextProtocol> as Transport>`
              found `<MysqlPandasTransport<'_, TextProtocol> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/mysql.rs:51:1
   |
51 | / impl_transport!(
52 | |     name = MysqlPandasTransport<'tp, TextProtocol>,
53 | |     error = ConnectorXPythonError,
54 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
83 | |     }
84 | | );
   | |_^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/mysql.rs:51:1
   |
51 | / impl_transport!(
52 | |     name = MysqlPandasTransport<'tp, TextProtocol>,
53 | |     error = ConnectorXPythonError,
54 | |     systems = MySQLTypeSystem => PandasTypeSystem,
...  |
83 | |     }
84 | | );
   | |_^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/oracle.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = OraclePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = OracleTypeSystem => PandasTypeSystem,
...  |
31 | |     }
32 | | );
   | |_^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/oracle.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = OraclePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = OracleTypeSystem => PandasTypeSystem,
...  |
31 | |     }
32 | | );
   | |_^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/oracle.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = OraclePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = OracleTypeSystem => PandasTypeSystem,
...  |
31 | |     }
32 | | );
   | |_^
   = note: expected `<OraclePandasTransport<'tp> as Transport>`
              found `<OraclePandasTransport<'_> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/oracle.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = OraclePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = OracleTypeSystem => PandasTypeSystem,
...  |
31 | |     }
32 | | );
   | |_^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/oracle.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = OraclePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = OracleTypeSystem => PandasTypeSystem,
...  |
31 | |     }
32 | | );
   | |_^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/postgres.rs:61:1
   |
61 | impl_postgres_transport!(BinaryProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:61:1
   |
61 | impl_postgres_transport!(BinaryProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:61:1
   |
61 | impl_postgres_transport!(BinaryProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, connectorx::sources::postgres::BinaryProtocol, NoTls> as Transport>`
              found `<PostgresPandasTransport<'_, connectorx::sources::postgres::BinaryProtocol, NoTls> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:61:1
   |
61 | impl_postgres_transport!(BinaryProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:61:1
   |
61 | impl_postgres_transport!(BinaryProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/postgres.rs:62:1
   |
62 | impl_postgres_transport!(BinaryProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:62:1
   |
62 | impl_postgres_transport!(BinaryProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:62:1
   |
62 | impl_postgres_transport!(BinaryProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, connectorx::sources::postgres::BinaryProtocol, MakeTlsConnector> as Transport>`
              found `<PostgresPandasTransport<'_, connectorx::sources::postgres::BinaryProtocol, MakeTlsConnector> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:62:1
   |
62 | impl_postgres_transport!(BinaryProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:62:1
   |
62 | impl_postgres_transport!(BinaryProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/postgres.rs:63:1
   |
63 | impl_postgres_transport!(CSVProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:63:1
   |
63 | impl_postgres_transport!(CSVProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:63:1
   |
63 | impl_postgres_transport!(CSVProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, CSVProtocol, NoTls> as Transport>`
              found `<PostgresPandasTransport<'_, CSVProtocol, NoTls> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:63:1
   |
63 | impl_postgres_transport!(CSVProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:63:1
   |
63 | impl_postgres_transport!(CSVProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/postgres.rs:64:1
   |
64 | impl_postgres_transport!(CSVProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:64:1
   |
64 | impl_postgres_transport!(CSVProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:64:1
   |
64 | impl_postgres_transport!(CSVProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, CSVProtocol, MakeTlsConnector> as Transport>`
              found `<PostgresPandasTransport<'_, CSVProtocol, MakeTlsConnector> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:64:1
   |
64 | impl_postgres_transport!(CSVProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:64:1
   |
64 | impl_postgres_transport!(CSVProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, CursorProtocol, NoTls> as Transport>`
              found `<PostgresPandasTransport<'_, CursorProtocol, NoTls> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, CursorProtocol, MakeTlsConnector> as Transport>`
              found `<PostgresPandasTransport<'_, CursorProtocol, MakeTlsConnector> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
   = note: expected `<SqlitePandasTransport<'tp> as Transport>`
              found `<SqlitePandasTransport<'_> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0495`.
error: could not compile `connectorx-python` due to 24 previous errors
error: Recipe `ci-build-python-extention` failed on line 83 with exit code 101
wangxiaoying commented 2 years ago

@lonestriker can you try this nightly version: https://github.com/sfu-db/connector-x/issues/186#issuecomment-1011311830 ?

lonestriker commented 2 years ago

@lonestriker can you try this nightly version: #186 (comment) ?

Perfect, that fixed up the build issue on my M1 mac!

Edit: Looks like the build generates the wrong filename though for the distribution. It's listing it as 'intel':

connectorx-0.2.4a5-cp39-cp39-macosx_10_15_intel.whl

but when I check the generated library but it's arm64:

$ file connectorx_python.cpython-39-darwin.so
connectorx_python.cpython-39-darwin.so: Mach-O 64-bit dynamically linked shared library arm64

Steps to build for those who also hit the same issue:

Fresh clone of source

$ git clone https://github.com/sfu-db/connector-x.git

Install nightly-2021-10-21

$ rustup install nightly-2021-10-21
info: syncing channel updates for 'nightly-2021-10-21-aarch64-apple-darwin'

  nightly-2021-10-21-aarch64-apple-darwin unchanged - rustc 1.58.0-nightly (efd048394 2021-10-20)

info: checking for self-updates

Override default project toolchain

$ rustup default nightly-2021-10-21
$ rustup override set nightly-2021-10-21
$ rustup show
Default host: aarch64-apple-darwin
rustup home:  /Users/user/.rustup

installed toolchains
--------------------

stable-aarch64-apple-darwin
nightly-2021-10-21-aarch64-apple-darwin (default)
nightly-aarch64-apple-darwin

active toolchain
----------------

nightly-2021-10-21-aarch64-apple-darwin (default)
rustc 1.58.0-nightly (efd048394 2021-10-20)

Build

$ just bootstrap-python
$ just ci-build-python-extention ci-build-python-wheel ci-rename-wheel
wangxiaoying commented 2 years ago

Hi @lonestriker , thanks so much for sharing the building procedure!

Looks like the build generates the wrong filename though for the distribution.

You are right. We build and publish the release using github-hosted environment. So our script is hard coded for the macOS environment it has. You may need to manually change the file name for now. Sorry for the inconvenience.

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying

May I ask which database are you using and what does the query look like?

I use a postgreSQL database on AWS and the query looks like:

query = f"""
SELECT  id,
        id_user,
        id_timezone,
        timestamp,
        code_content,
        id_box
FROM {tableName}
WHERE id_user = {id_user}
    AND id_box < {num_max_leitner_decks}
ORDER BY id ASC
"""

Also it would be great if you could show the log so we can see which step is consuming the time. You can set the RUST_LOG environment variable before importing the library like the following example:

import os
os.environ["RUST_LOG"]="connectorx=debug,connectorx_python=debug"
import connectorx as cx

df = cx.read_sql(conn_url, query, return_type="pandas")

Please find below the log:

[2022-01-17T09:49:47Z DEBUG connectorx_python::pandas] Protocol: binary
[2022-01-17T09:49:47Z DEBUG connectorx_python::pandas] Running dispatcher
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Fetching metadata
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Try get row rounts for entire result
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Manually count rows of each partitioned query and sum up
[2022-01-17T09:49:47Z DEBUG connectorx::sql] Transformed count query: SELECT count(*) FROM (SELECT id, id_user, id_timezone, timestamp, code_content, id_box FROM leitner_decks_pprod WHERE id_user = 1 AND id_box < 8) AS CXTMPTAB_COUNT
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Allocate destination memory: 9x6
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Create destination partition
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Start writing
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Finalize partition 0
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Partition 0 finished
[2022-01-17T09:49:47Z DEBUG connectorx::dispatcher] Writing finished
wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching , thanks for the log.

Looks like your query is very small (only 9 rows). For pandas, ConnectorX will issue a COUNT query (Transformed count query:) prior to the actual input query (Start writing) in order to preallocate the pandas dataframe (Allocate destination memory:). Doing this makes writing the result directly into the final destination possible after getting each value from database, which could reduce the number of copying.

This overhead of this extra COUNT query can be ignored when the query result is relatively large. But when the result is very small as in your test case, its overhead becomes obvious.

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying thanks for the detailed explanation.

Could you estimate from which size of query result would ConnectorX be better than pandas?

wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching it will be affected by factors such as network condition so I cannot give you an estimation.

Another thing is we also provide the "arrow" destination, and arrow provides this to_pandas api to efficiently convert arrow to pandas. Since arrow does not require continuous memory buffer, we do not issue the COUNT query for it. You can try whether it can reach the speed you want. Here is an example:

import connectorx as cx

table = cx.read_sql(conn, query, return_type="arrow")
df = table.to_pandas(split_blocks=False, date_as_object=False)
Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying I tried what you suggested with arrow:

os.environ["RUST_BACKTRACE"]="1"
os.environ["RUST_LOG"]="connectorx=debug,connectorx_python=debug"
import connectorx as cx

table = cx.read_sql(postgres_url, query, return_type="arrow")
df_ld_items = table.to_pandas(split_blocks=False, date_as_object=False)

But got the following error:

[2022-01-20T11:01:51Z DEBUG connectorx_python::arrow] Protocol: binary
[2022-01-20T11:01:51Z DEBUG connectorx_python::arrow] Running dispatcher
[2022-01-20T11:01:51Z DEBUG connectorx::dispatcher] Fetching metadata
[2022-01-20T11:01:51Z DEBUG connectorx::dispatcher] Do not need counts in advance
[2022-01-20T11:01:51Z DEBUG connectorx::dispatcher] Allocate destination memory: 0x6
thread '<unnamed>' panicked at 'not implemented', connectorx/src/destinations/arrow/arrow_assoc.rs:170:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/std/src/panicking.rs:495:5
   1: core::panicking::panic_fmt
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/panicking.rs:107:14
   2: core::panicking::panic
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/panicking.rs:50:5
   3: <core::option::Option<chrono::datetime::DateTime<chrono::offset::utc::Utc>> as connectorx::destinations::arrow::arrow_assoc::ArrowAssoc>::field
             at ./connector-x/connectorx/src/destinations/arrow/arrow_assoc.rs:170:9
   4: <connectorx::destinations::arrow::ArrowDestination as connectorx::destinations::Destination>::allocate::{{closure}}
             at ./connector-x/connectorx/src/destinations/arrow/mod.rs:84:32
   5: core::iter::adapters::map::map_try_fold::{{closure}}
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/adapters/map.rs:91:28
   6: core::iter::traits::iterator::Iterator::try_fold
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/traits/iterator.rs:1995:21
   7: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/adapters/map.rs:117:9
   8: <core::iter::adapters::ResultShunt<I,E> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/adapters/mod.rs:178:9
   9: core::iter::traits::iterator::Iterator::find
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/traits/iterator.rs:2383:9
  10: <core::iter::adapters::ResultShunt<I,E> as core::iter::traits::iterator::Iterator>::next
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/adapters/mod.rs:160:9
  11: alloc::vec::Vec<T,A>::extend_desugared
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/alloc/src/vec/mod.rs:2614:35
  12: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/alloc/src/vec/spec_extend.rs:18:9
  13: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/alloc/src/vec/spec_from_iter_nested.rs:37:9
  14: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/alloc/src/vec/spec_from_iter.rs:33:9
  15: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/alloc/src/vec/mod.rs:2517:9
  16: core::iter::traits::iterator::Iterator::collect
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/traits/iterator.rs:1745:9
  17: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/result.rs:1887:53
  18: core::iter::adapters::process_results
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/adapters/mod.rs:149:17
  19: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/result.rs:1887:9
  20: core::iter::traits::iterator::Iterator::collect
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/core/src/iter/traits/iterator.rs:1745:9
  21: <connectorx::destinations::arrow::ArrowDestination as connectorx::destinations::Destination>::allocate
             at ./connector-x/connectorx/src/destinations/arrow/mod.rs:80:22
  22: connectorx::dispatcher::Dispatcher<S,D,TP>::run
             at ./connector-x/connectorx/src/dispatcher.rs:99:9
  23: connectorx_python::arrow::write_arrow
             at ./connector-x/connectorx-python/src/arrow.rs:89:21
  24: connectorx_python::read_sql::read_sql
             at ./connector-x/connectorx-python/src/read_sql.rs:92:23
  25: connectorx_python::read_sql
             at ./connector-x/connectorx-python/src/lib.rs:47:5
  26: connectorx_python::__pyo3_raw_read_sql::{{closure}}
             at ./connector-x/connectorx-python/src/lib.rs:38:1
  27: pyo3::callback::handle_panic::{{closure}}
             at ./.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.14.5/src/callback.rs:247:9
  28: std::panicking::try::do_call
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/std/src/panicking.rs:403:40
  29: std::panicking::try
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/std/src/panicking.rs:367:19
  30: std::panic::catch_unwind
             at /rustc/efd0483949496b067cd5f7569d1b28cd3d5d3c72/library/std/src/panic.rs:133:14
  31: pyo3::callback::handle_panic
             at ./.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.14.5/src/callback.rs:245:24
  32: connectorx_python::__pyo3_raw_read_sql
             at ./connector-x/connectorx-python/src/lib.rs:38:1
  33: cfunction_vectorcall_FASTCALL_KEYWORDS
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Objects/methodobject.c:441
  34: _PyObject_Vectorcall
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/./Include/cpython/abstract.h:127
  35: call_function
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:4963
  36: _PyEval_EvalFrameDefault
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:3515
  37: PyEval_EvalFrameEx
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:741
  38: _PyEval_EvalCodeWithName
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:4298
  39: _PyFunction_Vectorcall
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Objects/call.c:436
  40: _PyObject_Vectorcall
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/./Include/cpython/abstract.h:127
  41: call_function
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:4963
  42: _PyEval_EvalFrameDefault
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:3515
  43: PyEval_EvalFrameEx
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:741
  44: _PyEval_EvalCodeWithName
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:4298
  45: PyEval_EvalCodeEx
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:4327
  46: PyEval_EvalCode
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/ceval.c:718
  47: run_eval_code_obj
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/pythonrun.c:1166
  48: run_mod
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/pythonrun.c:1188
  49: pyrun_file
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/pythonrun.c:1085
  50: pyrun_simple_file
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/pythonrun.c:439
  51: PyRun_SimpleFileExFlags
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Python/pythonrun.c:472
  52: pymain_run_file
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Modules/main.c:385
  53: pymain_run_python
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Modules/main.c:610
  54: Py_RunMain
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Modules/main.c:689
  55: pymain_main
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Modules/main.c:719
  56: Py_BytesMain
             at /tmp/python-build.20220114124724.3240/Python-3.8.12/Modules/main.c:743
  57: __libc_start_main
  58: <unknown>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Traceback (most recent call last):
  File "test_ConnectorX.py", line 42, in <module>
    table = cx.read_sql(postgres_url, query, return_type="arrow")
  File "/home/ec2-user/connector-x/connectorx-python/connectorx/__init__.py", line 151, in read_sql
    result = _read_sql(
pyo3_runtime.PanicException: not implemented
wangxiaoying commented 2 years ago

@Matthieu-Tinycoaching Looks like the there is a type that is not supported. What are the types of your query result columns?

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying I have a postgres AWS database and please find below the types of my query result columns:

id: integer
id_user: ineger
id_timezone: integer
timestamp: timestamp with timzone
code_content: character varying
id_box: integer
wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching , we are actually currently working on moving from arrow to arrow2 crate to construct arrow record batches. And timestamp type is only supported in arrow2. Can you set the return_type="arrow2". It is supported by versions >= 0.2.4-alpha.3. Other code remains the same.

Matthieu-Tinycoaching commented 2 years ago

Hi @wangxiaoying I tried your suggestion but got the following error:

[2022-01-24T08:09:31Z DEBUG connectorx_python::arrow2] Protocol: binary
[2022-01-24T08:09:31Z DEBUG connectorx_python::arrow2] Running dispatcher
[2022-01-24T08:09:31Z DEBUG connectorx::dispatcher] Fetching metadata
Traceback (most recent call last):
  File "test_ConnectorX.py", line 42, in <module>
    table = cx.read_sql(postgres_url, query, return_type="arrow2")
  File "/home/ec2-user/connector-x/connectorx-python/connectorx/__init__.py", line 151, in read_sql
    result = _read_sql(
RuntimeError: No conversion rule from TimestampTz(true) to connectorx::destinations::arrow2::typesystem::Arrow2TypeSystem.

I supposed this is because I don't have the right version of ConnectorX. Where could I find versions >= 0.2.4-alpha.3 and should I build ConnectorX from this version on ARM instance?

wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching , sorry I forgot you need to compile the code manually. I think for now you can directly download our newest main branch and compile.

should I build ConnectorX from this version on ARM instance?

Yes, I believe if you need to install it on ARM instance then you need to compile it on ARM instance

Matthieu-Tinycoaching commented 2 years ago

Thanks @wangxiaoying but the latest release of the main branch is 0.2.3 not the one you mentioned?

wangxiaoying commented 2 years ago

Thanks @wangxiaoying but the latest release of the main branch is 0.2.3 not the one you mentioned?

I think the current main branch is under the development of 0.2.4-alpha.5 You can check the commit messages here: https://github.com/sfu-db/connector-x/commits/main and the version number in connectorx-python/pyproject.toml

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying and how to clone this last development version?

wangxiaoying commented 2 years ago

@wangxiaoying and how to clone this last development version?

Can you use the methods provided by github? like git clone or download the zip file directly from https://github.com/sfu-db/connector-x/archive/refs/heads/main.zip

Matthieu-Tinycoaching commented 2 years ago

@wangxiaoying I checked the version I built ConnectorX from and it's already 0.2.4-alpha.5 according to connectorx-python/pyproject.toml file:

[tool.poetry]
authors = ["SFU Database System Lab <dsl.cs.sfu@gmail.com>"] 
classifiers = [
  "Development Status :: 4 - Beta",
  "Topic :: Software Development :: Build Tools",
  "Environment :: Console",
  "Operating System :: OS Independent",
  "Intended Audience :: Science/Research",
  "Intended Audience :: Developers",
  "Intended Audience :: Financial and Insurance Industry",
  "Intended Audience :: Healthcare Industry",
  "Topic :: Scientific/Engineering",
  "Framework :: IPython",
] 
description = "Load data from databases to dataframes, the fastest way." 
include = ["connectorx/*.so", "connectorx/*.pyd", "LICENSE"] 
keywords = ["read_sql"] 
license = "MIT" 
maintainers = ["Weiyuan Wu <youngw@sfu.ca>"] 
name = "connectorx" 
readme = "README.md" # Markdown files are supported
version = "0.2.4-alpha.5" 

[tool.poetry.dependencies]
dask = {version = "^2021", optional = true, extras = ["dataframe"]}
modin = {version = ">=0.10", optional = true}
pandas = {version = "^1", optional = true}
polars = {version = ">=0.8", optional = true}
pyarrow = {version = ">=4", optional = true}
python = ">=3.7.1,<3.11"
numpy = ">=1.21.5"

[tool.poetry.extras]
all = ["dask", "pandas", "modin", "polars", "pyarrow"]
dask = ["dask", "pandas"]
modin = ["modin", "pandas"]
pandas = ["pandas"]
polars = ["pyarrow", "polars"]
pyarrow = ["pyarrow"]

[tool.poetry.dev-dependencies]
PyHive = "^0.6.4"
clickhouse-driver = "^0.2.1"
contexttimer = "^0.3.3"
cx-Oracle = "^8.2.1"
docopt = "^0.6.2"
ipython = "^7.25.0"
maturin = "^0.9.4"
pandahouse = "^0.2.7"
presto-python-client = "^0.8.1"
pyarrow = ">=4"
twine = "^3.4.1"
black = "^21.4b0"

[tool.pytest.ini_options]
minversion = "6.0"
python_functions = "test_* bench_*"
# python_files = check_*.py
# python_classes = Check
# addopts = "-ra -q"
# testpaths = [
#   "tests",
#   "integration",
# ]

[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["poetry_core>=1.0.0"]

And still got the issue:

[2022-01-24T08:09:31Z DEBUG connectorx_python::arrow2] Protocol: binary
[2022-01-24T08:09:31Z DEBUG connectorx_python::arrow2] Running dispatcher
[2022-01-24T08:09:31Z DEBUG connectorx::dispatcher] Fetching metadata
Traceback (most recent call last):
  File "test_ConnectorX.py", line 42, in <module>
    table = cx.read_sql(postgres_url, query, return_type="arrow2")
  File "/home/ec2-user/connector-x/connectorx-python/connectorx/__init__.py", line 151, in read_sql
    result = _read_sql(
RuntimeError: No conversion rule from TimestampTz(true) to connectorx::destinations::arrow2::typesystem::Arrow2TypeSystem.

Is there anything wrong I have done when building it manually?

wangxiaoying commented 2 years ago

Hi @Matthieu-Tinycoaching , looks like the transport rule is missing for TimestampTz from postgres to arrow. I just added it https://github.com/sfu-db/connector-x/commit/6db7187cfc4a1cf0f64f41bbf2b9ccf6b5dbb052. Can you pull the main branch and try again? Thank you!

Matthieu-Tinycoaching commented 2 years ago

Hi @wangxiaoying thanks for the update!

I finally managed to build a new version of ConnectorX including return_type="arrow2" but got similar results as return_type="pandas", i.e. basically around 370 ms per request:

[2022-01-26T10:04:07Z DEBUG connectorx_python::arrow2] Protocol: binary
[2022-01-26T10:04:07Z DEBUG connectorx_python::arrow2] Running dispatcher
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Fetching metadata
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Do not need counts in advance
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Allocate destination memory: 0x6
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Create destination partition
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Start writing
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Finalize partition 0
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Partition 0 finished
[2022-01-26T10:04:07Z DEBUG connectorx::dispatcher] Writing finished
Perform delta computation took 0.3681950569152832 seconds
   id  id_user  id_timezone                        timestamp code_content  id_box
0   6        1            1 2021-11-24 16:17:10.157618+00:00      CNT1742       0
1   7        1            1 2021-11-24 16:18:36.431020+00:00      CNT1150       2
2   8        1            1 2021-11-24 16:19:28.919014+00:00      CNT1152       0
3   9        1            1 2021-11-24 16:20:18.356929+00:00      CNT1153       0
4  11        1            1 2021-11-24 16:42:17.985951+00:00      CNT1154       0
5  12        1            1 2021-11-29 08:51:56.226414+00:00      CNT2354       0
6  13        1            3 2021-11-29 08:56:20.552893+00:00       CNT238       0
7  14        1            3 2021-12-02 16:07:12.540698+00:00       CNT071       0
8  15        1            3 2021-12-03 11:32:06.690006+00:00      CNT1699       1
lonestriker commented 2 years ago

I'm trying to compile the new 0.2.4 version of connectorx on my M1 Mac and am getting a different error now.

Using the old toolchain:

active toolchain
----------------

nightly-2021-10-21-aarch64-apple-darwin (directory override for '/Users/runner/connectorx-build/connector-x')
rustc 1.58.0-nightly (efd048394 2021-10-20)

I'm getting this compile error:

   Compiling connectorx-python v0.2.4 (/Users/runner/connectorx-build/connector-x/connectorx-python)
   Compiling postgres v0.19.2
   Compiling postgres-native-tls v0.5.0
   Compiling rust_decimal v1.16.0
   Compiling r2d2_postgres v0.18.1
   Compiling tiberius v0.5.16
   Compiling rust_decimal_macros v1.16.0
   Compiling polars-arrow v0.19.1 (https://github.com/pola-rs/polars?rev=d7bc52e1d9b984e521094bc926e214b8fef417c8#d7bc52e1)
   Compiling hyper v0.14.15
   Compiling bb8-tiberius v0.5.2
   Compiling polars-core v0.19.1 (https://github.com/pola-rs/polars?rev=d7bc52e1d9b984e521094bc926e214b8fef417c8#d7bc52e1)
error[E0658]: use of unstable library feature 'option_result_unwrap_unchecked': newly added
   --> /Users/runner/.cargo/git/checkouts/polars-b0d90607192fd414/d7bc52e/polars/polars-core/src/chunked_array/ops/aggregate.rs:103:58
    |
103 | ...                   let val = v.to_f64().unwrap_unchecked();
    |                                            ^^^^^^^^^^^^^^^^
    |
    = note: see issue #81383 <https://github.com/rust-lang/rust/issues/81383> for more information
    = help: add `#![feature(option_result_unwrap_unchecked)]` to the crate attributes to enable

error[E0658]: use of unstable library feature 'option_result_unwrap_unchecked': newly added
   --> /Users/runner/.cargo/git/checkouts/polars-b0d90607192fd414/d7bc52e/polars/polars-core/src/chunked_array/ops/aggregate.rs:112:58
    |
112 | ...                   let val = v.to_f64().unwrap_unchecked();
    |                                            ^^^^^^^^^^^^^^^^
    |
    = note: see issue #81383 <https://github.com/rust-lang/rust/issues/81383> for more information
    = help: add `#![feature(option_result_unwrap_unchecked)]` to the crate attributes to enable

error[E0658]: use of unstable library feature 'option_result_unwrap_unchecked': newly added
   --> /Users/runner/.cargo/git/checkouts/polars-b0d90607192fd414/d7bc52e/polars/polars-core/src/chunked_array/builder/list.rs:169:48
    |
169 |         unsafe { self.builder.try_push_valid().unwrap_unchecked() };
    |                                                ^^^^^^^^^^^^^^^^
    |
    = note: see issue #81383 <https://github.com/rust-lang/rust/issues/81383> for more information
    = help: add `#![feature(option_result_unwrap_unchecked)]` to the crate attributes to enable

Switching to latest nightly toolchain:

active toolchain
----------------

nightly-aarch64-apple-darwin (directory override for '/Users/runner/connectorx-build/connector-x')
rustc 1.61.0-nightly (10913c000 2022-03-03)

Gives us our old error with lifetimes:

65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, CursorProtocol, NoTls> as Transport>`
              found `<PostgresPandasTransport<'_, CursorProtocol, NoTls> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:65:1
   |
65 | impl_postgres_transport!(CursorProtocol, NoTls);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected `<PostgresPandasTransport<'tp, CursorProtocol, MakeTlsConnector> as Transport>`
              found `<PostgresPandasTransport<'_, CursorProtocol, MakeTlsConnector> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/postgres.rs:66:1
   |
66 | impl_postgres_transport!(CursorProtocol, MakeTlsConnector);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'tp` due to conflicting requirements
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
   |
note: first, the lifetime cannot outlive the lifetime `'tp` as defined here...
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
note: ...so that the types are compatible
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
   = note: expected `<SqlitePandasTransport<'tp> as Transport>`
              found `<SqlitePandasTransport<'_> as Transport>`
note: but, the lifetime must be valid for the lifetime `'d` as defined here...
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
note: ...so that the type `PandasDestination<'_>` will meet its required lifetime bounds
  --> connectorx-python/src/pandas/transports/sqlite.rs:13:1
   |
13 | / impl_transport!(
14 | |     name = SqlitePandasTransport<'tp>,
15 | |     error = ConnectorXPythonError,
16 | |     systems = SQLiteTypeSystem => PandasTypeSystem,
...  |
29 | |     }
30 | | );
   | |_^
   = note: this error originates in the macro `impl_transport` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0495`.
error: could not compile `connectorx-python` due to 24 previous errors
error: Recipe `ci-build-python-extention` failed on line 94 with exit code 101
wangxiaoying commented 2 years ago

Hi @lonestriker , can you try nightly-2021-11-18? https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L34

lonestriker commented 2 years ago

Hi @lonestriker , can you try nightly-2021-11-18? https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L34

Thank you, worked perfectly. I'll know where to look for the next update as well :)