jwills / target-duckdb

A Singer.io target for DuckDB
Other
17 stars 12 forks source link

Support Python 3.10 on macOS #16

Closed edgarrmondragon closed 1 year ago

edgarrmondragon commented 2 years ago

Is your feature request related to a problem? Please describe.

Installing in Python 3.9 works fine ``` $ pipx install target-duckdb --verbose --python=$(pyenv which python3.9) pipx >(setup:757): pipx version is 1.1.0 pipx >(setup:758): Default python interpreter is '/usr/local/Cellar/pipx/1.1.0/libexec/bin/python3.10' pipx >(package_name_from_spec:323): Determined package name: target-duckdb pipx >(package_name_from_spec:324): Package name determined in 0.0s creating virtual environment... pipx >(run_subprocess:173): running /Users/edgarramirez/.pyenv/versions/3.9.14/bin/python3.9 -m venv --without-pip /Users/edgarramirez/.local/pipx/venvs/target-duckdb pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/venvs/target-duckdb/bin/python -c import sysconfig; print(sysconfig.get_path('purelib')) pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib')) pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/venvs/target-duckdb/bin/python --version pipx >(_parsed_package_to_package_or_url:128): cleaned package spec: target-duckdb installing target-duckdb... pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/venvs/target-duckdb/bin/python -m pip install target-duckdb pipx >(run_subprocess:173): running pipx >(get_venv_metadata_for_package:303): get_venv_metadata_for_package: 195ms pipx >(_parsed_package_to_package_or_url:128): cleaned package spec: target-duckdb pipx >(needs_upgrade:69): Time since last upgrade of shared libs, in seconds: 82909. Upgrade will be run by pipx if greater than 2592000. installed package target-duckdb 0.4.1, installed using Python 3.9.14 These apps are now globally available - target-duckdb done! ✨ 🌟 ✨ ```

but fails on 3.10:

$ pipx install target-duckdb --verbose --python=$(pyenv which python3.10)
pipx >(setup:757): pipx version is 1.1.0
pipx >(setup:758): Default python interpreter is '/usr/local/Cellar/pipx/1.1.0/libexec/bin/python3.10'
pipx >(package_name_from_spec:323): Determined package name: target-duckdb
pipx >(package_name_from_spec:324): Package name determined in 0.0s
creating virtual environment...
pipx >(run_subprocess:173): running /Users/edgarramirez/.pyenv/versions/3.10.7/bin/python3.10 -m venv --without-pip /Users/edgarramirez/.local/pipx/venvs/target-duckdb
pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/venvs/target-duckdb/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/venvs/target-duckdb/bin/python --version
pipx >(_parsed_package_to_package_or_url:128): cleaned package spec: target-duckdb
installing target-duckdb...
pipx >(run_subprocess:173): running /Users/edgarramirez/.local/pipx/venvs/target-duckdb/bin/python -m pip install target-duckdb
pipx >(subprocess_post_check_handle_pip_error:335): '/Users/edgarramirez/.local/pipx/venvs/target-duckdb/bin/python -m pip install target-duckdb' failed
pipx >(subprocess_post_check_handle_pip_error:352): Fatal error from pip prevented installation. Full pip output in file:
    /Users/edgarramirez/.local/pipx/logs/cmd_2022-10-26_17.52.51_pip_errors.log

pipx >(analyze_pip_output:310): pip seemed to fail to build package:
    orjson==3.6.1

Some possibly relevant errors from pip install:
    error: subprocess-exited-with-error
    error: metadata-generation-failed

pipx >(rmdir:55): removing directory /Users/edgarramirez/.local/pipx/venvs/target-duckdb
Error installing target-duckdb.

As you can see, the root issue seems to be pipelinewise-singer-python depending on orjson==3.6.1, which didn't publish binary wheels for macOS+cp310.

Describe the solution you'd like

Dropping the dependency on pipelinewise-singer-python might be the quickest way to fix this, otherwise we'd have to wait for them to release a new package with a newer orjson pinned.

Describe alternatives you've considered

Don't run this on macOS 😅

jwills commented 2 years ago

@tayloramurphy do you have a (useful) recommended way forward here to get target-duckdb working w/Python 3.10? Does Meltano support 3.10 yet?

tayloramurphy commented 2 years ago

@jwills Meltano itself supports 3.10 (https://github.com/meltano/meltano/issues/2830) - a dependency of this target does not. Seems like getting https://github.com/transferwise/pipelinewise-singer-python/pull/92/files merged would be a way to go.

edgarrmondragon commented 2 years ago

@jwills @tayloramurphy afaict the only thing being imported from the singer library is get_logger, which can easily be ported to this package.

If we also make sure to explicitly bring any transitive dependencies of the library, we can remove it.

jwills commented 2 years ago

@edgarrmondragon amaze, let me try it out

jwills commented 2 years ago

Okay PR is up, PTAL @edgarrmondragon

edgarrmondragon commented 1 year ago

This is fixed now :)