fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://fugue-tutorials.readthedocs.io/
Apache License 2.0
1.98k stars 94 forks source link

[BUG] can not import `fugue` on python 3.10 #480

Closed anirban-ray-publicis-sapient closed 1 year ago

anirban-ray-publicis-sapient commented 1 year ago

Minimal Code To Reproduce

  1. create a new python 3.10 environment
  2. install fugue using pip
Result of `pip freeze`: ``` adagio==0.2.4 antlr4-python3-runtime==4.11.1 appdirs==1.4.4 fs==2.4.16 fugue==0.8.4 fugue-sql-antlr==0.1.6 Jinja2==3.1.2 MarkupSafe==2.1.3 numpy==1.24.3 pandas==2.0.2 pyarrow==12.0.0 python-dateutil==2.8.2 pytz==2023.3 qpd==0.4.2 six==1.16.0 sqlglot==15.0.0 triad==0.9.0 tzdata==2023.3 ```
import fugue

Describe the bug Running import fugue leads to the following error:

ModuleNotFoundError: No module named '_qpd_antlr'

Here is the full traceback: ```pycon Python 3.10.11 (main, Apr 20 2023, 13:59:00) [Clang 14.0.6 ] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import fugue Traceback (most recent call last): File "", line 1, in File "/path/to/venv/lib/python3.10/site-packages/fugue/__init__.py", line 5, in from fugue.api import out_transform, transform File "/path/to/venv/lib/python3.10/site-packages/fugue/api.py", line 34, in from .execution.api import ( File "/path/to/venv/lib/python3.10/site-packages/fugue/execution/__init__.py", line 2, in from .api import * File "/path/to/venv/lib/python3.10/site-packages/fugue/execution/api.py", line 17, in from .factory import make_execution_engine, try_get_context_execution_engine File "/path/to/venv/lib/python3.10/site-packages/fugue/execution/factory.py", line 15, in from .native_execution_engine import NativeExecutionEngine File "/path/to/venv/lib/python3.10/site-packages/fugue/execution/native_execution_engine.py", line 6, in from qpd_pandas import run_sql_on_pandas File "/path/to/venv/lib/python3.10/site-packages/qpd_pandas/__init__.py", line 3, in from qpd_pandas.engine import QPDPandasEngine, run_sql_on_pandas File "/path/to/venv/lib/python3.10/site-packages/qpd_pandas/engine.py", line 9, in from qpd import QPDEngine, run_sql File "/path/to/venv/lib/python3.10/site-packages/qpd/__init__.py", line 6, in from qpd.run import run_sql File "/path/to/venv/lib/python3.10/site-packages/qpd/run.py", line 3, in from qpd._parser.sql import QPDSql File "/path/to/venv/lib/python3.10/site-packages/qpd/_parser/sql.py", line 6, in from _qpd_antlr import QPDLexer, QPDParser ModuleNotFoundError: No module named '_qpd_antlr' >>> ``` (`/path/to/venv` part is edited, rest is unchanged)

Expected behavior No error.

Environment (please complete the following information):

kvnkho commented 1 year ago

Hi @anirban-ray-publicis-sapient , sorry about that. We made a release last night on one of the dependencies and that seemed to have broken things. We have retracted the package so you should be able to just re-install Fugue and it should work.

cmditch commented 1 year ago

This still seems to be an issue. Unfortunately has caused all our code/jobs to fail today 😞 . Despite qpd==0.4.2 being yanked on pypi, it's still being installed, which seems very odd given that it's.... yanked. lol.

It looks like qpd used to live on github and was maintained by @goodwanghan, but looks like the repo has since moved. Anyone know where this project lives now?

kvnkho commented 1 year ago

Oh sorry about that. I think you might need to do

pip install  --no-cache-dir   ...

so that it doesn't used the cached wheels. We're about to push a real fix soon. We actually never open sourced qpd because we wanted to eventually deprecate it, but we still maintain it. If it persists, I'm happy to hop on a call.

cmditch commented 1 year ago

Ahh, thanks for the help and info @kvnkho !

Also, thanks to you, @goodwanghan, and all the other fugue contributors for a wonderful tool! It's been very nice to adopt this in our codebase.

kvnkho commented 1 year ago

Hey @anirban-ray-publicis-sapient, @cmditch

The fix has been released. It was a packaging issue in qpd. Sorry about that!

The working versions are:

Note that if you already have Fugue installed in the environment and you do:

pip install -U fugue

This may not upgrade those packages if Fugue does not upgrade. So it's much better to install these explicitly:

pip install -U qpd triad

Thanks for reporting the issue and sorry about that! Closing this issue for now but please feel free to ping us if there are other issues.

anirban-ray-publicis-sapient commented 1 year ago

Yes, the new version solves the problem for us. Thanks for the prompt fix, much appreciated.

goodwanghan commented 1 year ago

Thank you @anirban-ray-publicis-sapient and @cmditch for reporting this issue. Please feel free to ask questions here or on the slack.