search

CAMeL-Lab / camel_tools

A suite of Arabic natural language processing tools developed by the CAMeL Lab at New York University Abu Dhabi.
MIT License
409 stars 72 forks source link

[BUG] pip install fails on python 3.10 with misleading error messages #81

Closed paregorios closed 2 years ago

paregorios commented 2 years ago

Describe the bug Under python 3.10.2:

But under python 3.9.10, camel-tools installs clean

To Reproduce

  1. Build a clean virtual environment using pyenv and direnv 
    $ echo 'layout pyenv 3.10.2' > .envrc
direnv: error /Users/foo/Documents/camel-test/.envrc is blocked. Run `direnv allow` to approve its content
$ direnv allow
direnv: loading ~/Documents/camel-test/.envrc
direnv: export +PYENV_VERSION +VIRTUAL_ENV ~PATH
$ which python
/Users/foo/Documents/camel-test/.direnv/python-3.10.2/bin/python
$ pip install -U pip
Requirement already satisfied: pip in ./.direnv/python-3.10.2/lib/python3.10/site-packages (21.2.4)
Collecting pip
Downloading pip-22.0.3-py3-none-any.whl (2.1 MB)
 |████████████████████████████████| 2.1 MB 3.6 MB/s 
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.2.4
Uninstalling pip-21.2.4:
  Successfully uninstalled pip-21.2.4
Successfully installed pip-22.0.3
$ pip cache purge
Files removed: 36
  2. try to install camel-tools from pypi with pip 
    
$ pip install camel-tools
pip install camel-tools
Collecting camel-tools
Downloading camel_tools-1.2.0.tar.gz (58 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.1/58.1 KB 1.1 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting future
Downloading future-0.18.2.tar.gz (829 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 829.2/829.2 KB 8.4 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting six
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting docopt
Downloading docopt-0.6.2.tar.gz (25 kB)
Preparing metadata (setup.py) ... done
Collecting cachetools
Downloading cachetools-5.0.0-py3-none-any.whl (9.1 kB)
Collecting numpy
Downloading numpy-1.22.2-cp310-cp310-macosx_11_0_arm64.whl (12.8 MB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 15.0 MB/s eta 0:00:00
Collecting scipy
Downloading scipy-1.8.0-cp310-cp310-macosx_12_0_arm64.whl (28.7 MB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.7/28.7 MB 15.0 MB/s eta 0:00:00
Collecting pandas
Downloading pandas-1.4.1-cp310-cp310-macosx_11_0_arm64.whl (10.5 MB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/10.5 MB 15.8 MB/s eta 0:00:00
Collecting scikit-learn
Downloading scikit_learn-1.0.2-cp310-cp310-macosx_12_0_arm64.whl (6.9 MB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 15.3 MB/s eta 0:00:00
Collecting dill
Downloading dill-0.3.4-py2.py3-none-any.whl (86 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.9/86.9 KB 3.4 MB/s eta 0:00:00
Collecting camel-tools
Downloading camel_tools-1.1.0.tar.gz (56 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.2/56.2 KB 2.1 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Downloading camel_tools-1.0.1.tar.gz (54 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.9/54.9 KB 1.6 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
ERROR: Cannot install camel-tools==1.0.1, camel-tools==1.1.0 and camel-tools==1.2.0 because these package versions have conflicting dependencies.

The conflict is caused by: camel-tools 1.2.0 depends on torch>=1.3 camel-tools 1.1.0 depends on torch>=1.3 camel-tools 1.0.1 depends on torch>=1.3

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

3. Try instead to install with pip from github:

$ pip cache purge Files removed: 23 pip install git+https://github.com/CAMeL-Lab/camel_tools.git Collecting git+https://github.com/CAMeL-Lab/camel_tools.git Cloning https://github.com/CAMeL-Lab/camel_tools.git to /private/var/folders/1f/w7zrbf5n4p5_vbys0mlx23k40000gn/T/pip-req-build-um_xvl55 Running command git clone --filter=blob:none --quiet https://github.com/CAMeL-Lab/camel_tools.git /private/var/folders/1f/w7zrbf5n4p5_vbys0mlx23k40000gn/T/pip-req-build-um_xvl55 Resolved https://github.com/CAMeL-Lab/camel_tools.git to commit c26947b53ea93526099a1291ebebc52902fb7ff9 Preparing metadata (setup.py) ... done Collecting future Downloading future-0.18.2.tar.gz (829 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 829.2/829.2 KB 7.5 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Collecting six Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting docopt Downloading docopt-0.6.2.tar.gz (25 kB) Preparing metadata (setup.py) ... done Collecting cachetools Downloading cachetools-5.0.0-py3-none-any.whl (9.1 kB) Collecting numpy Downloading numpy-1.22.2-cp310-cp310-macosx_11_0_arm64.whl (12.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 16.1 MB/s eta 0:00:00 Collecting scipy Downloading scipy-1.8.0-cp310-cp310-macosx_12_0_arm64.whl (28.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.7/28.7 MB 15.1 MB/s eta 0:00:00 Collecting pandas Downloading pandas-1.4.1-cp310-cp310-macosx_11_0_arm64.whl (10.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/10.5 MB 15.5 MB/s eta 0:00:00 Collecting scikit-learn Downloading scikit_learn-1.0.2-cp310-cp310-macosx_12_0_arm64.whl (6.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 15.4 MB/s eta 0:00:00 Collecting dill Downloading dill-0.3.4-py2.py3-none-any.whl (86 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.9/86.9 KB 3.1 MB/s eta 0:00:00 ERROR: Could not find a version that satisfies the requirement torch>=1.3 (from camel-tools) (from versions: none) ERROR: No matching distribution found for torch>=1.3

4. Try using python 3.9 instead of 3.10. Remove and replace the virtual environment:

$ rm -rf .direnv $ rm -rf .envrc direnv: unloading $ pyenv install 3.9.10 python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew Downloading Python-3.9.10.tar.xz... -> https://www.python.org/ftp/python/3.9.10/Python-3.9.10.tar.xz Installing Python-3.9.10... python-build: use readline from homebrew python-build: use zlib from xcode sdk Installed Python-3.9.10 to /Users/foo/.pyenv/versions/3.9.10 $ echo 'layout pyenv 3.9.10' > .envrc direnv: error /Users/foo/Documents/camel-test/.envrc is blocked. Run direnv allow to approve its content $ direnv allow direnv: loading ~/Documents/camel-test/.envrc direnv: export +PYENV_VERSION +VIRTUAL_ENV ~PATH $ which python /Users/foo/Documents/camel-test/.direnv/python-3.9.10/bin/python $ pip cache purge Files removed: 19

5. Try again using pip and pypi

$ pip cache purge Files removed: 270 $ pip install camel-tools Collecting camel-tools Downloading camel_tools-1.2.0.tar.gz (58 kB) |████████████████████████████████| 58 kB 2.8 MB/s Collecting future Downloading future-0.18.2.tar.gz (829 kB) |████████████████████████████████| 829 kB 4.9 MB/s Collecting six Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting docopt Downloading docopt-0.6.2.tar.gz (25 kB) Collecting cachetools Downloading cachetools-5.0.0-py3-none-any.whl (9.1 kB) Collecting numpy Downloading numpy-1.22.2-cp39-cp39-macosx_11_0_arm64.whl (12.8 MB) |████████████████████████████████| 12.8 MB 40.1 MB/s Collecting scipy Downloading scipy-1.8.0-cp39-cp39-macosx_12_0_arm64.whl (28.7 MB) |████████████████████████████████| 28.7 MB 17.7 MB/s Collecting pandas Downloading pandas-1.4.1-cp39-cp39-macosx_11_0_arm64.whl (10.5 MB) |████████████████████████████████| 10.5 MB 13.7 MB/s Collecting scikit-learn Downloading scikit_learn-1.0.2-cp39-cp39-macosx_12_0_arm64.whl (6.9 MB) |████████████████████████████████| 6.9 MB 15.1 MB/s Collecting dill Downloading dill-0.3.4-py2.py3-none-any.whl (86 kB) |████████████████████████████████| 86 kB 15.1 MB/s Collecting torch>=1.3 Downloading torch-1.10.2-cp39-none-macosx_11_0_arm64.whl (44.6 MB) |████████████████████████████████| 44.6 MB 13.9 MB/s Collecting transformers>=3.0.2 Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB) |████████████████████████████████| 3.8 MB 13.3 MB/s Collecting editdistance Downloading editdistance-0.6.0-cp39-cp39-macosx_11_0_arm64.whl (19 kB) Collecting requests Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) |████████████████████████████████| 63 kB 5.4 MB/s Collecting camel-kenlm Downloading camel-kenlm-2021.12.27.tar.gz (418 kB) |████████████████████████████████| 418 kB 12.7 MB/s Collecting typing-extensions Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) Collecting sacremoses Downloading sacremoses-0.0.47-py2.py3-none-any.whl (895 kB) |████████████████████████████████| 895 kB 12.6 MB/s Collecting pyyaml>=5.1 Downloading PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl (173 kB) |████████████████████████████████| 173 kB 15.6 MB/s Collecting regex!=2019.12.17 Downloading regex-2022.3.2-cp39-cp39-macosx_11_0_arm64.whl (281 kB) |████████████████████████████████| 281 kB 18.8 MB/s Collecting tokenizers!=0.11.3,>=0.11.1 Downloading tokenizers-0.11.6-cp39-cp39-macosx_12_0_arm64.whl (3.4 MB) |████████████████████████████████| 3.4 MB 14.8 MB/s Collecting tqdm>=4.27 Downloading tqdm-4.63.0-py2.py3-none-any.whl (76 kB) |████████████████████████████████| 76 kB 14.5 MB/s Collecting huggingface-hub<1.0,>=0.1.0 Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB) |████████████████████████████████| 67 kB 12.5 MB/s Collecting packaging>=20.0 Downloading packaging-21.3-py3-none-any.whl (40 kB) |████████████████████████████████| 40 kB 16.1 MB/s Collecting filelock Downloading filelock-3.6.0-py3-none-any.whl (10.0 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB) |████████████████████████████████| 98 kB 29.4 MB/s Collecting pytz>=2020.1 Downloading pytz-2021.3-py2.py3-none-any.whl (503 kB) |████████████████████████████████| 503 kB 52.2 MB/s Collecting python-dateutil>=2.8.1 Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) |████████████████████████████████| 247 kB 18.0 MB/s Collecting certifi>=2017.4.17 Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB) |████████████████████████████████| 149 kB 17.1 MB/s Collecting charset-normalizer~=2.0.0 Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) Collecting idna<4,>=2.5 Downloading idna-3.3-py3-none-any.whl (61 kB) |████████████████████████████████| 61 kB 15.2 MB/s Collecting urllib3<1.27,>=1.21.1 Downloading urllib3-1.26.8-py2.py3-none-any.whl (138 kB) |████████████████████████████████| 138 kB 17.0 MB/s Collecting joblib Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB) |████████████████████████████████| 306 kB 12.0 MB/s Collecting click Downloading click-8.0.4-py3-none-any.whl (97 kB) |████████████████████████████████| 97 kB 30.7 MB/s Collecting threadpoolctl>=2.0.0 Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB) Using legacy 'setup.py install' for camel-tools, since package 'wheel' is not installed. Using legacy 'setup.py install' for camel-kenlm, since package 'wheel' is not installed. Using legacy 'setup.py install' for docopt, since package 'wheel' is not installed. Using legacy 'setup.py install' for future, since package 'wheel' is not installed. Installing collected packages: urllib3, pyparsing, idna, charset-normalizer, certifi, typing-extensions, tqdm, six, requests, regex, pyyaml, packaging, numpy, joblib, filelock, click, tokenizers, threadpoolctl, scipy, sacremoses, pytz, python-dateutil, huggingface-hub, transformers, torch, scikit-learn, pandas, future, editdistance, docopt, dill, camel-kenlm, cachetools, camel-tools Running setup.py install for future ... done Running setup.py install for docopt ... done Running setup.py install for camel-kenlm ... done Running setup.py install for camel-tools ... done Successfully installed cachetools-5.0.0 camel-kenlm-2021.12.27 camel-tools-1.2.0 certifi-2021.10.8 charset-normalizer-2.0.12 click-8.0.4 dill-0.3.4 docopt-0.6.2 editdistance-0.6.0 filelock-3.6.0 future-0.18.2 huggingface-hub-0.4.0 idna-3.3 joblib-1.1.0 numpy-1.22.2 packaging-21.3 pandas-1.4.1 pyparsing-3.0.7 python-dateutil-2.8.2 pytz-2021.3 pyyaml-6.0 regex-2022.3.2 requests-2.27.1 sacremoses-0.0.47 scikit-learn-1.0.2 scipy-1.8.0 six-1.16.0 threadpoolctl-3.1.0 tokenizers-0.11.6 torch-1.10.2 tqdm-4.63.0 transformers-4.17.0 typing-extensions-4.1.1 urllib3-1.26.8



**Expected behavior**
Camel-tools should install via pip as outlined in package documentation, but it does not. Currently, package `setup.py` specifies `python_requires='>=3.6.0'`, but python versions >= 3.10.0 fail on installation as demonstrated above. 

**Desktop (please complete the following information):**
 - OS: macOS Monterey Version 12.2.1 (21D62)
 - Hardware: MacBook Pro (14-inch, 2021); chip: Apple M1 Pro
 - Python versions: 3.10.2, 3.9.10
 - camel-tools default/latest version via pypi (camel_tools-1.1.0.tar.gz)

**Additional context**
n/a
paregorios commented 2 years ago

This seems to be due to the fact that pytorch has not yet released 3.10 support, although it is in development.

owo commented 2 years ago

Sadly, this is why we don't officially support Python 3.10 yet. Also, we currently can't properly support M1 Macs either as we don't have one to test with. We'll keep you updated when the situation changes. In the meantime we'll add an upper limit to python_requires to reflect the Python versions we do support.

owo commented 2 years ago

So we have updated the python_requires to reflect the appropriate Python versions we support. Unfortunately, pip still downloads and installs all dependencies and only warns of the incompatible Python version at the end when trying to install camel-tools.