search

elastic / eland

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
https://eland.readthedocs.io
Apache License 2.0
628 stars 98 forks source link

[NLP] Unable to install eland[pytorch] using pip due to unable to find matching pytorch #655

Closed maxhniebergall closed 5 months ago

maxhniebergall commented 6 months ago

When trying to install eland[pytorch] on python 3.10 in a jupyter notebook I get the following error:

INFO: pip is looking at multiple versions of eland[pytorch] to determine which version is compatible with other requirements. This could take a while.
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement torch<2.0,>=1.13.1; extra == "pytorch" (from eland[pytorch]) (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2)
ERROR: No matching distribution found for torch<2.0,>=1.13.1; extra == "pytorch"

Running !pip install torch==1.13.1 (as suggested by the readme) fails with the error:

ERROR: Could not find a version that satisfies the requirement torch==1.13.1 (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2)
ERROR: No matching distribution found for torch==1.13.1

I was able to work around this error by installing miniconda with python version 3.10, and manually installing pytorch prior to eland.!conda install -y pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 -c pytorch

pquentin commented 6 months ago

Thanks for the report, Max. However, I can't reproduce.

Going to https://colab.research.google.com/github/elastic/elasticsearch-labs/blob/main/supporting-blog-content/multilingual-e5/multilingual-e5.ipynb and running the !pip install elasticsearch eland[pytorch] code block does work:

Collecting elasticsearch
  Downloading elasticsearch-8.12.0-py3-none-any.whl (431 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 431.9/431.9 kB 4.2 MB/s eta 0:00:00
Collecting eland[pytorch]
  Downloading eland-8.12.0-py3-none-any.whl (163 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.5/163.5 kB 9.4 MB/s eta 0:00:00
Collecting elastic-transport<9,>=8 (from elasticsearch)
  Downloading elastic_transport-8.12.0-py3-none-any.whl (59 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.9/59.9 kB 6.0 MB/s eta 0:00:00
Requirement already satisfied: pandas<2,>=1.5 in /usr/local/lib/python3.10/dist-packages (from eland[pytorch]) (1.5.3)
Requirement already satisfied: matplotlib>=3.6 in /usr/local/lib/python3.10/dist-packages (from eland[pytorch]) (3.7.1)
Requirement already satisfied: numpy<2,>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from eland[pytorch]) (1.23.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from eland[pytorch]) (23.2)
Collecting torch<2.0,>=1.13.1 (from eland[pytorch])
  Downloading torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl (887.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 887.5/887.5 MB 1.7 MB/s eta 0:00:00
Collecting sentence-transformers<=2.2.2,>=2.1.0 (from eland[pytorch])
  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.0/86.0 kB 11.3 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting transformers[torch]<=4.33.2,>=4.31.0 (from eland[pytorch])
  Downloading transformers-4.33.2-py3-none-any.whl (7.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 81.7 MB/s eta 0:00:00
Requirement already satisfied: urllib3<3,>=1.26.2 in /usr/local/lib/python3.10/dist-packages (from elastic-transport<9,>=8->elasticsearch) (2.0.7)
Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from elastic-transport<9,>=8->elasticsearch) (2023.11.17)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland[pytorch]) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland[pytorch]) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland[pytorch]) (4.47.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland[pytorch]) (1.4.5)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland[pytorch]) (9.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland[pytorch]) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.6->eland[pytorch]) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<2,>=1.5->eland[pytorch]) (2023.3.post1)
Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (4.35.2)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (4.66.1)
Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (0.16.0+cu121)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (1.2.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (1.11.4)
Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (3.8.1)
Collecting sentencepiece (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch])
  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 64.0 MB/s eta 0:00:00
Requirement already satisfied: huggingface-hub>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (0.20.3)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch<2.0,>=1.13.1->eland[pytorch]) (4.5.0)
Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch<2.0,>=1.13.1->eland[pytorch])
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 59.9 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu11==8.5.0.96 (from torch<2.0,>=1.13.1->eland[pytorch])
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 2.4 MB/s eta 0:00:00
Collecting nvidia-cublas-cu11==11.10.3.66 (from torch<2.0,>=1.13.1->eland[pytorch])
  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 2.5 MB/s eta 0:00:00
Collecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch<2.0,>=1.13.1->eland[pytorch])
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 51.8 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<2.0,>=1.13.1->eland[pytorch]) (67.7.2)
Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<2.0,>=1.13.1->eland[pytorch]) (0.42.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (3.13.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (2023.6.3)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (2.31.0)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch])
  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 77.2 MB/s eta 0:00:00
Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (0.4.1)
Collecting accelerate>=0.20.3 (from transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch])
  Downloading accelerate-0.26.1-py3-none-any.whl (270 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 270.9/270.9 kB 27.9 MB/s eta 0:00:00
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.20.3->transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (5.9.5)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (2023.6.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib>=3.6->eland[pytorch]) (1.16.0)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (8.1.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk->sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (1.3.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[torch]<=4.33.2,>=4.31.0->eland[pytorch]) (3.6)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch]) (3.2.0)
INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.
Collecting torchvision (from sentence-transformers<=2.2.2,>=2.1.0->eland[pytorch])
  Downloading torchvision-0.16.2-cp310-cp310-manylinux1_x86_64.whl (6.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.8/6.8 MB 90.7 MB/s eta 0:00:00
  Downloading torchvision-0.16.1-cp310-cp310-manylinux1_x86_64.whl (6.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.8/6.8 MB 61.2 MB/s eta 0:00:00
  Downloading torchvision-0.16.0-cp310-cp310-manylinux1_x86_64.whl (6.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 63.5 MB/s eta 0:00:00
  Downloading torchvision-0.15.2-cp310-cp310-manylinux1_x86_64.whl (6.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 91.1 MB/s eta 0:00:00
  Downloading torchvision-0.15.1-cp310-cp310-manylinux1_x86_64.whl (6.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 59.8 MB/s eta 0:00:00
  Downloading torchvision-0.14.1-cp310-cp310-manylinux1_x86_64.whl (24.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.2/24.2 MB 14.2 MB/s eta 0:00:00
Building wheels for collected packages: sentence-transformers
  Building wheel for sentence-transformers (setup.py) ... done
  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125923 sha256=95fc17059701a3d5ccf9b50cb5433913fd2faedbcbabb91f493ef9b0c29c2e45
  Stored in directory: /root/.cache/pip/wheels/62/f2/10/1e606fd5f02395388f74e7462910fe851042f97238cbbd902f
Successfully built sentence-transformers
Installing collected packages: tokenizers, sentencepiece, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cublas-cu11, elastic-transport, nvidia-cudnn-cu11, elasticsearch, transformers, torch, eland, torchvision, accelerate, sentence-transformers
  Attempting uninstall: tokenizers
    Found existing installation: tokenizers 0.15.1
    Uninstalling tokenizers-0.15.1:
      Successfully uninstalled tokenizers-0.15.1
  Attempting uninstall: transformers
    Found existing installation: transformers 4.35.2
    Uninstalling transformers-4.35.2:
      Successfully uninstalled transformers-4.35.2
  Attempting uninstall: torch
    Found existing installation: torch 2.1.0+cu121
    Uninstalling torch-2.1.0+cu121:
      Successfully uninstalled torch-2.1.0+cu121
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.16.0+cu121
    Uninstalling torchvision-0.16.0+cu121:
      Successfully uninstalled torchvision-0.16.0+cu121
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchaudio 2.1.0+cu121 requires torch==2.1.0, but you have torch 1.13.1 which is incompatible.
torchdata 0.7.0 requires torch==2.1.0, but you have torch 1.13.1 which is incompatible.
torchtext 0.16.0 requires torch==2.1.0, but you have torch 1.13.1 which is incompatible.
Successfully installed accelerate-0.26.1 eland-8.12.0 elastic-transport-8.12.0 elasticsearch-8.12.0 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 sentence-transformers-2.2.2 sentencepiece-0.1.99 tokenizers-0.13.3 torch-1.13.1 torchvision-0.14.1 transformers-4.33.2

(The error at the end is about torch packages that we don't use, and can be ignored.)

That said, I'm intrigued by this line in your output:

ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11

This suggests that pip was not using Python 3.10, otherwise Requires-Python >=3.7,<3.11 would be satisfied. You can use !python --version to check that your Python version is correct and !pip --version to make sure that pip is using that version. Sometimes, python -m pip ... is required to make sure pip uses the correct Python version. If that fixes it for you, we could update the notebook.

maxhniebergall commented 5 months ago

Thanks for trying to reproduce this @pquentin, I ran the version commands and indeed it seems that I was not running the correct version of python.

pyenv: python: command not found

The python command exists in these Python versions:
  3.8.16
  3.9.16
  3.10.9
  3.10.10
  3.11.4

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
pip 23.2.1 from [/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pip](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pip) (python 3.11)`

I used VS code to run this juptyer notebook, and used the UI element in VS code to select the python version/kernel, and it seems that selecting the kernel in the UI didn't translate to the python and pip calls in the jupyter notebook, so it seems like this is actually a bug in VS code.