search

facebookresearch / mmf

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
https://mmf.sh/
Other
5.51k stars 938 forks source link

Trouble installing MMF on Colab #1322

Open catherinening opened 1 year ago

catherinening commented 1 year ago

Hi, I have been able to successfully install mmf on my local machine (an intel Mac), but it does not support a GPU version of pytorch. Thus, I am trying to get mmf running on Colab, following the same process I did to install mmf locally, but I keep running into the same error. ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects. I was able to get past this error locally using the approach I summarized below, but not on Colab. Does anyone have a solution? Do I need to make further changes to requirements.txt?

Steps to reproduce the error:

%cd '/content/drive/MyDrive/[project_dir]'
# install fastText locally
!git clone https://github.com/facebookresearch/fastText.git
!cd fastText
!make
%cd '/content/drive/MyDrive/[project_dir]/fastText' # install fastText
!sudo python setup.py install

# install tokenizers, lmdb, transformers
%cd '/content/drive/MyDrive/[project_dir]'
!pip install tokenizers
!pip install lmdb
!pip install transformers

# clone mmf repo
!git clone https://github.com/facebookresearch/mmf.git mmf

Then, I updated the requirements.txt file from the cloned repo, editing the fastText, lmdb, tokenizers, and transformers versions, and changing the pycocotools version to 2.0.7

Then, I run:

!pip install -r requirements.txt
!pip install --editable .

My full output, including the error message:

/content/drive/MyDrive/project/CS7643_project/mmf
Collecting torch==1.11.0 (from -r requirements.txt (line 1))
  Using cached torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl (750.6 MB)
Collecting torchaudio==0.11.0 (from -r requirements.txt (line 2))
  Using cached torchaudio-0.11.0-cp310-cp310-manylinux1_x86_64.whl (2.9 MB)
Collecting torchvision==0.12.0 (from -r requirements.txt (line 3))
  Using cached torchvision-0.12.0-cp310-cp310-manylinux1_x86_64.whl (21.0 MB)
Collecting numpy<=1.21.4,>=1.16.6 (from -r requirements.txt (line 4))
  Using cached numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
Collecting tqdm<4.50.0,>=4.43.0 (from -r requirements.txt (line 5))
  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
Collecting torchtext==0.12.0 (from -r requirements.txt (line 6))
  Using cached torchtext-0.12.0-cp310-cp310-manylinux1_x86_64.whl (10.4 MB)
Collecting GitPython==3.1.30 (from -r requirements.txt (line 7))
  Using cached GitPython-3.1.30-py3-none-any.whl (184 kB)
Collecting requests==2.23.0 (from -r requirements.txt (line 8))
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: fasttext==0.9.2 in /usr/local/lib/python3.10/dist-packages/fasttext-0.9.2-py3.10-linux-x86_64.egg (from -r requirements.txt (line 9)) (0.9.2)
Collecting nltk==3.6.6 (from -r requirements.txt (line 10))
  Using cached nltk-3.6.6-py3-none-any.whl (1.5 MB)
Collecting editdistance==0.5.3 (from -r requirements.txt (line 11))
  Using cached editdistance-0.5.3-cp310-cp310-linux_x86_64.whl
Collecting transformers<=4.10.1,>=3.4.0 (from -r requirements.txt (line 12))
  Using cached transformers-4.10.1-py3-none-any.whl (2.8 MB)
Collecting sklearn==0.0 (from -r requirements.txt (line 13))
  Using cached sklearn-0.0-py2.py3-none-any.whl
Collecting omegaconf<=2.1,>=2.0.6 (from -r requirements.txt (line 14))
  Using cached omegaconf-2.1.0-py3-none-any.whl (74 kB)
Requirement already satisfied: lmdb==1.4.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 15)) (1.4.1)
Collecting termcolor==1.1.0 (from -r requirements.txt (line 16))
  Using cached termcolor-1.1.0-py3-none-any.whl
Collecting iopath==0.1.8 (from -r requirements.txt (line 17))
  Using cached iopath-0.1.8-py3-none-any.whl (19 kB)
Collecting datasets==1.2.1 (from -r requirements.txt (line 18))
  Using cached datasets-1.2.1-py3-none-any.whl (159 kB)
Collecting matplotlib==3.3.4 (from -r requirements.txt (line 19))
  Using cached matplotlib-3.3.4-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: pycocotools==2.0.7 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 20)) (2.0.7)
Collecting ftfy==5.8 (from -r requirements.txt (line 21))
  Using cached ftfy-5.8-py3-none-any.whl
Collecting pytorch-lightning==1.6.0 (from -r requirements.txt (line 22))
  Using cached pytorch_lightning-1.6.0-py3-none-any.whl (582 kB)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 23)) (5.9.5)
Collecting pillow==9.3.0 (from -r requirements.txt (line 24))
  Using cached Pillow-9.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.3 MB)
Collecting sentencepiece (from -r requirements.txt (line 25))
  Using cached sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==1.11.0->-r requirements.txt (line 1)) (4.5.0)
Collecting gitdb<5,>=4.0.1 (from GitPython==3.1.30->-r requirements.txt (line 7))
  Using cached gitdb-4.0.11-py3-none-any.whl (62 kB)
Collecting chardet<4,>=3.0.2 (from requests==2.23.0->-r requirements.txt (line 8))
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5 (from requests==2.23.0->-r requirements.txt (line 8))
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests==2.23.0->-r requirements.txt (line 8))
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests==2.23.0->-r requirements.txt (line 8)) (2023.7.22)
Requirement already satisfied: pybind11>=2.2 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->-r requirements.txt (line 9)) (2.11.1)
Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->-r requirements.txt (line 9)) (67.7.2)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->-r requirements.txt (line 10)) (8.1.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->-r requirements.txt (line 10)) (1.3.2)
Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->-r requirements.txt (line 10)) (2023.6.3)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sklearn==0.0->-r requirements.txt (line 13)) (1.2.2)
Collecting portalocker (from iopath==0.1.8->-r requirements.txt (line 17))
  Using cached portalocker-2.8.2-py3-none-any.whl (17 kB)
Requirement already satisfied: pyarrow>=0.17.1 in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->-r requirements.txt (line 18)) (9.0.0)
Collecting dill (from datasets==1.2.1->-r requirements.txt (line 18))
  Using cached dill-0.3.7-py3-none-any.whl (115 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->-r requirements.txt (line 18)) (1.5.3)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->-r requirements.txt (line 18)) (3.4.1)
Collecting multiprocess (from datasets==1.2.1->-r requirements.txt (line 18))
  Using cached multiprocess-0.70.15-py310-none-any.whl (134 kB)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (0.12.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (1.4.5)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (3.1.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (2.8.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from ftfy==5.8->-r requirements.txt (line 21)) (0.2.9)
Requirement already satisfied: PyYAML>=5.4 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (6.0.1)
Requirement already satisfied: fsspec[http]!=2021.06.0,>=2021.05.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2023.6.0)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2.14.1)
Collecting torchmetrics>=0.4.1 (from pytorch-lightning==1.6.0->-r requirements.txt (line 22))
  Using cached torchmetrics-1.2.0-py3-none-any.whl (805 kB)
Collecting pyDeprecate<0.4.0,>=0.3.1 (from pytorch-lightning==1.6.0->-r requirements.txt (line 22))
  Using cached pyDeprecate-0.3.2-py3-none-any.whl (10 kB)
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (23.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12)) (3.13.1)
Requirement already satisfied: huggingface-hub>=0.0.12 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12)) (0.17.3)
Collecting sacremoses (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12))
  Using cached sacremoses-0.1.1-py3-none-any.whl (897 kB)
Collecting tokenizers<0.11,>=0.10.1 (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12))
  Using cached tokenizers-0.10.3.tar.gz (212 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting antlr4-python3-runtime==4.8 (from omegaconf<=2.1,>=2.0.6->-r requirements.txt (line 14))
  Using cached antlr4_python3_runtime-4.8-py3-none-any.whl
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.8.6)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython==3.1.30->-r requirements.txt (line 7))
  Using cached smmap-5.0.1-py3-none-any.whl (24 kB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.1->matplotlib==3.3.4->-r requirements.txt (line 19)) (1.16.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.4.0)
Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.59.2)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2.17.3)
Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.0.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.5.1)
Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.20.3)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.0.1)
Collecting lightning-utilities>=0.8.0 (from torchmetrics>=0.4.1->pytorch-lightning==1.6.0->-r requirements.txt (line 22))
  Using cached lightning_utilities-0.9.0-py3-none-any.whl (23 kB)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets==1.2.1->-r requirements.txt (line 18)) (2023.3.post1)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->-r requirements.txt (line 13)) (1.11.3)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->-r requirements.txt (line 13)) (3.2.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.3.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (4.0.3)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.3.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.3.1)
INFO: pip is looking at multiple versions of scipy to determine which version is compatible with other requirements. This could take a while.
Collecting scipy>=1.3.2 (from scikit-learn->sklearn==0.0->-r requirements.txt (line 13))
  Using cached scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.4 MB)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2.1.3)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (0.5.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.2.2)
Building wheels for collected packages: tokenizers
  error: subprocess-exited-with-error

  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
Obtaining file:///content/drive/MyDrive/project/CS7643_project/mmf
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting torch==1.11.0 (from mmf==1.0.0rc12)
  Using cached torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl (750.6 MB)
Collecting torchaudio==0.11.0 (from mmf==1.0.0rc12)
  Using cached torchaudio-0.11.0-cp310-cp310-manylinux1_x86_64.whl (2.9 MB)
Collecting torchvision==0.12.0 (from mmf==1.0.0rc12)
  Using cached torchvision-0.12.0-cp310-cp310-manylinux1_x86_64.whl (21.0 MB)
Collecting numpy<=1.21.4,>=1.16.6 (from mmf==1.0.0rc12)
  Using cached numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
Collecting tqdm<4.50.0,>=4.43.0 (from mmf==1.0.0rc12)
  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
Collecting torchtext==0.12.0 (from mmf==1.0.0rc12)
  Using cached torchtext-0.12.0-cp310-cp310-manylinux1_x86_64.whl (10.4 MB)
Collecting GitPython==3.1.30 (from mmf==1.0.0rc12)
  Using cached GitPython-3.1.30-py3-none-any.whl (184 kB)
Collecting requests==2.23.0 (from mmf==1.0.0rc12)
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: fasttext==0.9.2 in /usr/local/lib/python3.10/dist-packages/fasttext-0.9.2-py3.10-linux-x86_64.egg (from mmf==1.0.0rc12) (0.9.2)
Collecting nltk==3.6.6 (from mmf==1.0.0rc12)
  Using cached nltk-3.6.6-py3-none-any.whl (1.5 MB)
Collecting editdistance==0.5.3 (from mmf==1.0.0rc12)
  Using cached editdistance-0.5.3-cp310-cp310-linux_x86_64.whl
Collecting transformers<=4.10.1,>=3.4.0 (from mmf==1.0.0rc12)
  Using cached transformers-4.10.1-py3-none-any.whl (2.8 MB)
Collecting sklearn==0.0 (from mmf==1.0.0rc12)
  Using cached sklearn-0.0-py2.py3-none-any.whl
Collecting omegaconf<=2.1,>=2.0.6 (from mmf==1.0.0rc12)
  Using cached omegaconf-2.1.0-py3-none-any.whl (74 kB)
Requirement already satisfied: lmdb==1.4.1 in /usr/local/lib/python3.10/dist-packages (from mmf==1.0.0rc12) (1.4.1)
Collecting termcolor==1.1.0 (from mmf==1.0.0rc12)
  Using cached termcolor-1.1.0-py3-none-any.whl
Collecting iopath==0.1.8 (from mmf==1.0.0rc12)
  Using cached iopath-0.1.8-py3-none-any.whl (19 kB)
Collecting datasets==1.2.1 (from mmf==1.0.0rc12)
  Using cached datasets-1.2.1-py3-none-any.whl (159 kB)
Collecting matplotlib==3.3.4 (from mmf==1.0.0rc12)
  Using cached matplotlib-3.3.4-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: pycocotools==2.0.7 in /usr/local/lib/python3.10/dist-packages (from mmf==1.0.0rc12) (2.0.7)
Collecting ftfy==5.8 (from mmf==1.0.0rc12)
  Using cached ftfy-5.8-py3-none-any.whl
Collecting pytorch-lightning==1.6.0 (from mmf==1.0.0rc12)
  Using cached pytorch_lightning-1.6.0-py3-none-any.whl (582 kB)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from mmf==1.0.0rc12) (5.9.5)
Collecting pillow==9.3.0 (from mmf==1.0.0rc12)
  Using cached Pillow-9.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.3 MB)
Collecting sentencepiece (from mmf==1.0.0rc12)
  Using cached sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Requirement already satisfied: pyarrow>=0.17.1 in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->mmf==1.0.0rc12) (9.0.0)
Collecting dill (from datasets==1.2.1->mmf==1.0.0rc12)
  Using cached dill-0.3.7-py3-none-any.whl (115 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->mmf==1.0.0rc12) (1.5.3)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->mmf==1.0.0rc12) (3.4.1)
Collecting multiprocess (from datasets==1.2.1->mmf==1.0.0rc12)
  Using cached multiprocess-0.70.15-py310-none-any.whl (134 kB)
Requirement already satisfied: pybind11>=2.2 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->mmf==1.0.0rc12) (2.11.1)
Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->mmf==1.0.0rc12) (67.7.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from ftfy==5.8->mmf==1.0.0rc12) (0.2.9)
Collecting gitdb<5,>=4.0.1 (from GitPython==3.1.30->mmf==1.0.0rc12)
  Using cached gitdb-4.0.11-py3-none-any.whl (62 kB)
Collecting portalocker (from iopath==0.1.8->mmf==1.0.0rc12)
  Using cached portalocker-2.8.2-py3-none-any.whl (17 kB)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (0.12.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (1.4.5)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (3.1.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (2.8.2)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->mmf==1.0.0rc12) (8.1.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->mmf==1.0.0rc12) (1.3.2)
Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->mmf==1.0.0rc12) (2023.6.3)
Requirement already satisfied: PyYAML>=5.4 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (6.0.1)
Requirement already satisfied: fsspec[http]!=2021.06.0,>=2021.05.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2023.6.0)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2.14.1)
Collecting torchmetrics>=0.4.1 (from pytorch-lightning==1.6.0->mmf==1.0.0rc12)
  Using cached torchmetrics-1.2.0-py3-none-any.whl (805 kB)
Collecting pyDeprecate<0.4.0,>=0.3.1 (from pytorch-lightning==1.6.0->mmf==1.0.0rc12)
  Using cached pyDeprecate-0.3.2-py3-none-any.whl (10 kB)
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (23.2)
Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (4.5.0)
Collecting chardet<4,>=3.0.2 (from requests==2.23.0->mmf==1.0.0rc12)
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5 (from requests==2.23.0->mmf==1.0.0rc12)
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests==2.23.0->mmf==1.0.0rc12)
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests==2.23.0->mmf==1.0.0rc12) (2023.7.22)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sklearn==0.0->mmf==1.0.0rc12) (1.2.2)
Collecting antlr4-python3-runtime==4.8 (from omegaconf<=2.1,>=2.0.6->mmf==1.0.0rc12)
  Using cached antlr4_python3_runtime-4.8-py3-none-any.whl
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12) (3.13.1)
Requirement already satisfied: huggingface-hub>=0.0.12 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12) (0.17.3)
Collecting sacremoses (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12)
  Using cached sacremoses-0.1.1-py3-none-any.whl (897 kB)
Collecting tokenizers<0.11,>=0.10.1 (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12)
  Using cached tokenizers-0.10.3.tar.gz (212 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.8.6)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython==3.1.30->mmf==1.0.0rc12)
  Using cached smmap-5.0.1-py3-none-any.whl (24 kB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.1->matplotlib==3.3.4->mmf==1.0.0rc12) (1.16.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.4.0)
Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.59.2)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2.17.3)
Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.0.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.5.1)
Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.20.3)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.0.1)
Collecting lightning-utilities>=0.8.0 (from torchmetrics>=0.4.1->pytorch-lightning==1.6.0->mmf==1.0.0rc12)
  Using cached lightning_utilities-0.9.0-py3-none-any.whl (23 kB)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets==1.2.1->mmf==1.0.0rc12) (2023.3.post1)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->mmf==1.0.0rc12) (1.11.3)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->mmf==1.0.0rc12) (3.2.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.3.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (4.0.3)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.3.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.3.1)
INFO: pip is looking at multiple versions of scipy to determine which version is compatible with other requirements. This could take a while.
Collecting scipy>=1.3.2 (from scikit-learn->sklearn==0.0->mmf==1.0.0rc12)
  Using cached scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.4 MB)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2.1.3)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (0.5.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.2.2)
Building wheels for collected packages: mmf, tokenizers
  Building editable for mmf (pyproject.toml) ... done
  Created wheel for mmf: filename=mmf-1.0.0rc12-0.editable-cp310-cp310-linux_x86_64.whl size=10777 sha256=69fd1c4e0c8b6f8becbb6ff719bad5c3d359909e8f27534832e8d3f4c00ea5c0
  Stored in directory: /tmp/pip-ephem-wheel-cache-28ewnq3v/wheels/27/dc/f5/d98920883e2d6e3e0a99bb29bc96ae614580aa1be72498e749
  error: subprocess-exited-with-error

  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Successfully built mmf
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
forro1 commented 6 months ago

Hey, I met the same error, do you have a solution?