search

AI4Bharat / IndicTrans2

Translation models for 22 scheduled languages of India
https://ai4bharat.iitm.ac.in/indic-trans2
MIT License
208 stars 57 forks source link

Installation issue. #85

Closed sandeep-krutrim closed 2 weeks ago

sandeep-krutrim commented 1 month ago

Getting this error when using install.sh

`Successfully installed absl-py-2.1.0 aiofiles-23.2.1 altair-5.3.0 annotated-types-0.7.0 anyio-4.4.0 astunparse-1.6.3 attrs-23.2.0 click-7.1.2 colorama-0.4.6 contourpy-1.2.1 ctranslate2-3.9.0 cycler-0.12.1 dill-0.3.8 docopt-0.6.2 exceptiongroup-1.2.1 fastapi-0.110.3 ffmpy-0.3.2 flatbuffers-24.3.25 fonttools-4.53.0 future-1.0.0 gast-0.6.0 google-pasta-0.2.0 gradio-4.26.0 gradio-client-0.15.1 grpcio-1.64.1 h11-0.14.0 h5py-3.11.0 httpcore-1.0.5 httpx-0.27.0 huggingface-hub-0.23.4 importlib-resources-6.4.0 joblib-1.4.2 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 keras-3.4.1 kiwisolver-1.4.5 libclang-18.1.1 lxml-5.2.2 markdown-3.6 markdown-it-py-3.0.0 matplotlib-3.9.0 mdurl-0.1.2 ml-dtypes-0.3.2 mock-5.1.0 mosestokenizer-1.2.1 namex-0.0.8 nltk-3.8.1 numpy-1.26.4 openfile-0.0.7 opt-einsum-3.3.0 optree-0.11.0 orjson-3.10.6 pillow-10.4.0 portalocker-2.10.0 promise-2.3 protobuf-3.20.3 pydantic-2.8.2 pydantic-core-2.20.1 pydub-0.25.1 pyparsing-3.1.2 python-multipart-0.0.9 pyyaml-6.0.1 referencing-0.35.1 regex-2024.5.15 rich-13.7.1 rpds-py-0.18.1 ruff-0.5.0 sacrebleu-2.3.1 sacremoses-0.1.1 semantic-version-2.10.0 shellingham-1.5.4 sniffio-1.3.1 starlette-0.37.2 tabulate-0.9.0 tensorboard-2.16.2 tensorboard-data-server-0.7.2 tensorflow-2.16.2 tensorflow-addons-0.23.0 tensorflow-datasets-3.2.1 tensorflow-io-gcs-filesystem-0.37.1 tensorflow-metadata-1.15.0 termcolor-2.4.0 tf2crf-0.1.33 tokenizers-0.13.3 tomlkit-0.12.0 toolwrapper-2.1.0 toolz-0.12.1 tqdm-4.66.4 transformers-4.28.1 typeguard-2.13.3 typer-0.10.0 uctools-1.3.0 urduhack-1.1.1 uvicorn-0.30.1 websockets-11.0.3 werkzeug-3.0.3 wrapt-1.16.0 2024-07-04 06:04:54.259356: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used. 2024-07-04 06:04:54.261982: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used. 2024-07-04 06:04:54.272369: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:479] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-07-04 06:04:54.293163: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:10575] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-07-04 06:04:54.293203: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1442] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-07-04 06:04:54.305759: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-07-04 06:04:54.937893: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning:

TensorFlow Addons (TFA) has ended development and introduction of new features. TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024. Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP).

For more information see: https://github.com/tensorflow/addons/issues/2807

warnings.warn( /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/utils/ensure_tf_install.py:53: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.13.0 and strictly below 2.16.0 (nightly versions are not supported). The versions of TensorFlow you are currently using is 2.16.2 and is not supported. Some things might work, some things might not. If you were to encounter a bug, do not file an issue. If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version. You can find the compatibility matrix in TensorFlow Addon's readme: https://github.com/tensorflow/addons warnings.warn( Traceback (most recent call last): File "", line 1, in File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/init.py", line 6, in from .pipeline import Pipeline File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/pipeline/init.py", line 4, in from .core import Pipeline File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/pipeline/core.py", line 7, in from .parsers.ner import NerParser File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/pipeline/parsers/ner.py", line 4, in from urduhack.models.ner import predict_ner File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/models/ner/init.py", line 4, in from .predict import predict_ner File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/models/ner/predict.py", line 11, in from urduhack.models.ner.model import _bi_lstm_crf_model File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/urduhack/models/ner/model.py", line 5, in from tf2crf import CRF File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tf2crf/init.py", line 2, in from .crf import CRF File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tf2crf/crf.py", line 2, in import tensorflow_addons as tfa File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/init.py", line 23, in from tensorflow_addons import activations File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/activations/init.py", line 17, in from tensorflow_addons.activations.gelu import gelu File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/activations/gelu.py", line 19, in from tensorflow_addons.utils.types import TensorLike File "/home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages/tensorflow_addons/utils/types.py", line 29, in from keras.src.engine import keras_tensor ModuleNotFoundError: No module named 'keras.src.engine' [nltk_data] Downloading package punkt to [nltk_data] /home/sandeep.pandey/nltk_data... [nltk_data] Package punkt is already up-to-date! Collecting sentencepiece Using cached sentencepiece-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB) Using cached sentencepiece-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) Installing collected packages: sentencepiece Successfully installed sentencepiece-0.2.0 fatal: destination path 'fairseq' already exists and is not an empty directory. Processing /home/sandeep.pandey/local_setup/new/IndicEmbeddings/llm2vec_modified/translation/new/IndicTrans2/fairseq Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting cffi (from fairseq==0.12.2) Using cached cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB) Collecting cython (from fairseq==0.12.2) Using cached Cython-3.0.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB) Collecting hydra-core<1.1,>=1.0.7 (from fairseq==0.12.2) Using cached hydra_core-1.0.7-py3-none-any.whl.metadata (3.7 kB) Collecting omegaconf<2.1 (from fairseq==0.12.2) Using cached omegaconf-2.0.6-py3-none-any.whl.metadata (3.0 kB) WARNING: Ignoring version 2.0.6 of omegaconf since it has invalid metadata: Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/d0/eb/9d63ce09dd8aa85767c65668d5414958ea29648a0eec80a4a7d311ec2684/omegaconf-2.0.6-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: . suffix can only be used with == or != operators PyYAML (>=5.1.)


Please use pip<24.1 if you need to use this version.
  Using cached omegaconf-2.0.5-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.5 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/e5/f6/043b6d255dd6fbf2025110cea35b87f4c5100a181681d8eab496269f0d5b/omegaconf-2.0.5-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    PyYAML (>=5.1.*)
            ~~~~~~^
Please use pip<24.1 if you need to use this version.
  Using cached omegaconf-2.0.4-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.4 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/92/b1/4f3023143436f12c98bab53f0b3db617bd18a7d223627d5030e13a7b4fc2/omegaconf-2.0.4-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    PyYAML (>=5.1.*)
            ~~~~~~^
Please use pip<24.1 if you need to use this version.
  Using cached omegaconf-2.0.3-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.3 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/29/08/a88210c2c1aa0a3f65f05d8a6c98939ccb84b6fb982aa6567dec4e6773f9/omegaconf-2.0.3-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    PyYAML (>=5.1.*)
            ~~~~~~^
Please use pip<24.1 if you need to use this version.
  Using cached omegaconf-2.0.2-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.2 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/72/fe/f8d162aa059fb4f327fd75144dd69aa7e8acbb6d8d37013e4638c8490e0b/omegaconf-2.0.2-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    PyYAML (>=5.1.*)
            ~~~~~~^
Please use pip<24.1 if you need to use this version.
  Using cached omegaconf-2.0.1-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.1 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/86/ec/605805e60abdb025b06664d107335031bb8ebdc52e0a90bdbad6a7130279/omegaconf-2.0.1-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    PyYAML (>=5.1.*)
            ~~~~~~^
Please use pip<24.1 if you need to use this version.
  Using cached omegaconf-2.0.0-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: numpy>=1.21.3 in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (1.26.4)
Requirement already satisfied: regex in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (2024.5.15)
Requirement already satisfied: sacrebleu>=1.4.12 in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (2.3.1)
Requirement already satisfied: torch>=1.13 in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (2.3.1+cu118)
Requirement already satisfied: tqdm in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (4.66.4)
Collecting bitarray (from fairseq==0.12.2)
  Using cached bitarray-2.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
Collecting torchaudio>=0.8.0 (from fairseq==0.12.2)
  Using cached torchaudio-2.3.1-cp39-cp39-manylinux1_x86_64.whl.metadata (6.4 kB)
Collecting scikit-learn (from fairseq==0.12.2)
  Using cached scikit_learn-1.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Requirement already satisfied: packaging in /home/sandeep.pandey/miniconda3/envs/itv2/lib/python3.9/site-packages (from fairseq==0.12.2) (24.1)
Collecting omegaconf<2.1 (from fairseq==0.12.2)
  Using cached omegaconf-2.0.6-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.6 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/d0/eb/9d63ce09dd8aa85767c65668d5414958ea29648a0eec80a4a7d311ec2684/omegaconf-2.0.6-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    PyYAML (>=5.1.*)
            ~~~~~~^
Please use pip<24.1 if you need to use this version.
  Using cached omegaconf-2.0.5-py3-none-any.whl.metadata (3.0 kB)
WARNING: Ignoring version 2.0.5 of omegaconf since it has invalid metadata:
Requested omegaconf<2.1 from https://files.pythonhosted.org/packages/e5/f6/043b6d255dd6fbf2025110cea35b87f4c5100a181681d8eab496269f0d5b/omegaconf-2.0.5-py3-none-any.whl (from fairseq==0.12.2) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    PyYAML (>=5.1.*)
            ~~~~~~^
Please use pip<24.1 if you need to use this version.
INFO: pip is looking at multiple versions of hydra-core to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install fairseq and fairseq==0.12.2 because these package versions have conflicting dependencies.

The conflict is caused by:
    fairseq 0.12.2 depends on omegaconf<2.1
    hydra-core 1.0.7 depends on omegaconf<2.1 and >=2.0.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Setup completed!`
cmrajan commented 1 month ago

Pin the pip version to 24.0 and try. You can change below line to python3 -m pip install --upgrade pip==24.0 https://github.com/AI4Bharat/IndicTrans2/blob/f897168d81b362c59207dc1fa2cf2c77041230b7/install.sh#L15