search

huggingface / optimum-neuron

Easy, fast and very cheap training and inference on AWS Trainium and Inferentia chips.
Apache License 2.0
196 stars 59 forks source link

HF DL AMI v20240318 optimum-neuron install generates error on import #545

Closed jimburtoft closed 5 months ago

jimburtoft commented 6 months ago

System Info

Brand new HF DL AMI v20240318

Who can help?

@dacorvo

Information

Tasks

Reproduction (minimal, reproducible, runnable)

Open the python prompt and type from optimum.neuron import pipeline

You will get an error similar to the following:

=============================================================================
       __|  __|_  )
       _|  (     /   HuggingFace Deep Learning Neuron AMI (Ubuntu 20.04)
      ___|\___|___|
=============================================================================
Welcome to the HuggingFace Deep Learning Neuron AMI (Ubuntu 20.04)
* Examples:     /home/ubuntu/huggingface-neuron-samples
* Notebooks:    /home/ubuntu/huggingface-neuron-notebooks
* Documentation: https://huggingface.co/docs/optimum-neuron/
=============================================================================
(aws_neuron_venv_pytorch) ubuntu@ip-172-31-36-228:~$ python
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from optimum.neuron import pipeline
Traceback (most recent call last):
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1382, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum_neuron-0.0.20.dev0-py3.8.egg/optimum/neuron/pipelines/transformers/__init__.py", line 16, in <module>
    from .base import (
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum_neuron-0.0.20.dev0-py3.8.egg/optimum/neuron/pipelines/transformers/base.py", line 39, in <module>
    from optimum.neuron.modeling_base import NeuronBaseModel
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum_neuron-0.0.20.dev0-py3.8.egg/optimum/neuron/modeling_base.py", line 30, in <module>
    from ..exporters.neuron import export
ModuleNotFoundError: No module named 'optimum.exporters.neuron'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1373, in __getattr__
    value = getattr(module, name)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1372, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1384, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import optimum.neuron.pipelines.transformers because of the following error (look up to see its traceback):
No module named 'optimum.exporters.neuron'

Expected behavior

I expect an import from optimum.neuron to succeed

jimburtoft commented 6 months ago

If I run: pip install -U optimum.neuron from the commandline and then run the command again in python, it works.

jimburtoft commented 6 months ago

Before I ran any upgrades, I verified the environment:

(aws_neuron_venv_pytorch) ubuntu@ip-172-31-36-228:~$ pip list | grep neuron
aws-neuronx-runtime-discovery 2.9
libneuronxla                  0.5.809
neuronx-cc                    2.12.68.0+4480452af
neuronx-distributed           0.6.0
neuronx-hwm                   2.12.0.0+422c9037c
optimum-neuron                0.0.20.dev0
tensorboard-plugin-neuronx    2.6.1.0
torch-neuronx                 1.13.1.1.13.1
torch-xla                     1.13.1+torchneurond

Now, even though the import succeeds, I get this error:

>>> from optimum.neuron import pipeline
>>>
>>> p = pipeline('text-generation', 'aws-neuron/CodeLlama-7b-hf-neuron-8xlarge')
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 892/892 [00:00<00:00, 317kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/pipelines/transformers/base.py", line 229, in pipeline
    model, model_id, tokenizer, feature_extractor = load_pipeline(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/pipelines/transformers/base.py", line 122, in load_pipeline
    model = neuronx_model_class.from_pretrained(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/modeling_base.py", line 399, in from_pretrained
    return from_pretrained_method(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/utils/require_utils.py", line 46, in wrapper
    raise ModuleNotFoundError(
ModuleNotFoundError: _from_pretrained requires the `transformers_neuronx` package. You can install it by running: pip install transformers_neuronx

If it matters, when I logged in I got dumped to a venv, but I thought that was just the way the new image worked.

If I follow the instructions, pip install transformers_neuronx, I get: 

(aws_neuron_venv_pytorch) ubuntu@ip-172-31-36-228:~$ pip install transformers_neuronx
Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Collecting transformers_neuronx
  Downloading https://pip.repos.neuron.amazonaws.com/transformers-neuronx/transformers_neuronx-0.9.474-py3-none-any.whl (194 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.7/194.7 kB 8.0 MB/s eta 0:00:00
Requirement already satisfied: accelerate in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers_neuronx) (0.23.0)
Requirement already satisfied: safetensors in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers_neuronx) (0.4.2)
Requirement already satisfied: torch-neuronx in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers_neuronx) (1.13.1.1.13.1)
Requirement already satisfied: transformers in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers_neuronx) (4.36.2)
Requirement already satisfied: numpy>=1.17 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from accelerate->transformers_neuronx) (1.24.4)
Requirement already satisfied: packaging>=20.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from accelerate->transformers_neuronx) (21.3)
Requirement already satisfied: psutil in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from accelerate->transformers_neuronx) (5.9.8)
Requirement already satisfied: pyyaml in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from accelerate->transformers_neuronx) (6.0.1)
Requirement already satisfied: torch>=1.10.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from accelerate->transformers_neuronx) (1.13.1)
Requirement already satisfied: huggingface-hub in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from accelerate->transformers_neuronx) (0.20.3)
Requirement already satisfied: torch-xla==1.13.1+torchneurond in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch-neuronx->transformers_neuronx) (1.13.1+torchneurond)
Requirement already satisfied: libneuronxla==0.5.809 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch-neuronx->transformers_neuronx) (0.5.809)
Requirement already satisfied: protobuf<5 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch-neuronx->transformers_neuronx) (3.19.6)
Requirement already satisfied: aws-neuronx-runtime-discovery~=2.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (2.9)
Requirement already satisfied: neuronx-cc~=2.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (2.12.68.0+4480452af)
Requirement already satisfied: boto3~=1.26 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (1.34.41)
Requirement already satisfied: botocore~=1.29 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (1.34.41)
Requirement already satisfied: typing-extensions in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch>=1.10.0->accelerate->transformers_neuronx) (4.9.0)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch>=1.10.0->accelerate->transformers_neuronx) (11.7.99)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch>=1.10.0->accelerate->transformers_neuronx) (8.5.0.96)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch>=1.10.0->accelerate->transformers_neuronx) (11.10.3.66)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch>=1.10.0->accelerate->transformers_neuronx) (11.7.99)
Requirement already satisfied: absl-py>=1.0.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (2.1.0)
Requirement already satisfied: cloud-tpu-client>=0.10.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (0.10)
Requirement already satisfied: setuptools in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.10.0->accelerate->transformers_neuronx) (69.1.0)
Requirement already satisfied: wheel in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.10.0->accelerate->transformers_neuronx) (0.42.0)
Requirement already satisfied: filelock in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers->transformers_neuronx) (3.13.1)
Requirement already satisfied: regex!=2019.12.17 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers->transformers_neuronx) (2023.12.25)
Requirement already satisfied: requests in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers->transformers_neuronx) (2.31.0)
Requirement already satisfied: tokenizers<0.19,>=0.14 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers->transformers_neuronx) (0.15.2)
Requirement already satisfied: tqdm>=4.27 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from transformers->transformers_neuronx) (4.66.2)
Requirement already satisfied: fsspec>=2023.5.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from huggingface-hub->accelerate->transformers_neuronx) (2023.10.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from packaging>=20.0->accelerate->transformers_neuronx) (3.1.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests->transformers->transformers_neuronx) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests->transformers->transformers_neuronx) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests->transformers->transformers_neuronx) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests->transformers->transformers_neuronx) (2024.2.2)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from boto3~=1.26->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (1.0.1)
Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from boto3~=1.26->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (0.10.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from botocore~=1.29->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (2.8.2)
Requirement already satisfied: google-api-python-client==1.8.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (1.8.0)
Requirement already satisfied: oauth2client in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (4.1.3)
Requirement already satisfied: httplib2<1dev,>=0.9.2 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (0.22.0)
Requirement already satisfied: google-auth>=1.4.1 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (2.27.0)
Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (0.2.0)
Requirement already satisfied: google-api-core<2dev,>=1.13.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (1.34.0)
Requirement already satisfied: six<2dev,>=1.6.1 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (1.16.0)
Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (3.0.1)
Requirement already satisfied: neuronx-hwm==2.12.0.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (2.12.0.0+422c9037c)
Requirement already satisfied: networkx<=2.6.3 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (2.6.3)
Requirement already satisfied: scipy<=1.11.2 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (1.10.1)
Requirement already satisfied: python-daemon>=2.2.4 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (3.0.1)
Requirement already satisfied: requests-unixsocket>=0.1.5 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (0.3.0)
Requirement already satisfied: islpy<=2023.1,>2021.1 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (2023.1)
Requirement already satisfied: pgzip>=0.3.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (0.3.5)
Requirement already satisfied: ec2-metadata<=2.10.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (2.10.0)
Requirement already satisfied: docutils in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from python-daemon>=2.2.4->neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (0.16)
Requirement already satisfied: lockfile>=0.10 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from python-daemon>=2.2.4->neuronx-cc~=2.0->libneuronxla==0.5.809->torch-neuronx->transformers_neuronx) (0.12.2)
Requirement already satisfied: pyasn1>=0.1.7 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from oauth2client->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (0.5.1)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from oauth2client->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (0.3.0)
Requirement already satisfied: rsa>=3.1.4 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from oauth2client->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (4.7.2)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.56.2 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-api-core<2dev,>=1.13.0->google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (1.62.0)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from google-auth>=1.4.1->google-api-python-client==1.8.0->cloud-tpu-client>=0.10.0->torch-xla==1.13.1+torchneurond->torch-neuronx->transformers_neuronx) (5.3.2)
Installing collected packages: transformers_neuronx
Successfully installed transformers_neuronx-0.9.474

At that point, I can successfully run the following example code:

from optimum.neuron import pipeline

p = pipeline('text-generation', 'aws-neuron/CodeLlama-7b-hf-neuron-8xlarge')
p("import socket\n\ndef ping_exponential_backoff(host: str):",
    do_sample=True,
    top_k=10,
    temperature=0.1,
    top_p=0.95,
    num_return_sequences=1,
    max_length=200,
)

And it starts to download and compile.

juliensimon commented 6 months ago

Same here. Confirming that pip install -U optimum && pip install -U optimum.neuron on a fresh instance with this AMI solves the issue.

JingyaHuang commented 5 months ago

Hi @jimburtoft, the installation issue shall be solved by the latest DLAMI(20240409). Thanks again for reporting it. I will close the issue as it's solved, please feel free to reopen if there is any further issue.