kta-intel
commented
10 months ago Thanks for reporting. I was able to reproduce on a m7i.16xlarge (256 GB memory) instance. We are looking into this.
Closed gukejun1 closed 10 months ago
kta-intel
commented
10 months ago Thanks for reporting. I was able to reproduce on a m7i.16xlarge (256 GB memory) instance. We are looking into this.
gukejun1
commented
10 months ago sad. I changed the m7i.16xlarge and the same error was reported, as follows:
ogging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.10.2+f0ef3eaa, git-hash=f0ef3eaa, git-branch=gma/run-opt-branch
[2023-10-18 09:43:38,982] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-10-18 09:43:38,982] [INFO] [logging.py:96:log_dist] [Rank 0] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Traceback (most recent call last):
File "/home/hh/workdir/run_generation_with_deepspeed.py", line 265, in <module>
model = deepspeed.init_inference(
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/deepspeed-0.10.2+f0ef3eaa-py3.9.egg/deepspeed/__init__.py", line 342, in init_inference
engine = InferenceEngine(model, config=ds_inference_config)
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/deepspeed-0.10.2+f0ef3eaa-py3.9.egg/deepspeed/inference/engine.py", line 154, in __init__
self.module.to(device)
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 1896, in to
return super().to(*args, **kwargs)
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1164, in to
return self._apply(convert)
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 814, in _apply
module._apply(fn)
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 814, in _apply
module._apply(fn)
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 837, in _apply
param_applied = fn(param)
File "/home/hh/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1162, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!
[2023-10-18 09:43:40,026] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 9956
[2023-10-18 09:43:40,026] [ERROR] [launch.py:321:sigkill_handler] ['numactl', '-m', '0', '-C', '0-31', '/home/hh/miniconda3/envs/py39/bin/python', '-u', 'run_generation_with_deepspeed.py', '--local_rank=0', '--benchmark', '-m', 'EleutherAI/pythia-70m', '--dtype', 'float32', '--ipex', '--jit', '--print-memory'] exits with return code = 1
jingxu10
commented
10 months ago Hi @jianan-gu , could you take a look at this issue?
kta-intel
commented
10 months ago @gukejun1 Can you try again with the latest script run_generation_with_deepspeed.py
Seems an issue with tensor parallelism size is causing the out of meta tensor issue. It was observed that gpt-j is not supporting --jit, and it is advisable to remove that flag.
try: deepspeed --bind_cores_to_rank run_generation_with_deepspeed.py --benchmark -m EleutherAI/gpt-j-6b --dtype float32 --ipex --print-memory
jingxu10
commented
10 months ago sad . same error :
pls check the comment above.
gukejun1
commented
10 months ago sad . same error :
pls check the comment above.
....
[2023-10-25 14:56:02,328] [INFO] [utils.py:811:see_memory_usage] CPU Virtual Memory: used = 26.54 GB, percent = 10.7%
Traceback (most recent call last):
File "/home/ff/workdir/intel-extension-for-pytorch/examples/cpu/inference/python/llm/distributed/run_generation_with_deepspeed.py", line 362, in <module>
model = ipex.optimize_transformers(
AttributeError: module 'intel_extension_for_pytorch' has no attribute 'optimize_transformers'
[2023-10-25 14:56:04,128] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 13405
[2023-10-25 14:56:04,128] [ERROR] [launch.py:321:sigkill_handler] ['numactl', '-m', '0', '-C', '0-31', '/home/ff/miniconda3/envs/py39/bin/python', '-u', 'run_generation_with_deepspeed.py', '--local_rank=0', '--benchmark', '-m', 'EleutherAI/gpt-j-6b', '--dtype', 'float32', '--ipex', '--print-memory'] exits with return code = 1The running directory is the latest script run_generation_with_deepspeed.py intel-extension-for-pytorch/examples/cpu/inference/python/llm/distributed.
In addition, my intel-extension-for-pytorch is installed in the following way: python -m pip install https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_dev/cpu/intel_extension_for_pytorch-2.1.0.dev0%2Bcpu.llm-cp39-cp39-linux_x86_64.whl. How to obtain attribute'optiimize_transformers' ?
kta-intel
commented
10 months ago You should be okay to go ahead and install the latest released binary v2.1.0 instead of the dev branch. LLM optimizations have been released with it. https://intel.github.io/intel-extension-for-pytorch/#installation
My guess is that wheel file is still using ._optimize_transformers
gukejun1
commented
10 months ago You should be okay to go ahead and install the latest released binary
v2.1.0instead of the dev branch. LLM optimizations have been released with it. https://intel.github.io/intel-extension-for-pytorch/#installationMy guess is that wheel file is still using
._optimize_transformers
it works ，thanks
Describe the issue
Dear：
When I run Intel® Extension for PyTorch on the aws instance r7iz.4xlarge, I get the following error:
pip environment information:
Server Information: aws
how to solve it ?