Extending Hugging Face transformers APIs for Transformer-based models and improve the productivity of inference deployment. With extremely compressed models, the toolkit can greatly improve the inference efficiency on Intel platforms.
Apache License 2.0
0
stars
0
forks
source link
AssertionError: Fail to convert pytorch model with 'Intel/neural-chat-7b-v3-3' WOQ #20
from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM, WeightOnlyQuantConfig
model_name = "Intel/neural-chat-7b-v3-3"
# for int8, should set weight_dtype="int8"
config = WeightOnlyQuantConfig(compute_dtype="bf16", weight_dtype="int4")
prompt = "Once upon a time, there existed a little girl,"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=config)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)
results in:
cmd: ['python', PosixPath('/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py'), '--outfile', 'runtime_outs/ne_mistral_f32.bin', '--outtype', 'f32', 'Intel/neural-chat-7b-v3-3']
Loading model file Intel/neural-chat-7b-v3-3
Traceback (most recent call last):
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1272, in
main()
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1252, in main
model_plus = load_some_model(args.model)
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1178, in load_some_model
models_plus.append(lazy_load_file(path))
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 946, in lazy_load_file
fp = open(path, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'Intel/neural-chat-7b-v3-3'
Traceback (most recent call last):
File "/home/sdp/leaderboard/inference/serve.py", line 26, in
neural_chat_7B_v3_3_Model, neural_chat_7B_v3_3_Tokenizer = ITREXLoader('Intel/neural-chat-7b-v3-3')
File "/home/sdp/leaderboard/inference/utils/loader.py", line 16, in ITREXLoader
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=config)
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 178, in from_pretrained
model.init(
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/init.py", line 122, in init
assert os.path.exists(fp32_bin), "Fail to convert pytorch model"
results in:
cmd: ['python', PosixPath('/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py'), '--outfile', 'runtime_outs/ne_mistral_f32.bin', '--outtype', 'f32', 'Intel/neural-chat-7b-v3-3'] Loading model file Intel/neural-chat-7b-v3-3 Traceback (most recent call last): File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1272, in
main()
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1252, in main
model_plus = load_some_model(args.model)
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1178, in load_some_model
models_plus.append(lazy_load_file(path))
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 946, in lazy_load_file
fp = open(path, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'Intel/neural-chat-7b-v3-3'
Traceback (most recent call last):
File "/home/sdp/leaderboard/inference/serve.py", line 26, in
neural_chat_7B_v3_3_Model, neural_chat_7B_v3_3_Tokenizer = ITREXLoader('Intel/neural-chat-7b-v3-3')
File "/home/sdp/leaderboard/inference/utils/loader.py", line 16, in ITREXLoader
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=config)
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 178, in from_pretrained
model.init(
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/init.py", line 122, in init
assert os.path.exists(fp32_bin), "Fail to convert pytorch model"
env:
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 0.27.2 pypi_0 pypi aiofiles 23.2.1 pypi_0 pypi altair 5.2.0 pypi_0 pypi annotated-types 0.6.0 pypi_0 pypi anyio 4.3.0 pypi_0 pypi attrs 23.2.0 pypi_0 pypi ca-certificates 2023.12.12 h06a4308_0
certifi 2024.2.2 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi click 8.1.7 pypi_0 pypi colorama 0.4.6 pypi_0 pypi contextlib2 21.6.0 pypi_0 pypi contourpy 1.2.0 pypi_0 pypi cycler 0.12.1 pypi_0 pypi deprecated 1.2.14 pypi_0 pypi einops 0.7.0 pypi_0 pypi exceptiongroup 1.2.0 pypi_0 pypi fastapi 0.110.0 pypi_0 pypi ffmpy 0.3.2 pypi_0 pypi filelock 3.13.1 pypi_0 pypi fonttools 4.49.0 pypi_0 pypi fsspec 2024.2.0 pypi_0 pypi gradio 4.20.1 pypi_0 pypi gradio-client 0.11.0 pypi_0 pypi h11 0.14.0 pypi_0 pypi httpcore 1.0.4 pypi_0 pypi httpx 0.27.0 pypi_0 pypi huggingface-hub 0.21.4 pypi_0 pypi idna 3.6 pypi_0 pypi importlib-resources 6.1.3 pypi_0 pypi intel-extension-for-pytorch 2.1.100 pypi_0 pypi intel-extension-for-transformers 1.2.2 pypi_0 pypi jinja2 3.1.3 pypi_0 pypi joblib 1.3.2 pypi_0 pypi jsonschema 4.21.1 pypi_0 pypi jsonschema-specifications 2023.12.1 pypi_0 pypi kiwisolver 1.4.5 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
markdown-it-py 3.0.0 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi matplotlib 3.8.3 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi ncurses 6.4 h6a678d5_0
networkx 3.2.1 pypi_0 pypi neural-compressor 2.4.1 pypi_0 pypi neural-speed 0.2 pypi_0 pypi numpy 1.26.4 pypi_0 pypi nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi nvidia-curand-cu12 10.3.2.106 pypi_0 pypi nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi nvidia-nccl-cu12 2.18.1 pypi_0 pypi nvidia-nvjitlink-cu12 12.4.99 pypi_0 pypi nvidia-nvtx-cu12 12.1.105 pypi_0 pypi opencv-python-headless 4.9.0.80 pypi_0 pypi openssl 1.1.1w h7f8727e_0
orjson 3.9.15 pypi_0 pypi packaging 23.2 pypi_0 pypi pandas 2.2.1 pypi_0 pypi pillow 10.2.0 pypi_0 pypi pip 23.3.1 py39h06a4308_0
prettytable 3.10.0 pypi_0 pypi psutil 5.9.8 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi pycocotools 2.0.7 pypi_0 pypi pydantic 2.6.3 pypi_0 pypi pydantic-core 2.16.3 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pygments 2.17.2 pypi_0 pypi pyparsing 3.1.2 pypi_0 pypi python 3.9.0 hdb3f193_2
python-dateutil 2.9.0.post0 pypi_0 pypi python-multipart 0.0.9 pypi_0 pypi pytz 2024.1 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi readline 8.2 h5eee18b_0
referencing 0.33.0 pypi_0 pypi regex 2023.12.25 pypi_0 pypi requests 2.31.0 pypi_0 pypi rich 13.7.1 pypi_0 pypi rpds-py 0.18.0 pypi_0 pypi ruff 0.3.1 pypi_0 pypi safetensors 0.4.2 pypi_0 pypi schema 0.7.5 pypi_0 pypi scikit-learn 1.4.1.post1 pypi_0 pypi scipy 1.12.0 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi sentencepiece 0.2.0 pypi_0 pypi setuptools 68.2.2 py39h06a4308_0
shellingham 1.5.4 pypi_0 pypi six 1.16.0 pypi_0 pypi sniffio 1.3.1 pypi_0 pypi sqlite 3.41.2 h5eee18b_0
starlette 0.36.3 pypi_0 pypi sympy 1.12 pypi_0 pypi threadpoolctl 3.3.0 pypi_0 pypi tk 8.6.12 h1ccaba5_0
tokenizers 0.14.1 pypi_0 pypi tomlkit 0.12.0 pypi_0 pypi toolz 0.12.1 pypi_0 pypi torch 2.1.1 pypi_0 pypi tqdm 4.66.2 pypi_0 pypi transformers 4.34.1 pypi_0 pypi triton 2.1.0 pypi_0 pypi typer 0.9.0 pypi_0 pypi typing-extensions 4.10.0 pypi_0 pypi tzdata 2024.1 pypi_0 pypi urllib3 2.2.1 pypi_0 pypi uvicorn 0.27.1 pypi_0 pypi wcwidth 0.2.13 pypi_0 pypi websockets 11.0.3 pypi_0 pypi wheel 0.41.2 py39h06a4308_0
wrapt 1.16.0 pypi_0 pypi xz 5.4.6 h5eee18b_0
zipp 3.17.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0
Running on Baremetal dual socker 4th Gen Xeon Machine in Intel Developer Cloud