ACupofAir
commented
1 month ago I used your code to try to reproduce it in the same environment, and found no time anomalies. The warmup time was about 2 seconds.（p.s.: comment the import ipex line)
Here is the result.
Here is the test code:
import torch
import gc
from pathlib import Path
from ipex_llm.transformers import AutoModel, AutoModelForCausalLM
from transformers import AutoTokenizer, LlamaTokenizer
# import intel_extension_for_pytorch as ipex
# from configs.model_config import MODEL_ROOT_PATH
import time
llm_warmup_prompts = ["what is ai?"]
print(">> NOTE: The one-time warmup may take several minutes. Please be patient until it finishes warm-up...")
print("-"*15, " Start warming-up LLM chatglm3-6b on MTL iGPU ", "-"*15)
# model_path = f'{MODEL_ROOT_PATH}/Qwen1.5-1.8B-Chat' # "Qwen2-1.5B" #"chatglm3-6b" #
model_path = '../model_tmp/Qwen1.5-1.8B-Chat'
print(f"model:{model_path}")
if model_path.endswith("int4") or model_path.endswith("4bit"):
model = AutoModelForCausalLM.load_low_bit(model_path,
optimize_model=True,
trust_remote_code=True,
use_cache=True)
else:
model = AutoModelForCausalLM.from_pretrained(model_path,
load_in_4bit=True,
optimize_model=True,
trust_remote_code=True,
use_cache=True)
model = model.half().to('xpu')
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
#tokenizer = LlamaTokenizer.from_pretrained(model_path, trust_remote_code=True)
start_time = time.time()
with torch.inference_mode():
for prompt in llm_warmup_prompts:
# messages = [
# {"role": "system", "content": "You are a helpful assistant."},
# {"role": "user", "content": prompt}
# ]
# text = tokenizer.apply_chat_template(
# messages,
# tokenize=False,
# add_generation_prompt=True
# )
# input_ids = tokenizer([text], return_tensors="pt").to("xpu").input_ids
input_ids = tokenizer.encode(prompt, return_tensors="pt").to('xpu')
output = model.generate(input_ids, max_new_tokens=32)
end_time = time.time()
delta_time = end_time - start_time
print(f'use {delta_time:.4f}s')
print("-"*15, " Warming-up of LLM chatglm3-6b on MTL iGPU is completed (1/4) ", "-"*15)
model.to('cpu')
torch.xpu.synchronize()
torch.xpu.empty_cache()
del model
gc.collect()Here is machine hardware parameters:
- System: Windows 11 (22H2)
- CPU: Ultra 5 125H
- Memory: 32GB (7467MHz)
- Storage: 1T
makejiang
jason-dai
Hi ipex-llm, First time warmup with Qwen1.5-1.8B-Chat took about 20 minutes on my machine. The second time was good.
is it normal?
log:
Code snippet
environments: