search

intel-analytics / text-generation-webui

A Gradio Web UI for running local LLM on Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) using IPEX-LLM.
GNU Affero General Public License v3.0
14 stars 7 forks source link

Mistral-7B-Instruct-v0.2 fails for transformers 4.39 #34

Closed hkvision closed 2 weeks ago

hkvision commented 4 months ago

Error for 4.39.3

Traceback (most recent call last):
  File "/home/arda/kai/webui/text-generation-webui/modules/callbacks.py", line 61, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "/home/arda/kai/webui/text-generation-webui/modules/text_generation.py", line 375, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/generation/utils.py", line 1575, in generate
    result = self._sample(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/generation/utils.py", line 2697, in _sample
    outputs = self(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 1157, in forward
    outputs = self.model(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/ipex_llm/transformers/models/mistral.py", line 148, in mistral_model_forward_4_36
    return MistralModel.forward(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 1042, in forward
    layer_outputs = decoder_layer(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 757, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/ipex_llm/transformers/models/mistral.py", line 528, in mistral_attention_forward_4_36
    return forward_function(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/ipex_llm/transformers/models/mistral.py", line 806, in mistral_attention_forward_4_36_original
    past_key_value.seen_tokens += key_states.shape[-2]
AttributeError: can't set attribute

Can siucceed for 4.38.2

@jason-dai @shane-huang Seems the upstream is continuously upgrading transformers and our implementation may fail for most models...

hkvision commented 4 months ago

image QWen can run, the initial output is good, but seems output something not related after it should finish...

hkvision commented 4 months ago

Baichuan2-7b error image

hkvision commented 4 months ago

image Qwen-1.8B seems working well.

hkvision commented 4 months ago

image Chatglm3 seems working well.

github-actions[bot] commented 3 months ago

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.

github-actions[bot] commented 2 months ago

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.

github-actions[bot] commented 2 weeks ago

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.