Closed Jacoby1218 closed 11 months ago
I can't replicate this. Also please put the full logs.
Pushed general fixes for IPEX, can you try again?
those fixes seemingly fixed this issue, now getting
RuntimeError: The size of tensor a (2048) must match the size of tensor b (3313) at non-singleton dimension 3
(tried Pygmalion 2.7b since no quantization means it's one of the only ones i can fit into vram)
onednn_verbose,info,cpu,runtime:threadpool,nthr:8
onednn_verbose,info,cpu,isa:Intel AVX2
onednn_verbose,info,gpu,runtime:DPC++
onednn_verbose,info,gpu,engine,0,backend:Level Zero,name:Intel(R) Graphics [0x56a0],driver_version:1.3.26241,binary_kernels:enabled
onednn_verbose,info,experimental features are enabled
onednn_verbose,info,use batch_normalization stats one pass is enabled
onednn_verbose,info,prim_template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,error,ocl,Error during the build of OpenCL program. Build log:
onednn_verbose,error,ocl,errcode -30,CL_INVALID_VALUE,src/gpu/ocl/ocl_gpu_engine.cpp:291
2023-09-07 08:57:37,478 - KoboldAI - ERROR - Exception on /api/v1/generate [POST]
Traceback (most recent call last):
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/flask/app.py", line 2528, in wsgi_app
response = self.full_dispatch_request()
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "aiserver.py", line 906, in g
return f(*args, **kwargs)
File "aiserver.py", line 828, in decorated
response = f(schema, *args, **kwargs)
File "aiserver.py", line 811, in decorated
raise e
File "aiserver.py", line 802, in decorated
return f(*args, **kwargs)
File "aiserver.py", line 8494, in post_generate
return _generate_text(body)
File "aiserver.py", line 8359, in _generate_text
genout = apiactionsubmit(body.prompt, use_memory=body.use_memory, use_story=body.use_story, use_world_info=body.use_world_info, use_authors_note=body.use_authors_note)
File "aiserver.py", line 3576, in apiactionsubmit
genout = apiactionsubmit_generate(tokens, minimum, maximum)
File "aiserver.py", line 3467, in apiactionsubmit_generate
_genout, already_generated = tpool.execute(model.core_generate, txt, set())
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
six.reraise(c, e, tb)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/six.py", line 719, in reraise
raise value
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
rv = meth(*args, **kwargs)
File "/home/jacob/KoboldAI/modeling/inference_model.py", line 356, in core_generate
result = self.raw_generate(
File "/home/jacob/KoboldAI/modeling/inference_model.py", line 629, in raw_generate
result = self._raw_generate(
File "/home/jacob/KoboldAI/modeling/inference_models/hf_torch.py", line 344, in _raw_generate
genout = self.model.generate(
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/transformers/generation/utils.py", line 1648, in generate
return self.sample(
File "/home/jacob/KoboldAI/modeling/inference_models/hf_torch.py", line 267, in new_sample
return new_sample.old_sample(self, *args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/transformers/generation/utils.py", line 2730, in sample
outputs = self(
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/transformers/models/opt/modeling_opt.py", line 944, in forward
outputs = self.model.decoder(
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/transformers/models/opt/modeling_opt.py", line 710, in forward
layer_outputs = decoder_layer(
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/transformers/models/opt/modeling_opt.py", line 353, in forward
hidden_states = self.fc1(hidden_states)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jacob/KoboldAI/modeling/ipex/hijacks.py", line 22, in <lambda>
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "/home/jacob/KoboldAI/modeling/ipex/hijacks.py", line 33, in __call__
return self.__orig_func(*args, **kwargs)
File "/home/jacob/KoboldAI/runtime/envs/koboldai-ipex/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: could not create a primitive```
OK I give up, seems something is either fundamentally broken with my WSL2 install or IPEX is just a mess.
seem to be getting output now, but it's quite broken
turns out that for some reason, my input being as long as it is (somewhere around 3k tokens from SillyTavern) is causing crashes. tried a character card with no context and it worked. think i may be silently running out of vram. output is still not working correctly though, but it is at least outputting something now
Max token length for Pygmalion 2.7b is 2048.
"model_max_length": 2048,
https://huggingface.co/PygmalionAI/pygmalion-2.7b/blob/main/tokenizer_config.json
oh so this doesn't do rope like llama.cpp does, noted. Problem was me then, i got it working.