search

InternLM / InternLM-XComposer

InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
Apache License 2.0
2.47k stars 153 forks source link

TypeError: forward() takes 2 positional arguments but 3 were given #166

Open libhot opened 8 months ago

libhot commented 8 months ago

warnings.warn( Traceback (most recent call last): File "/share/ai/InternLM-XComposer-main/./demo_asset/serve_utils.py", line 65, in gentask ret = self.mfunc(callback=_callback, self.kwargs) File "/share/ai/InternLM-XComposer-main/examples/gradio_demo_composition.py", line 699, in generate_with_callback self.model.generate(kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/transformers/generation/utils.py", line 1648, in generate return self.sample( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/transformers/generation/utils.py", line 2730, in sample outputs = self( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 362, in forward outputs = self.model( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, *kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 929, in forward layer_outputs = decoder_layer( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 625, in forward hidden_states, self_attn_weights, present_key_value = self.attention( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 391, in forward qkv_states = self.wqkv(hidden_states, im_mask) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, **kwargs) TypeError: forward() takes 2 positional arguments but 3 were given <object object at 0x7f857af2ea60>

Traceback (most recent call last): File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/gradio/queueing.py", line 489, in call_prediction output = await route_utils.call_process_api( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/gradio/route_utils.py", line 232, in call_process_api output = await app.get_blocks().process_api( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/gradio/blocks.py", line 1561, in process_api result = await self.call_function( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/gradio/blocks.py", line 1179, in call_function prediction = await anyio.to_thread.run_sync( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread return await future File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, args) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/gradio/utils.py", line 678, in wrapper response = f(args, kwargs) File "/share/ai/InternLM-XComposer-main/examples/gradio_demo_composition.py", line 499, in insert_images inject_text, locs = self.generate_loc(idx_text_sections, upimages, img_num) File "/share/ai/InternLM-XComposer-main/examples/gradio_demo_composition.py", line 200, in generate_loc output_text = self.generate(instruction, True, 1, 200, 1.005) File "/share/ai/InternLM-XComposer-main/examples/gradio_demo_composition.py", line 139, in generate generate = self.model.generate(input_ids.cuda(), File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/transformers/generation/utils.py", line 1648, in generate return self.sample( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/transformers/generation/utils.py", line 2730, in sample outputs = self( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 362, in forward outputs = self.model( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 929, in forward layer_outputs = decoder_layer( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, *kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 625, in forward hidden_states, self_attn_weights, present_key_value = self.attention( File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 391, in forward qkv_states = self.wqkv(hidden_states, im_mask) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/opt/miniconda3/envs/xcomposer2/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) TypeError: forward() takes 2 positional arguments but 3 were given

myownskyW7 commented 7 months ago

@libhot Hi, please post your code and input data so that we can reproduce the bug, we will have a try. thx~

Elisewyh commented 7 months ago

@libhot 您的这个问题解决了吗，我也遇到相同问题了

libhot commented 7 months ago

没有解决，按照官方文档安装的，可能是版本不对，将来再重试一下

panzhang0212 commented 7 months ago

@libhot @Elisewyh Hi, please post your code and input data so that we can reproduce the bug, we will have a try. thx~

shaoyan1223 commented 6 months ago

@panzhang0212 遇到了同样的问题，相同数据集在全量微调脚本上可以运行，不知道是否是peft版本问题导致的，peft==0.6.0

/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
n136-024-045:108487:108487 [4] NCCL INFO comm 0xb9132310 rank 4 nranks 8 cudaDev 4 busId 88000 - Abort COMPLETE
n136-024-045:108485:108485 [2] NCCL INFO comm 0xb3178c30 rank 2 nranks 8 cudaDev 2 busId 3d000 - Abort COMPLETE
Traceback (most recent call last):
  File "/opt/tiger/internlm-xcomposer/finetune/finetune.py", line 311, in <module>
    train()
  File "/opt/tiger/internlm-xcomposer/finetune/finetune.py", line 301, in train
    trainer.train()
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
    return inner_training_loop(
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 1938, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2759, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2784, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/deepspeed/runtime/engine.py", line 1735, in forward
    loss = self.module(*inputs, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.local/lib/python3.9/site-packages/peft/peft_model.py", line 977, in forward
    return self.base_model(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.local/lib/python3.9/site-packages/peft/tuners/tuners_utils.py", line 106, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 366, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 921, in forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/usr/local/lib/python3.9/dist-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py", line 107, in forward
    outputs = run_function(*args)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 916, in custom_forward
    return module(*inputs, output_attentions, None,
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 625, in forward
    hidden_states, self_attn_weights, present_key_value = self.attention(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 391, in forward
    qkv_states = self.wqkv(hidden_states, im_mask)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() takes 2 positional arguments but 3 were given
dle666 commented 5 months ago

@panzhang0212 遇到了同样的问题，相同的数据集在全量配置脚本上可以运行，不知道是否是peft版本问题导致的，peft==0.6.0

/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
n136-024-045:108487:108487 [4] NCCL INFO comm 0xb9132310 rank 4 nranks 8 cudaDev 4 busId 88000 - Abort COMPLETE
n136-024-045:108485:108485 [2] NCCL INFO comm 0xb3178c30 rank 2 nranks 8 cudaDev 2 busId 3d000 - Abort COMPLETE
Traceback (most recent call last):
  File "/opt/tiger/internlm-xcomposer/finetune/finetune.py", line 311, in <module>
    train()
  File "/opt/tiger/internlm-xcomposer/finetune/finetune.py", line 301, in train
    trainer.train()
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
    return inner_training_loop(
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 1938, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2759, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/tiger/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2784, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/deepspeed/runtime/engine.py", line 1735, in forward
    loss = self.module(*inputs, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.local/lib/python3.9/site-packages/peft/peft_model.py", line 977, in forward
    return self.base_model(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.local/lib/python3.9/site-packages/peft/tuners/tuners_utils.py", line 106, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 366, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 921, in forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/usr/local/lib/python3.9/dist-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/checkpoint.py", line 107, in forward
    outputs = run_function(*args)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 916, in custom_forward
    return module(*inputs, output_attentions, None,
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 625, in forward
    hidden_states, self_attn_weights, present_key_value = self.attention(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/tiger/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm2.py", line 391, in forward
    qkv_states = self.wqkv(hidden_states, im_mask)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() takes 2 positional arguments but 3 were given

我遇到了同样的问题，请问你解决了吗

Patrick-CH commented 5 months ago

I've met the same problem when using peft == 0.4.0. Updating peft to 0.10.0 seems to solve the problem in my case.

kabachuha commented 3 months ago

Updating peft or b&b didn't solve my problem, but I figured out it was Plora deriving from nn.Linear instead of nn.Module (and b&b targeting for vanilla linear layers) and I made a script to convert the checkpoint to the launchable format. And it requires swapping build_mlp's Plora to one with inner Linear layer.

import torch, os
bin_file = 'pytorch_model-00002-of-00002.bin'
sd = torch.load(bin_file)
suffixes = ['.w1', '.w3', '.w2', '.wqkv', '.wo']
upd = {}
to_pop = []

for k,v in sd.items():
    for s in suffixes:
        if s in str(k) and 'Plora' not in k:
            new_name = str(k).split(s)[0] + s + ".linear" + str(k).split(s)[1]
            to_pop.append(k)
            upd[new_name] = v

print(len(to_pop))
sd.update(upd)
for k in to_pop:
    j = sd.pop(k)

os.remove(bin_file)
torch.save(sd, bin_file)

https://huggingface.co/internlm/internlm-xcomposer2-7b-4bit/blob/main/build_mlp.py