fail to reproduce the example

[dongwhfdyer]

conda create -n mplug_owl python=3.10
conda activate mplug_owl
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt

I strictly followed the installation instructions above. python -m serve.web_server --base-model 'pretrained_weights/mplug-owl-llama-7b' --port 8501 --bf16 But when I try to run a simple example like this one below.

I got an error.

Traceback (most recent call last):
  File "/data/pcl/proj/mPLUG-Owl/serve/model_utils.py", line 64, in gentask
    ret = self.mfunc(callback=_callback, **self.kwargs)
  File "/data/pcl/proj/mPLUG-Owl/serve/model_worker.py", line 110, in generate_with_callback
    self.model.generate(**kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 1524, in generate
    query_outputs = self.abstractor(
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 1092, in forward
    encoder_outputs = self.encoder(
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 936, in forward
    layer_outputs = layer_module(
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 874, in forward
    cross_attention_outputs = self.crossattention(
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 846, in forward
    attention_output = self.output(self_outputs[0], hidden_states)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 791, in forward
    input_tensor = input_tensor + self.mlp(self.norm2(input_tensor))
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 679, in forward
    hidden_states = self.ffn_ln(hidden_states)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/data/pcl/proj/mPLUG-Owl/mplug_owl/modeling_mplug_owl.py", line 158, in forward
    output = torch.nn.functional.layer_norm(
  File "/data/pcl/miniconda3/envs/ovd2/lib/python3.10/site-packages/torch/nn/functional.py", line 2515, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [2816] and normalized_shape = [2048]

If there is any help, I am very grateful!!

[MAGAer13]

I think the your codebase it not the latest one. Try to pull the latest one we have fixed some bugs in https://github.com/X-PLUG/mPLUG-Owl/commit/6f695ac5fc6cd6d6c99763f298c2acb421e28564. Thanks!

[LukeForeverYoung]

https://github.com/X-PLUG/mPLUG-Owl/blob/6f695ac5fc6cd6d6c99763f298c2acb421e28564/mplug_owl/modeling_mplug_owl.py#L661-L677 We have made some updates to the MplugOwlVisualAbstractorMLP and configurations in the HF hub to comply with the requirements of HuggingFace. To resolve your issue, you can try executing a git pull command.

[dongwhfdyer]

Thanks a lot !!! Your work is wonderful!