search

ShareGPT4Omni / ShareGPT4Video

[NeurIPS 2024] An official implementation of ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
https://sharegpt4video.github.io/
1.26k stars 44 forks source link

ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([257, 1024])), this look incorrect. #10

Closed haruka-025 closed 4 months ago

haruka-025 commented 4 months ago

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Load vision tower from /public/home/djingwang/jxma/ViT-L-14 /public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() return self.fget.__get__(instance, owner)() Loading checkpoint shards: 75%|█████████████████████████████████████████████████████████████████████████████████ | 3/4 [00:02<00:00, 1.05it/s] Traceback (most recent call last): File "/public/home/djingwang/yjjin/Sharegpt4video/run.py", line 159, in <module> tokenizer, model, processor, context_len = load_pretrained_model( File "/public/home/djingwang/yjjin/Sharegpt4video/llava/model/builder.py", line 145, in load_pretrained_model model = LlavaLlamaForCausalLM.from_pretrained( File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3850, in from_pretrained ) = cls._load_pretrained_model( File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4284, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/transformers/modeling_utils.py", line 805, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 285, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([257, 1024])), this look incorrect. I followed the instructions but still encountered an issue. What should I do?

xiaoachen98 commented 4 months ago

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Load vision tower from /public/home/djingwang/jxma/ViT-L-14 /public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() return self.fget.__get__(instance, owner)() Loading checkpoint shards: 75%|█████████████████████████████████████████████████████████████████████████████████ | 3/4 [00:02<00:00, 1.05it/s] Traceback (most recent call last): File "/public/home/djingwang/yjjin/Sharegpt4video/run.py", line 159, in <module> tokenizer, model, processor, context_len = load_pretrained_model( File "/public/home/djingwang/yjjin/Sharegpt4video/llava/model/builder.py", line 145, in load_pretrained_model model = LlavaLlamaForCausalLM.from_pretrained( File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3850, in from_pretrained ) = cls._load_pretrained_model( File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4284, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/transformers/modeling_utils.py", line 805, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/public/home/djingwang/anaconda3/envs/yjjin/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 285, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([257, 1024])), this look incorrect. I followed the instructions but still encountered an issue. What should I do?

Sorry for the late response. Have you solved this problem? And you can detail me the command to reproduce this error.

haruka-025 commented 4 months ago

I still haven't solved this problem.I downloaded the model weights as required, then modified the parameters in the run.py file, and after running python run.py, I encountered the above error.I don't know where I went wrong in my operation.

xiaoachen98 commented 4 months ago

I still haven't solved this problem.I downloaded the model weights as required, then modified the parameters in the run.py file, and after running python run.py, I encountered the above error.I don't know where I went wrong in my operation.

You can give me your arguments and commands to reproduce the problem.