Closed wangzhao0217 closed 5 months ago
The issue solved by adding: --tune_mm_mlp_adapter True
However, got new error :
Command:
!python /content/LLaVA/llava/eval/run_llava.py --model-path /content/LLaVA/llava/checkpoints/ESA-llava-qlora \
--model-base /content/LLaVA/llava-v1.5-7b \
--image-file "/content/drive/MyDrive/train_rgb/Bad/R1C1 N1_103.tif" \
--query "What is the condition of this road"
Error:
]
!python /content/LLaVA/llava/eval/run_llava.py --model-path /content/LLaVA/llava/checkpoints/ESA-llava-qlora \
--model-base /content/LLaVA/llava-v1.5-7b \
--image-file "/content/drive/MyDrive/train_rgb/Bad/R1C1 N1_103.tif" \
--query "What is the condition of this road"
[2024-03-08 16:07:29,340] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2024-03-08 16:07:30.682805: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-08 16:07:30.682860: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-08 16:07:30.684501: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-08 16:07:31.916176: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Loading LLaVA from base model...
Loading checkpoint shards: 0% 0/2 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.__get__(instance, owner)()
Loading checkpoint shards: 100% 2/2 [00:04<00:00, 2.30s/it]
Loading additional LLaVA weights...
Loading LoRA weights...
Merging LoRA weights...
/usr/local/lib/python3.10/dist-packages/peft/tuners/lora/bnb.py:272: UserWarning: Merge lora module to 4-bit linear may get different generations due to rounding errors.
warnings.warn(
Model is loaded...
Traceback (most recent call last):
File "/content/LLaVA/llava/eval/run_llava.py", line 145, in <module>
eval_model(args)
File "/content/LLaVA/llava/eval/run_llava.py", line 115, in eval_model
output_ids = model.generate(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/content/LLaVA/llava/model/language_model/llava_llama.py", line 137, in generate
return super().generate(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 1525, in generate
return self.sample(
File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2622, in sample
outputs = self(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/content/LLaVA/llava/model/language_model/llava_llama.py", line 91, in forward
return super().forward(
File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 1201, in forward
logits = self.lm_head(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/nn/modules.py", line 414, in forward
assert self.weight.shape[1] == 1
AssertionError
solved using zero3.json
May I ask did you just changing zero2.json to zero3.json in finetune_qlora.sh? I tried but the ' assert self.weight.shape[1] == 1' error still exists.
May I ask did you just changing zero2.json to zero3.json in finetune_qlora.sh? I tried but the ' assert self.weight.shape[1] == 1' error still exists.
yes, that's work for me
can you help me , when i used zero3 i got this error :
ValueError: DeepSpeed Zero-3 is not compatible with low_cpu_mem_usage=True or with passing a device_map.
can you help me , when i used zero3 i got this error :
ValueError: DeepSpeed Zero-3 is not compatible with low_cpu_mem_usage=True or with passing a device_map.
I think maybe it can be solved by just commenting the 'device_map' parameter in the code, and pass low_cpu_mem_usage=False or commenting it as well.
Thank you @LordUky ! I still encountered an assertion error. Do you have a solution for this? Also, could you please tell me the PyTorch version you used?
Hi I completely followed the provided commands to set up the environment. Unfortunately, as mentioned in my previous comments, I still got this assertion error after switching to zero3.json. Actually I am not working on this project now, but good luck!
Question
Dear Haotian,
Congratulations on your outstanding work! I am currently engaged in a project sponsored by the European Space Agency, aiming to utilize satellite imagery and machine learning to determine road surface conditions. I plan to assess the capabilities of the LLaVA model in this scenario by fine-tuning it with our dataset.
For the fine-tuning process, I utilized Google Colab, and it was successful.
However, I encountered an issue when loading the model.
Error:
Do you have any suggestions on how to resolve this issue?
Best wishes, Zhao