Fail to load the clip model

[cocoshe]

I try to run the inference part, and following the command here:

python test/DS_SmartEdit_test.py --is_understanding_scenes True --model_name_or_path "./checkpoints/vicuna-13b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-13B-v1" --save_dir './checkpoints/SmartEdit-13B/Understand-15000' --steps 15000 --total_dir "./checkpoints/SmartEdit-13B" --sd_qformer_version "v1.1-13b" --resize_resolution 256

the output in terminal here:

/home/SmartEdit/test/InstructPix2PixSD_SM.py:35: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
  from diffusers.pipeline_utils import DiffusionPipeline
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:26<00:00,  8.73s/it]
> /home/SmartEdit/model/DS_SmartEdit_model.py(169)init_visual_features_extractor()
-> LLaVA_model = LlavaLlamaForCausalLM.from_pretrained(LLaVA_model_path)
(Pdb) c
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:31<00:00, 10.51s/it]
Traceback (most recent call last):
  File "/home/SmartEdit/SmartEdit/test/DS_SmartEdit_test.py", line 652, in <module>
    main()
  File "/home/SmartEdit/SmartEdit/test/DS_SmartEdit_test.py", line 215, in main
    model_.init_visual_features_extractor(LLaVA_model_path=LLaVA_model_path, sd_qformer_version=sd_qformer_version)
  File "/home/SmartEdit/SmartEdit/model/DS_SmartEdit_model.py", line 172, in init_visual_features_extractor
    self.vision_tower.load_model()
AttributeError: 'NoneType' object has no attribute 'load_model'

I read the code and maybe the initialize_vision_modules is some how not used in inference part?

https://github.com/TencentARC/SmartEdit/blob/a65f1262dfcba68c138ea95fe9936df1bd2c111d/llava/model/llava_arch.py#L41-L69

[yuzhou914]

Hi, maybe you have wrong packages version and it causes the problem. However, since I have never met your problem, I might not know which package causes the problem. I suggest you firstly conduct the LLaVA dialogue in order to make sure your LLaVA is correct. You can take a look at LLaVA instructions.

[mo230761]

Hello！How can this issue be resolved?

[cocoshe]

Hello！How can this issue be resolved?

Sorry I forget it since it's been too long ago, I suggest you can use pdb or any other debugger tools to check if the model path is correct and the vision tower is loaded successfully

[mo230761]

I found that the issue might be with the LLaVA model. Thank you very much!