search

bronyayang / Law_of_Vision_Representation_in_MLLMs

Official implementation of the Law of Vision Representation in MLLMs
https://arxiv.org/abs/2408.16357
137 stars 7 forks source link

Can't run the code #6

Open dingyue772 opened 1 month ago

dingyue772 commented 1 month ago

Thanks for the ingenious research work, but I get stuck when I want to replicate your work. I followed the tips in README, I use a conda environment with transformers=4.37.2, lmms-eval=0.2.3.dev0 and llava=1.7.0.dev0. with this environment, I can run lmms-eval and LLaVA-Next.

But when I tried to run the "law_of" code, I encountered LLaVA is not installed and then in "Law_of_Vision_Representation_in_MLLMs/llava/eval/lmms-eval/lmms_eval/models/llava.py", get_model_name_from_path is not defined. I am so confused by the problem.

bronyayang commented 1 month ago

Hi,

Make sure you build and install the llava inside this repo, if you need to use any vision representation implemented in this repo. If you use an existing environment for LLava-Next, I don't think it will be compatible with the lmms-eval in this repo.

Also, if you only need to run lmms-eval to do feature extraction for A score, you do not have to use the lmms-eval in this repo. You can use the one you can successfully run.

Let me know if you have any further issue. Thanks!

dingyue772 commented 1 month ago

Thank you for your response. I apologize that my previous description was not very clear. I am currently trying to complete the calculation of the A score. My llava package and lmms-eval package are both installed in the specified directory under the current project. The command line I used is: accelerate launch --num_processes=1 --main_process_port 12363 -m lmms_eval --model llava --model_args pretrained="/data/yangzhao/LLM_models/dy/liuhaotian/llava-v1.5-7b" --tasks mmbench_en_test --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mmbench_en_test --output_path ./logs/; Then I encountered the "LLaVA not installed" error mentioned above. I tried commenting the try...catch... block from line 25 to line 31 in Law_of_Vision_Representation_in_MLLMs/llava/eval/lmms-eval/lmms_eval/models/llava.py, and then I got another error: ValueError: Attempted to load model 'llava', but no model for this name found!These two errors occurred when I previously used the llava version and lmms-eval version specified in the pyproject.toml of the "law of" project. Refered to https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/19, I suspected that the two packages' versions were incompatible. I then tried pulling the latest versions of LLaVA-Next and lmms-eval, and got a conda environment where the versions matched. However, when I modified the pyproject.toml files in the "law of" project to use the two dev versions I tested worked, I still encountered this error. image image image image

I hava no idea where the bug comes from. Maybe have you came across to this bug? If not, I am considering moving the code in "the law" to "LLava-Next" and "lmms-eval" which I have tested, thus I can obtain the ingenious code from the "the law" project, I'll really appreciate it if you can give some tips.

bronyayang commented 1 month ago

@dingyue772 Thanks for your clarification. I personally never encountered this conflict. However, I did encounter "LLaVA not installed" for lmms-eval, but it is not because of the version conflict between two packages. That was because for some reason my llava was not built successfully, but the error is not very obvious. I rebuilt and solve the issue.

For A score feature extraction, you only need to change 3 places mentioned here. I hope if you end up using your own version combination, it does not cause difference in reproducing A score. Let me know if you cannot reproduce A score because of this.

Sorry for not being very helpful, and I will keep the issue open to see if others can find a solution.