When args.model_name is 'blip2', it dosen't have evaluate_vllm_scores function

WangFei-2019 / SNARE

Project for SNARE benchmark

10 stars 0 forks source link

When args.model_name is 'blip2', it dosen't have evaluate_vllm_scores function #2

Closed p1k0pan closed 11 months ago

p1k0pan commented 11 months ago

My running command is python SNARE_probing.py --device cuda:0 --model_name blip2 --download --dataset COCO_Semantic_Structure

Please help me with getting the right evaluation of COCO_Semantic_Structure on Blip2

WangFei-2019 commented 11 months ago

The "COCO_Semantic_Structure" task destroyed the syntactic structure of the sentence, and it is not appropriate to use the pipeline that evaluates MLLMs (relies on MLM to construct linguistic knowledge distribution) in our paper. Therefore, we did not provide the corresponding code.