search

AFeng-x / Draw-and-Understand

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
Apache License 2.0
45 stars 2 forks source link

metric problem #3

Open nicehuster opened 1 month ago

nicehuster commented 1 month ago

i evaluated the checkpoint at task of referring object classification on LVIS, the reproduced results is so much higher your official paper, your paper(LVIS, sim:87.06, IoU: 62.9 ), the reproduced results(sim:0.927,iou:0.698). Otherwise, the reproduced results on RefCOCOg is (METEOR:28.8, CIDEr: 384.3), your paper(METEOR:23.9, CIDEr: 162.5), where is the problem?

AFeng-x commented 1 month ago

Hi there, Actually, your procedure is correct. The ckpt provided in the repo is from a model trained with further improvements, whereas the ckpt used in our paper was from an earlier version that did not undergo extended training.

nicehuster commented 1 month ago

it's ok to give some details about the further improvements?