search

shikras / shikra

Other
734 stars 46 forks source link

Inconsistent performance on REC task #49

Open ZhanYang-nwpu opened 11 months ago

ZhanYang-nwpu commented 11 months ago

The performance of Shikra on the dataset of REC task is quite surprising. I am trying to get the shikra-7b model by using vicuna-7b as the base model and using the shikra-7b-delta-v1 as the delta model.

I evaluate the shikra-7b model on RefCOCO testA and RefCOCO testB, but only get 79.64% and 64.54% overall accuracy. It does not match the performance on the Table 3.

Do you have any suggestion?

niiickZ commented 9 months ago

Hi, I'm facing the same problem when reproducing the method. Have you solved it?

ZhanYang-nwpu commented 9 months ago

Hi, I'm facing the same problem when reproducing the method. Have you solved it?

I'm sorry, but I still haven't solved the problem. I was confused. I didn't use the shikra model after that.

Yeemkt commented 9 months ago

The performance of Shikra on the dataset of REC task is quite surprising. I am trying to get the shikra-7b model by using vicuna-7b as the base model and using the shikra-7b-delta-v1 as the delta model.

I evaluate the shikra-7b model on RefCOCO testA and RefCOCO testB, but only get 79.64% and 64.54% overall accuracy. It does not match the performance on the Table 3.

Do you have any suggestion?

can u tell me how to evaluate the modle on refcoco or other custom datasets. Thanks!

sunsmarterjie commented 7 months ago

Could you provide the test code on refcoco?