search

embeddings-benchmark / leaderboard

Code for the MTEB leaderboard
https://hf.co/spaces/mteb/leaderboard
9 stars 6 forks source link

gte-Qwen2-7b-instruct and related Qwen2 models have incorrect max_token size #6

Open tosaddler opened 1 month ago

tosaddler commented 1 month ago

Not sure where the info is pulled from, but it is showing 131072 rather than 32768.

image image
orionw commented 1 month ago

Thanks @tosaddler for noticing, will fix!

orionw commented 1 month ago

Seems like it may be re-appearing. Will take a deeper look later.

orionw commented 1 month ago

I can make a manual fix for this but would have to write it in.

FWIW, this is caused by extracting it out their their config.json file. I think it's a bug on their part. We can either provide some code to override the extraction or we can leave it.

Thoughts @KennethEnevoldsen?

KennethEnevoldsen commented 1 month ago

Ahh that is frustrating - I would probably create an overwrite on our end (create some sort of file which takes priority).