search

QwenLM / Qwen2.5-Coder

Qwen2.5-Coder is the code version of Qwen2.5, the large language model series developed by Qwen team, Alibaba Cloud.
3.03k stars 202 forks source link

请问官方是否有对Qwen2.5-Coder-7B-Instruct做过FIM相关数据集的评测？ #134

Closed kartikzheng closed 3 weeks ago

kartikzheng commented 1 month ago

对Qwen2.5-Coder-7B-Instruct进行FIM santacoder数据集的评测，发现相比humaneval数据集评测，低了有10个百分点左右，特别是python语言，pass@1只有50%左右。而对比其他代码大模型，pass@1并没有明显的下滑。

cyente commented 1 month ago

Could you please provide us with the specific evaluation script and the prompts that have been tested? We will check it out.

cyente commented 3 weeks ago

closed for no reply