Reproduce FIM Evaluation

deepseek-ai / DeepSeek-Coder

DeepSeek Coder: Let the Code Write Itself

https://coder.deepseek.com/

MIT License

6.6k stars 461 forks source link

Reproduce FIM Evaluation #130

Open Hambaobao opened 7 months ago

Hambaobao commented 7 months ago

Hello, thanks a lot for your great work. Could you kindly advise on how to reproduce the evaluation results of FIM as shown in Table 6 of your paper? Xnip2024-02-29_21-30-00 I tried to use bigcode-evaluation on Santacoder_FIM with the Insertion prompt structure from this repository, but it still generates very long content and I can't reproduce the results in the paper.

pkuzqh commented 6 months ago

<｜fim▁begin｜>def quick_sort(arr): if len(arr) <= 1: return arr pivot = arr[0] left = [] right = [] <｜fim▁hole｜> if arr[i] < pivot: left.append(arr[i]) else: right.append(arr[i]) return quick_sort(left) + [pivot] + quick_sort(right)<｜fim▁end｜> you need to use this prompt