请问可以发布更多关于数据清洗的细节吗？

deepseek-ai / DeepSeek-Coder

DeepSeek Coder: Let the Code Write Itself

https://coder.deepseek.com/

MIT License

6.74k stars 469 forks source link

Open Casi11as opened 12 months ago

Casi11as commented 12 months ago

目前第一步数据清洗是与starcoder相同，想学习了解后面是如何过滤掉低质量代码、语法错误或可读性差的代码的。

谢谢！

guoday commented 11 months ago

之后会有技术报告出来的

Casi11as commented 11 months ago

之后会有技术报告出来的好的，多谢，会持续关注的

i-love-doufunao commented 11 months ago

We also are closely paying attention to how to preprocessing code dataset, especially how to handle the dependencies among code file

Rosacess commented 11 months ago

Step 2: Parsing the dependencies of files within the same repository to rearrange the file positions based on their dependencies.
Step 3: Concatenating dependent files to form a single example and employ repo-level minhash for deduplication.
期待发布这两部分的更多细节

wyjksyjs commented 11 months ago

之后会有技术报告出来的

请问技术报告里包含SFT数据的构造方法吗，以及SFT数据是否开源？顺便问一下技术报告什么时候能出来，很期待👍

i-love-doufunao commented 11 months ago

请问这部分内容现在有更新吗？

ali8zake commented 11 months ago

ding 一个