search

jiahe7ay / MINI_LLM

This is a repository used by individuals to experiment and reproduce the pre-training process of LLM.
348 stars 53 forks source link

数据预处理问题 #15

Closed wtxfrancise closed 7 months ago

wtxfrancise commented 7 months ago

在generate_data 文件关于天工的数据处理的时候，新版没有加上标准的终止符<|im_end|>，这个是漏写了还是故意的？

jiahe7ay commented 7 months ago

已改