As I can see, the data of the corpus of cluecorpusSmall, zh-en translation data, csl, news commentary are structured a bit differently. Some corpus are plain line-by-line text, but some have lines split by tab and the columns have special relation with each other.
# 逐条纯文本
==> chinese_llama/CLUECorpusSmall.txt <==
130真是佩服这家店开这么久。尽管门面已经小了一圈，但还是开着不容易啊。我们不容易，老板也不容易。自助餐，你可以吃得比平时多，但决不能浪费。想吃回20元，那是不可能的，所以还是不要去了。菜真的 很一般，洗干净就好啦。什么都要另外付钱，一定要想好，别的不叫，只吃自助。
送货速度奇慢无比
==> chinese_llama/CLUECorpusSmall_bert.txt <==
【乐在搜索】第18期丨致微生活粉丝的一封信
致粉丝的一封信：初冬来了 秋却依依不舍 留下礼物 ——雨 滋润着大地万物 白驹过隙，乐仔与各位粉丝也一起走过了2个多月。
# tab分隔的多列文本。
# 这是 csl的论文数据，应该第一列是标题，第二列应该是摘要，其它列是论文关键词、学科等标签。
==> chinese_llama/csl_camera_readly.tsv <==
谷物联合收获机自动测产系统设计-基于变权分层激活扩散模型 为了使联合收割机具有自动测产功能，提出了一种基于变权分层激活扩散的产量预测误差剔除模型，并使用单片机设计了联合收获机测产系统。测产系统的主要功能是：在田间进行作业时，收割机可以测出当前的运行速度、收获面积及谷物的总体产量。数据的采集使用霍尔传感器和电容压力传感器，具有较高的精度。模拟信号的处理选用了 ADC0804差分式 A／D 转换芯片，可以有效地克服系统误差，数据传送到单片机处理中心，对每一次转换都进行一次判断，利用变权分层激活扩散模型剔除误差较大的数据，通过计算将数据最终在LCD显示屏进行显示。将系统应用在了收 割机上，通过测试得到了谷物产量的测量值，并与真实值进行比较，验证了系统的可靠性。 联合收割机_测产系统_变权分层_激活扩散 农业工程 工学
# 双语数据，中到英或者英到中，不同语言之间 tab 隔开
==> chinese_llama/news-commentary-v13-en-zh_sampled.txt <==
1929 or 1989? 1929年还是1989年?
PARIS – As the economic crisis deepens and widens, the world has been searching for historical analogies to help us understand what has been happening. 巴黎-随着经济危机不断加深和蔓延，整 个世界一直在寻找历史上的类似事件希望有助于我们了解目前正在发生的情况。
==> chinese_llama/news-commentary-v13-zh-en.txt <==
1929年还是1989年? 1929 or 1989?
巴黎-随着经济危机不断加深和蔓延，整个世界一直在寻找历史上的类似事件希望有助于我们了解目前正在发生的情况。 PARIS – As the economic crisis deepens and widens, the world has been searching for historical analogies to help us understand what has been happening.
In the doc, the instruction is to simply combined them into one txt file, so it means in the combined file, some lines are simple long sentences or paragraphs, while some are tab-split lines with each column has certain meaning. Is this expected?
Hi I am reading docs about the corpus used for pretraining stage:
As I can see, the data of the corpus of cluecorpusSmall, zh-en translation data, csl, news commentary are structured a bit differently. Some corpus are plain line-by-line text, but some have lines split by tab and the columns have special relation with each other.
In the doc, the instruction is to simply combined them into one txt file, so it means in the combined file, some lines are simple long sentences or paragraphs, while some are tab-split lines with each column has certain meaning. Is this expected?