search

togethercomputer / RedPajama-Data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.
Apache License 2.0
4.53k stars 346 forks source link

About the final result #105

Open Jdemon233 opened 7 months ago

Jdemon233 commented 7 months ago

Hi,I have finished all pipeline, but i finally get the result a few .parquet files and don't find where my processed text data

mauriceweber commented 7 months ago

Hi @Jdemon233 ,

Can you provide some more context on the steps you have run and the params you used? The parquet files are produced by the deduplication scripts which take as input the zipped jsonl files which contain the text documents.

Jdemon233 commented 7 months ago

Hi @Jdemon233 ,

Can you provide some more context on the steps you have run and the params you used? The parquet files are produced by the deduplication scripts which take as input the zipped jsonl files which contain the text documents.

thanks, i resolved my problem,