search

togethercomputer / RedPajama-Data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.
Apache License 2.0
4.43k stars 335 forks source link

What purpose cutoff.csv used in the cc_net pipeline? #106

Open kemalbastak opened 4 months ago

kemalbastak commented 4 months ago

I am trying to add turkish language (tr) to cutoff.csv on rp_v1 branch. There is few data on how the language score is calculated How do we add custom language score on this csv?

mauriceweber commented 4 months ago

Hi @kemalbastak

As far as I know, the CCNet pipeline does not support Turkish out of the box, but you can probably modify the pipeline to get it to support tr. We never went through that process, but to get there, I think you have to do the following steps:

  1. train your own reference wikipedia model (checkout the makefile and ccnet readme for that)
  2. collect the percentile statistics on the model distribution on the commoncrawl corpus
  3. add those statistics to the cutoff.csv
  4. run ccnet with turkish

I'd also recommend contacting the maintainers of the ccnet if there are issues related to that.

I hope this helps!

kemalbastak commented 4 months ago

I have calculated for 2023-50 CC dump and used 'perplexity' score on that data.

percentiles = {f"%{i}": np.percentile(all_pp_values, i) for i in range(1, 101)}

Got closer values with the existing languages in the cutoff.csv.

Thanks for the answer