Add scripts for converting memo

[TTTTao725]

Prerequisite: # Make sure you have git-lfs installed (https://git-lfs.com) git lfs install git clone https://huggingface.co/datasets/MiMe-MeMo/Corpus-v1.1

[TTTTao725]

Hi guys, I changed the commit to make it more readable, please check it out :)

[TTTTao725]

Hi guys, please check out the scripts for infomedia (converting and filtering)

unfiltered, filtered datasets and the list of empty files in the original dataset, all of them are here: dfm-data/pre-training/danews2.0

[KennethEnevoldsen]

@TTTTao725 what is the number of tokens without information in infomedia?

[TTTTao725]

6578205491 / 6753608085 tokens after filtering

[KennethEnevoldsen]

Why the slash?

[TTTTao725]

that's just the number before filtering :)

[KennethEnevoldsen]

so Infomedia is ~175m tokens

[TTTTao725]

Nope, it's 6578205491, we still got a lot after filtering

[TTTTao725]

Sorry for the confusion, I like to write it like a processing bar 😹

[TTTTao725]

Oh I forgot to say, it does not include titles, summaries, etc, just text. So basically I just sum over the 'WordCount' instead of using a tokenizer which would be super slow

[TTTTao725]

So it's actually more than 6578205491 :)

[KennethEnevoldsen]

Ah so information is ~175m tokens

[KennethEnevoldsen]

@TTTTao725 feel free to merge whenever you are ready

[TTTTao725]

I can't merge it Kenneth, merging is block

[KennethEnevoldsen]

just check the box and merge