DCLM Style Deduplications - Githubissues

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.

https://allenai.github.io/dolma/

Apache License 2.0

972 stars 107 forks source link

DCLM Style Deduplications #214

Open revbucket opened 1 month ago

revbucket commented 1 month ago

General updates to the dedupe command to do deduplication using a joint paragraph/document flow in the same way that DCLM does.

Nuanced update list: Bloom Filter updates:

Used better binary search to get optimal BF size
Initialize Bloom Filters with multicore parallelism in mind

Deduper updates:

Switched out threadpool for rayon (cleaner, but equivalently performant)
Added optional read/write of bloom filter file (usually it's not necessary to save this, right?)
Made the main rust fxn more modular, easier to add in different types of dedupe methods
Logged some after-dedupe stats: {sparsity, removal rate}
Added DCLM style deduplication

Other stuff:

Modified the dedupe config to want a "dedupe.dedupe_method" attribute to specify which type {documents, paragraphs, dclm} of deduplication we do
Updated tutorial/etc to include this modified config^