Option to use ngram overlap to dedupe paragraphs

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.

https://allenai.github.io/dolma/

Apache License 2.0

909 stars 94 forks source link

Closed rodneykinney closed 6 months ago

rodneykinney commented 7 months ago

Add optional by_ngram section to the dedupe.paragraph config object.

Constructs ngrams of a specified length and given stride. Fall back to whole-content deduping if the paragraph has fewer tokens than the ngram length

rodneykinney commented 7 months ago

Still need to fix test data

rodneykinney commented 7 months ago

Made an independent copy of the 000.json.gz input documents file, modified to accommodate the added test. Everything passing now