Sentence deduplication output

[solene-evain]

Hi,

I started to use datatrove for deduplication. If I managed to understand the minhash_deduplication script, I've got difficulties understanding the outputs of sentence_deduplication.py.

All I obtain are 'intermediate', 'sent_dups' and 'sent_sigs' folders.

1/ 'sent_sigs' is supposed to contain a signature for each document. I've got 15 docs, and only 9 output folders in here, with 3 c4_sig files in each that I can't read.

2/ 'sent_dups' contains also 9 folders, where I've got 2 c4_dup files in each. what does these files contain extactly?

3/ where is the output of SentenceDedupFilter? The final stats seem to be okay : " Stats: {total: 15, doc_len: 259 [min=33, max=106, 64.75±35/doc], removed_sentences: 32 [min=2, max=5, 2.91±1/doc], original_sentences: 36 [min=2, max=5, 3.27±1/doc]}" but I can't exactly figure out how since I can't find any new version of the documents with the removed_sentences.

Could you provide any help on that? Thanks

[guipenedo]

Hi, can you share the script you used? normally you would add a Writer after the filter and choose yourself where to save the final output

[solene-evain]

Yes sure!

I used this script: https://github.com/huggingface/datatrove/blob/b5443d2b8ef473262bc97b3d7717a217b6eaf1f3/examples/sentence_deduplication.py

That I modified like this: `from datatrove.executor.base import PipelineExecutor from datatrove.executor.local import LocalPipelineExecutor from datatrove.pipeline.dedup import SentenceDedupFilter, SentenceDedupSignature, SentenceFindDedups from datatrove.pipeline.dedup.sentence_dedup import SentDedupConfig from datatrove.pipeline.extractors import Trafilatura from datatrove.pipeline.filters import GopherQualityFilter, LanguageFilter from datatrove.pipeline.readers import JsonlReader, WarcReader from datatrove.pipeline.writers.jsonl import JsonlWriter from datatrove.utils.typeshelper import Languages from datatrove.pipeline.writers.disk_base import DiskWriter

""" example on how to use sentence-deduplication. sentence-deduplication implements deduplication as in: https://jmlr.org/papers/v21/20-074.html 'To deduplicate the data set, we discarded all but one of any three-sentence span occurring more than once in the data set.'

to run deduplication we need to run three different pipelines, pipeline 1: implements usual extraction + quality filtering, it ends with SentenceDedupSignature, preprended by a writer. pipeline 2: implements only SentenceFindDedups pipeline 3: implements SentenceDedupFilter prepended by a reader of the same writer-kind used during stage 1. after the SentenceDedupFilter. """

modify sentence dedup hyper params here

sent_dedup_config = SentDedupConfig( n_sentences=2, split_sentences=True, # set to False to split on \n instead only_dedup_in_index=True, min_doc_words=1, )

FINDER_WORKERS = 10 # this will speed up/parallelize step 2

1. create a signature for each sentence in each doc

def run_example(): pipeline_1 = [

WarcReader(data_folder="warc/", limit=1000),

    JsonlReader(data_folder="./", paths_file="path_file.txt"),
    #Trafilatura(),
    #GopherQualityFilter(min_stop_words=0),
    #LanguageFilter(language_threshold=0.5, languages=(Languages.english,)),
    JsonlWriter("sd_out/intermediate/"),
    SentenceDedupSignature(output_folder="sd_out/sent_sigs/", config=sent_dedup_config, language=Languages.french, finder_workers=FINDER_WORKERS),
]

2. reads all the signatures and loads them to check for duplicates.

pipeline_2 = [SentenceFindDedups(data_folder="sd_out/sent_sigs/", output_folder="sd_out/sent_dups/", config=sent_dedup_config)]

3. reads a documentpipeline and removes duplicated sentences found before

pipeline_3 = [
    JsonlReader(data_folder="sd_out/intermediate/"),
    SentenceDedupFilter(data_folder="sd_out/sent_dups/", config=sent_dedup_config, language=Languages.french),
]

executor_1: PipelineExecutor = LocalPipelineExecutor(pipeline=pipeline_1, workers=4, tasks=4)

executor_2: PipelineExecutor = LocalPipelineExecutor(pipeline=pipeline_2, workers=1, tasks=FINDER_WORKERS)

executor_3: PipelineExecutor = LocalPipelineExecutor(pipeline=pipeline_3, workers=4, tasks=4)

print(executor_1.run())
print(executor_2.run())
print(executor_3.run())

if name == "main": run_example() `

[guipenedo]

You're indeed missing a writer after the filter:

pipeline_3 = [
JsonlReader(data_folder="sd_out/intermediate/"),
SentenceDedupFilter(data_folder="sd_out/sent_dups/", config=sent_dedup_config, language=Languages.french),
JsonlWriter(data_folder="sd_out/final_output/")
]

[solene-evain]

Thank you! Now I've got the keys to understand the deduplication process 🙏

[solene-evain]

Suggestion: maybe this should be added to the original script https://github.com/huggingface/datatrove/blob/b5443d2b8ef473262bc97b3d7717a217b6eaf1f3/examples/sentence_deduplication.py as it is missing there too!

[guipenedo]

Good catch, I've added it