search

ChenghaoMou / text-dedup

All-in-one text de-duplication
Apache License 2.0
620 stars 71 forks source link

Performance issues #105

Open bowspider-man opened 1 week ago

bowspider-man commented 1 week ago

I have a dataset of about 1 billion. My machine has a 64-core cpu and 128g memory. I found that with this configuration, I can only deduplicate 30 million data at a time. I would like to ask if there are any parameter settings that can allow me to deduplicate more data at a time while keeping my machine configuration unchanged.

ChenghaoMou commented 1 week ago

Which script are you using and how big is your dataset (physically)?

bowspider-man commented 1 week ago

Which script are you using and how big is your dataset (physically)?

I use this script: python -m text_dedup.minhash, the physical size of my dataset is 4.7T

ChenghaoMou commented 1 week ago

I see. Unfortunately, right now, none of the scripts were designed to be used for such a low-memory case directly.

You would need to refactor the script to process the data in partitions and in stages, and it will take longer time to finish. For example, you can read each parquet file iteratively, calculate the hashes, store the results. Then read the hashes to form the graph, maybe even use a database, then find the connected components on top of that.