search

larsga / Duke

Duke is a fast and flexible deduplication engine written in Java
Apache License 2.0
613 stars 194 forks source link

Why there are repeated matches? #238

Open wjfjw opened 7 years ago

wjfjw commented 7 years ago

Hello, I can successfully run the deduplicate code with Duke, but the matched two records repeat twice in the match result, why? For example, if records of ID1 and ID2 match, there will be two matches. The first one is ID1 and ID2, and the second one is ID2 and ID1. They are the same match despite of the different order. I am looking forward to your reply. Thank you!

Here is the XML Configuration code: ` <object class="no.priv.garshol.duke.comparators.Levenshtein" name="StringComparator">

0.8 ID URL StringComparator 0.49 0.9 TITLE StringComparator 0.3 0.7 SOURCE StringComparator 0.49 0.51 TIME CONTENT StringComparator 0.2 0.9

`

ibuda commented 5 years ago

This is only natural. When Duke analyses the batch with records ID1 and compares it to the record ID2, it spots that they are duplicates and save this result to output. Analogously, when it analyses the batch with record ID2 and compares it against the already indexed records ID2, in your case it also spots that they are duplicates. I mentioned "in your case", because not always the distances (comparators) you are using are metrics, i.e. they are not always symmetric. In one word, you will not always get that: Comparator(ID1, ID2) == Comparator(ID2, ID1) This is why some users get these repeated matches, and others don't.