search

ahmia / ahmia-index

Ahmia's elasticsearch index
BSD 3-Clause "New" or "Revised" License
58 stars 32 forks source link

The index would use some deduplication #15

Closed samip5 closed 7 months ago

samip5 commented 9 months ago

If you have found the same site multiple times and crawled it, but each time different domains, it should get deduplicated.