Improve duplicate image detection

teolemon commented 1 year ago

What

Improve duplicate image detection to detect duplicates with byte size difference that are not perceivable.
We should have a tolerance threshold to be defined (1%, 5%)
We should also backprocess the 2 last years of edits by yuka
Why
We lack disk space, and since Yuka (mostly) has had this behaviour for at least 2 years, there are sizeable savings to be made.

https://world.openfoodfacts.org/product/3173289105708/quadro https://world.openfoodfacts.org/images/products/317/328/910/5708/2.jpg https://world.openfoodfacts.org/images/products/317/328/910/5708/4.jpg

Who for

Part of

1627

teolemon commented 1 year ago

https://openfoodfacts.slack.com/archives/CDC9K6RFT/p1684863284900529

github-actions[bot] commented 11 months ago

This issue has been open 90 days with no activity. Can you give it a little love by linking it to a parent issue, adding relevant labels and projets, creating a mockup if applicable, adding code pointers from https://github.com/openfoodfacts/openfoodfacts-server/blob/main/.github/labeler.yml, giving it a priority, editing the original issue to have a more comprehensive description… Thank you very much for your contribution to 🍊 Open Food Facts

github-actions[bot] commented 8 months ago

This issue has been open 90 days with no activity. Can you give it a little love by linking it to a parent issue, adding relevant labels and projets, creating a mockup if applicable, adding code pointers from https://github.com/openfoodfacts/openfoodfacts-server/blob/main/.github/labeler.yml, giving it a priority, editing the original issue to have a more comprehensive description… Thank you very much for your contribution to 🍊 Open Food Facts

teolemon commented 4 months ago

Potentially

Better detect image reuploads
https://pypi.org/project/ImageHash/

openfoodfacts / openfoodfacts-server