Deduplication debug

[star-nox]

Debugging duplication logic while ingesting documents.

[lintrule-review[bot]]

You need to setup a payment method to use Lintrule

You can fix that by putting in a card here.

[star-nox]

Can you leave a comment with a high-level description of how dedup works?

I know it's content-based dedup, but some basic description would be great.

Yup!

Here's how duplicate detection works in document ingest:

• The duplicate check is done after text is extracted from the documents (file upload or web scrape).
• Supabase is queried based on s3_path (minus the 37 char UUID)/URL to check if the document already exists in the database.
• If it returns nothing, the incoming document is considered brand new and is ingested.
• If the query returns some documents, we check for an exact filename or URL match among the returned documents.
  1. If no exact filename match, incoming document is not a duplicate and is ingested
  2. If exact filename exists, the contents of older document and newer document are compared and below scenarios arise.
     • If the file contents match, the incoming document is a duplicate and is not ingested.
     • If they do not match, the incoming document is considered an updated file. the older document is removed and the incoming document is ingested.