data ingestion : skip equal docs to improve performance and cost

[asaf]

Compare content with same external id, if equal skip re-index.

I would suggest to make the external_id along with its apps unique but im not sure it's possible as apps are in meta.