refractionPOINT / usp-adapters

LimaCharlie Universal Sensor Protocol (USP) Adapters to ingest from various sources.
https://limacharlie.io
Apache License 2.0
2 stars 2 forks source link

Deduping api #164

Closed maximelb closed 4 months ago

maximelb commented 4 months ago

Description of the change

Crreating a new interface for deduping and using it in O365. The new interface has a localDeduper built-in that will be used by default, but since it's an interface we will be able to plug in more thorough global deduping mechanisms in the cloud.

The interface takes a single string key to dedupe, it does NOT need to include some kind of partition (like an OID) since it is received per Adapter instance which is already single-tenant.

It uses a CheckAndAdd function so that the check-and-add-if-not-duplicate can be done atomically without further synchronization.

Type of change