tiki-archive / pool-kgraph

TIKI's Anonymous Contextual Knowledge Graph
MIT License
0 stars 1 forks source link

Determine how to quarantine data points less than epsilon #15

Closed sfaria27 closed 2 years ago

sfaria27 commented 2 years ago

Vertices should not be added to kgraph unless epsilon has been surpassed, however, it's an accumulation (so where/how do we start the accumulation)

Acceptance criteria:

  1. We don't add data to kgraph that is <epsilon
  2. When epsilon is surpassed, data points are added to kgraph
mike-audi commented 2 years ago

We want to create a circuit breaker pattern for data ingestion by edge. We do this by setting up an ingestion service that the app writes to, instead of the kgraph directly.

1) The service needs a table that maintains a list of all edges and a state of pass/fail. Pass being >= ε. Fail being < ε

2) When the service gets an edge that does not exist in its list we hash the fingerprint and store it in a secondary table, acting as a non-duplicative anonymous count.

3) When the service gets a edge that exists in the list but is fail and it's count < ε -1 we we hash the fingerprint and store it in a secondary table, increasing the non-duplicative anonymous count.

4) When the service gets a edge that exists in the list but is fail and it's count = ε -1 we clear the non-duplicative anonymous count and change the breaker state to pass. Then relay the edge downstream.

5) When the service gets a edge that exists in the list with a state of pass, relay the edge downstream.