Closed sfaria27 closed 2 years ago
We want to create a circuit breaker pattern for data ingestion by edge. We do this by setting up an ingestion service that the app writes to, instead of the kgraph directly.
1) The service needs a table that maintains a list of all edges and a state of pass/fail. Pass being >= ε. Fail being < ε
2) When the service gets an edge that does not exist in its list we hash the fingerprint and store it in a secondary table, acting as a non-duplicative anonymous count.
3) When the service gets a edge that exists in the list but is fail and it's count < ε -1 we we hash the fingerprint and store it in a secondary table, increasing the non-duplicative anonymous count.
4) When the service gets a edge that exists in the list but is fail and it's count = ε -1 we clear the non-duplicative anonymous count and change the breaker state to pass. Then relay the edge downstream.
5) When the service gets a edge that exists in the list with a state of pass, relay the edge downstream.
Vertices should not be added to kgraph unless epsilon has been surpassed, however, it's an accumulation (so where/how do we start the accumulation)
Acceptance criteria: