search

tulip / oplogtoredis

Publish MongoDB oplog entries to Redis
Apache License 2.0
38 stars 9 forks source link

Oplogtoredis: Add Configurable Denylist to HTTP Server #64

Closed alex-goodisman closed 4 months ago

alex-goodisman commented 5 months ago

This adds a new feature- a configurable denylist.

The denylist is a list of rules. Each rule is a list of keys and a regex.

Every incoming oplog message will be checked against every rule. The keys will be used sequentially to index into the oplog message, and if the value found is a string, it will be checked against the regex. If the regex is found for any rule in the denylist, the oplog message will be skipped and will not be reported to Redis.

The denylist can be viewed by HTTP methods (GET /denylist and GET /denylist/:ruleID), and can be configured in the same manner (PUT /denylist and DELETE /denylist/:ruleID).

Default behavior is unchanged, since the denylist is empty.

This is useful when an external context knows that some oplog messages are not necessary to be transferred. The external context can inform Oplogtoredis via the denylist, and then monitor the internal state via the HTTP API.

aranair commented 5 months ago

kinda surprised that we're adding this funtionality (presumably for us to ... toggle off/on during some restores or something). Were the other options not possible?

alex-goodisman commented 5 months ago

kinda surprised that we're adding this funtionality (presumably for us to ... toggle off/on during some restores or something). Were the other options not possible?

A full declarative approach doesn't look possible. Hence there must be some kind of communication between our tooling and the service. Given that, this seemed like the least intrusive approach