getsentry / relay

Sentry event forwarding and ingestion service.
https://docs.sentry.io/product/relay/
Other
319 stars 92 forks source link

Redis: Implement double write client #3900

Open Dav1dde opened 1 month ago

Dav1dde commented 1 month ago

For lossless Redis migration we will need to make it possible for Relay to double write quotas into two separate clusters. It should be possible to configure a 'double write' Redis client, with a primary and secondary (or multiple secondaries), the client only reads from the primary but writes to all configured Redis instances.

Sketeched implementation:

Config:

redis:
    a_good_namespace_for_multi_write:
    - url: redis://redis1 # primary
    - url: redis://redis2

# for the multi config
redis:
    project_configs:
        a_good_namespace_for_multi_write:
        - url: redis://redis1 # primary
        - url: redis://redis2
    misc: # No multi write
       url: redis://redis1 

Implementation:

There is already a Client Wrapper in relay-redis, implement a MultiWriteOrBetterNamedRedisClient which dispatches writes to all of it's clients but only reads from the primary. Writes should happen in concurrently (joined).

Error Behaviour:

Log errors of secondary nodes but not fail and only use the primary response.

Dav1dde commented 1 week ago

Planning this for early next week (~23.9.), ideally we'd like to separate the Redis cluster in the week after (~30.9.).