search

splitio / split-synchronizer

Golang agent for Split SDKs
Other
16 stars 15 forks source link

Load spikes in Redis when opening dashboard #255

Closed kibertoad closed 8 months ago

kibertoad commented 9 months ago

Synchronizer currently uses "KEYS" Redis operation, which is strongly discouraged for production workloads. See https://redis.io/commands/keys/

Currently it is used here: https://github.com/splitio/split-synchronizer/blob/1cd638d015e3fd566937665c67da446c2006e20a/splitio/admin/controllers/helpers.go#L82

We are observing big load spikes in our Redis (which is not only used by Split.io) whenever anyone opens the split.io dashboard, and KEYS queries clearly flare up in our metrics.

At the very least KEYS should be replaced with SCAN, but ideally entire storage shouldn't be scanned at all - maybe some lookup maps could be utilized?

hbqdev commented 8 months ago

Hi @kibertoad Would it be possible for you to run the Synchronizer with its own Redis database? This is the way we recommend customers use Redis + Synchonizer.

The use of KEY operation was by design, currently changing this would imply breaking changes across SDKs, our team will need to evaluate the architecture, and right now we do not have information or estimation for that.

Regards