Implement leader election

[aslafy-z]

The thread memory feature does not work well when using multiple replicas. Implementing leader election and forward inbound requests to the leader could solve this issue.

[mr-karan]

I think using redis would be an easier approach than implementing leader elections in this tiny simple app.

[h4wkmoon]

I think it is easier to add leaderelection, which is a kubernetes standard than add a redis client in your app, and make every query read or write to the redis. But I'm not a skilled dev. You can find example of that, in kyverno source code, for instance. From a "user" perpective, I would prefer that calert use election than having to deploy any additional redis pods. From an architecture perpective, one would want to run redis in ha, to avoid a spof. This is getting too heavy for calert.

If a spof is acceptable, then we can run calert on a single pod.