search

sentry-kubernetes / charts

Easily deploy Sentry on your Kubernetes Cluster
MIT License
1.07k stars 507 forks source link

High availability configuration #1500

Open maitredede opened 1 week ago

maitredede commented 1 week ago

Hello,

I am trying to craft a high availability values for installation. I have configured an external S3 filestore (minio), and I am trying to scale components to achieve minimal HA (1 node down) : 3 replicas for statefull components (the statefullsets), 2 for stateless (the deployments).

So far, kafka, zookeeper, clickhouse, memcached and rabbitmq, each one can be scaled to 3 easily (just had to add affinities).

For Redis, the chart used is redis with one master and many replicas. Do you think it can be changed to redis-cluster ? There is a simple image in the chart readme to explain the difference...

Also, is postgresql really necessary ? Can it be replaced with a cluster-ready alternative like cockroachdb or yugabyte ? (comparison of both). I saw that sentry >10 may use clickhouse instead of cockroachdb (here).

For stateless components, if they are scaled to more than one, do they behave correctly ? Or is there components that must not be scaled above one ?

Thanks :)

maitredede commented 1 week ago

I bump into clickhouse sharding issue : https://github.com/getsentry/snuba/issues/4897

I have applied workaround : https://github.com/getsentry/snuba/issues/4897#issuecomment-1909077805

nadecancode commented 6 days ago

It currently does not support clustered redis unfortunately, this is Sentry issue. When I tried to run Clustered Redis it looks like Sentry is unable to distinguish between read-only replicas and read-write leaders so it'll run into issues.