search

StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.87k stars 1.51k forks source link

Configuration approaches with Redis Sentinel (3 nodes) in Kubernetes and using StackExchange.Redis in an asp .net core #1582

Closed RoyKimYYZ closed 3 years ago

RoyKimYYZ commented 3 years ago

Anybody have success with setting up Redis Sentinel (3 nodes) in Kubernetes and using StackExchange.Redis in an asp .net core app? I like to see working example of AppSettings.json with the hosts/endpoints. I deployed Redis Sentinel using the bitnami helm chart.

emi662002 commented 3 years ago

Anybody have success with setting up Redis Sentinel (3 nodes) in Kubernetes and using StackExchange.Redis in an asp .net core app? I like to see working example of AppSettings.json with the hosts/endpoints. I deployed Redis Sentinel using the bitnami helm chart.

I'm strugglig with this as well, how far have you come?

emi662002 commented 3 years ago

Anybody have success with setting up Redis Sentinel (3 nodes) in Kubernetes and using StackExchange.Redis in an asp .net core app? I like to see working example of AppSettings.json with the hosts/endpoints. I deployed Redis Sentinel using the bitnami helm chart.

this is the correct way: (see my comment below) https://github.com/StackExchange/StackExchange.Redis/issues/1534

Vandersteen commented 3 years ago

I used the following connectionString:

cache-sentinel-dev-redis.cache.svc.cluster.local:26379,serviceName=mymaster

Basicly:

<service>.<namespace>.svc.cluster.local:26379,serviceName=mymaster

Do note that I only messed around a little with sentinel, and that this is not production tested. In my tests (rolling restarts of sentinels, caches, nodes, ...) I found that adding the following did help in keeping it working / stable (as the pods got new ip's around):

_connection = await ConnectionMultiplexer.ConnectAsync(_options.Value.Uri).ConfigureAwait(false);
_connection.ConnectionFailed += ConnectionOnConnectionFailed;

...
        private void ConnectionOnConnectionFailed(object sender, ConnectionFailedEventArgs e)
        {
            if (e.FailureType == ConnectionFailureType.AuthenticationFailure)
                return;

            _logger.LogWarning(e.Exception, "Connection issue encountered for endpoint: {0}, failure type: {1}, connection type: {2}", e.EndPoint.ToString(), e.FailureType.ToString(), e.ConnectionType.ToString());

            _connection.Configure();
        }
emi662002 commented 3 years ago

I used the following connectionString:

cache-sentinel-dev-redis.cache.svc.cluster.local:26379,serviceName=mymaster

Basicly:

<service>.<namespace>.svc.cluster.local:26379,serviceName=mymaster

Do note that I only messed around a little with sentinel, and that this is not production tested. In my tests (rolling restarts of sentinels, caches, nodes, ...) I found that adding the following did help in keeping it working / stable (as the pods got new ip's around):

_connection = await ConnectionMultiplexer.ConnectAsync(_options.Value.Uri).ConfigureAwait(false);
_connection.ConnectionFailed += ConnectionOnConnectionFailed;

...
        private void ConnectionOnConnectionFailed(object sender, ConnectionFailedEventArgs e)
        {
            if (e.FailureType == ConnectionFailureType.AuthenticationFailure)
                return;

            _logger.LogWarning(e.Exception, "Connection issue encountered for endpoint: {0}, failure type: {1}, connection type: {2}", e.EndPoint.ToString(), e.FailureType.ToString(), e.ConnectionType.ToString());

            _connection.Configure();
        }

well I guess I will give this a try. Do you have any fast way to simulate those rolling restarts of sentinels? (I'm using redis-ha in my k8s server) see https://github.com/StackExchange/StackExchange.Redis/issues/1584

Vandersteen commented 3 years ago

You can do that with:

kubectl rollout restart deployment/xxxx
kubectl rollout restart daemonset/xxxx
kubectl rollout restart statefulset/xxxx
...

Or do a rolling restart of your nodes

Or simulate a failover https://redis.io/topics/sentinel#testing-the-failover

emi662002 commented 3 years ago

You can do that with:

kubectl rollout restart deployment/xxxx
kubectl rollout restart daemonset/xxxx
kubectl rollout restart statefulset/xxxx
...

Or do a rolling restart of your nodes

Or simulate a failover https://redis.io/topics/sentinel#testing-the-failover

Amazing! I used rollout of the statefulset, failover from redis-cli didn't work for me.

The fix seems to work, you saved the day!

bedankt ;)

emi662002 commented 3 years ago

@RoyKimYYZ did you come any further? I'm still having the issue even with the event. I'm thinking of recreating the Connection when a failure in the readinessprobe is detected. Doesn't feel right though.

koo9 commented 3 years ago

@emi662002 do you mind sharing the yaml files? thx

emi662002 commented 3 years ago

@emi662002 do you mind sharing the yaml files? thx

What are you looking for?, I don't think yamls are relevant to this issue but I'll see if I can help.

koo9 commented 3 years ago

@emi662002 I try connecting to redis with the connection scheme but no luck. wonder what I?did wrong. the port I used is the redis port 6379. the serviceName=mymaster, the client will get a connection to sentinel but I saw in the log that first the connection is established but then the next line says redishubmanager cannot connect to redis. any idea why?

koo9 commented 3 years ago

StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: PUBLISH DTH.Management.API.Hub.ManagementEventHub:group:sale-3d194d9c-6767-4dd0-b80d-f994edc50d10; │ │ ---> StackExchange.Redis.RedisConnectionException: UnableToConnect on redis.redis.svc.cluster.local:5000,serviceName=mymaster:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConn │ │ at StackExchange.Redis.TaskExtensions.TimeoutAfter(Task task, Int32 timeoutMs) in C:\projects\stackexchange-redis\src\StackExchange.Redis\TaskExtensions.cs:line 48 │ │ at StackExchange.Redis.ConnectionMultiplexer.WaitAllIgnoreErrorsAsync(Task[] tasks, Int32 timeoutMilliseconds, TextWriter log, String caller, Int32 callerLineNumber) in C:\projects\stackexchange-r │ │ --- End of inner exception stack trace --- │ │ at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource1 source, Exception unthrownException) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultipl │ │ --- End of stack trace from previous location --- │ │ at Microsoft.AspNetCore.SignalR.StackExchangeRedis.RedisHubLifetimeManager1.PublishAsync(String channel, Byte[] payload) │ │ at DTH.Management.API.Hub.ManagementEventHub.ChaChing(TotalToDateDto totalToDate) in /app/DTH.Management.API/Hub/ManagementEventHub.cs:line 26 │ │ at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.ExecuteMethod(ObjectMethodExecutor methodExecutor, Hub hub, Object[] arguments) │ │ at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.<>c__DisplayClass16_0.<g__ExecuteInvocation|0>d.MoveNext()

see the error message above. text in bold. redis.redis.svc.cluster.local:5000,serviceName=mymaster.

Question is: is it the sentinel service name or the redis service name and port?

emi662002 commented 3 years ago

StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: PUBLISH DTH.Management.API.Hub.ManagementEventHub:group:sale-3d194d9c-6767-4dd0-b80d-f994edc50d10; │ │ ---> StackExchange.Redis.RedisConnectionException: UnableToConnect on redis.redis.svc.cluster.local:5000,serviceName=mymaster:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConn │ │ at StackExchange.Redis.TaskExtensions.TimeoutAfter(Task task, Int32 timeoutMs) in C:\projects\stackexchange-redis\src\StackExchange.Redis\TaskExtensions.cs:line 48 │ │ at StackExchange.Redis.ConnectionMultiplexer.WaitAllIgnoreErrorsAsync(Task[] tasks, Int32 timeoutMilliseconds, TextWriter log, String caller, Int32 callerLineNumber) in C:\projects\stackexchange-r │ │ --- End of inner exception stack trace --- │ │ at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource1 source, Exception unthrownException) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultipl │ │ --- End of stack trace from previous location --- │ │ at Microsoft.AspNetCore.SignalR.StackExchangeRedis.RedisHubLifetimeManager1.PublishAsync(String channel, Byte[] payload) │ │ at DTH.Management.API.Hub.ManagementEventHub.ChaChing(TotalToDateDto totalToDate) in /app/DTH.Management.API/Hub/ManagementEventHub.cs:line 26 │ │ at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.ExecuteMethod(ObjectMethodExecutor methodExecutor, Hub hub, Object[] arguments) │ │ at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.<>c__DisplayClass16_0.<g__ExecuteInvocation|0>d.MoveNext()

see the error message above. text in bold. redis.redis.svc.cluster.local:5000,serviceName=mymaster.

Question is: is it the sentinel service name or the redis service name and port?

6379 is the default redis port 26379 is the default sentinel port

your connection string should have this as redis server for example: redis-ha-dev-announce-0:26379,redis-ha-dev-announce-1:26379,redis-ha-dev-announce-2:26379

on the internals the connection multiplexer will ask a master from the sentinels and use that connection this is how you use it: (simplified version)

` var options = new ConfigurationOptions { AbortOnConnectFail = false, AllowAdmin = false, ClientName = this.ClientName, ConnectTimeout = 60000, Password = this.AccessKey, ConnectRetry = 5, ServiceName = "mymaster", Proxy = Proxy.None };

            foreach (var s in this.RedisServers.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries))
            {
                options.EndPoints.Add(s);
            }
            var sentinelConnection = ConnectionMultiplexer.SentinelConnect(this.RedisOptions);
            var masterConfig = new ConfigurationOptions { ServiceName = "mymaster" };
            this.connection = sentinelConnection.GetSentinelMasterConnection(masterConfig);

`