search

StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.87k stars 1.51k forks source link

Unable to connect to redis sentinel through kubernetes #2025

Open tomhobson opened 2 years ago

tomhobson commented 2 years ago

I've updated to the latest stackExchange Redis, this configuration used to work but now I've received this error.

I've removed commandMap as suggested in another issue. I've made sure all the ports are open.

Not sure what I'm doing wrong here... This is running in k8s

I've tried stepping through and debugging, however it looks like the Standalone type is just defaulted then I can't spot where it would be properly swapped to sentinel... If you could explain how this works to me then I might be able to debug this. Although If I've just made a numpty configuration mistake, let me know.

{"Defaults":{"AbortOnConnectFail":true,"AllowAdmin":false,"BacklogPolicy":{"QueueWhileDisconnected":true,"AbortPendingOnConnectionFailure":false},"CheckCertificateRevocation":true,"ConnectRetry":3,"ConnectTimeout":null,"CommandMap":null,"ConfigurationChannel":"__Booksleeve_MasterChanged","DefaultVersion":"3.0.0","KeepAliveInterval":"00:01:00","Proxy":0,"ReconnectRetryPolicy":null,"ResolveDns":false,"SyncTimeout":"00:00:05","TieBreaker":"__Booksleeve_TieBreak","ConfigCheckInterval":"00:01:00","ClientName":"mqube-auth-service-mqube-auth-service-56fcbb588c-b2nsh(SE.Redis-v2.5.43.42402)"},"AbortOnConnectFail":false,"AllowAdmin":false,"AsyncTimeout":10000,"UseSsl":false,"ChannelPrefix":{"IsNullOrEmpty":true},"CheckCertificateRevocation":true,"ClientName":null,"ConnectRetry":5,"CommandMap":{},"ConfigurationChannel":"__Booksleeve_MasterChanged","ConnectTimeout":10000,"DefaultDatabase":null,"DefaultVersion":"3.0.0","EndPoints":[{"Host":"redis-ha-announce-0","AddressFamily":0,"Port":26379},{"Host":"redis-ha-announce-1","AddressFamily":0,"Port":26379},{"Host":"redis-ha-announce-2","AddressFamily":0,"Port":26379}],"HighPrioritySocketThreads":true,"KeepAlive":60,"User":null,"Password":null,"PreserveAsyncOrder":false,"Proxy":0,"ReconnectRetryPolicy":{},"BacklogPolicy":{"QueueWhileDisconnected":true,"AbortPendingOnConnectionFailure":false},"ResolveDns":false,"ResponseTimeout":0,"ServiceName":"mymaster","SocketManager":null,"Ssl":false,"SslHost":null,"SslProtocols":null,"SyncTimeout":10000,"TieBreaker":"","WriteBuffer":0,"ConfigCheckSeconds":60}
Unhandled exception. StackExchange.Redis.RedisConnectionException: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone

We're using the https://github.com/DandyDeveloper/charts/tree/master/charts/redis-ha helm chart to deploy this.

NickCraver commented 2 years ago

When you say "used to work", what was different? A different library version? If so: does 2.2.88 connect successfully?

tomhobson commented 2 years ago

Hey @NickCraver, I mean we used to run it on 2.1.30, but that was in a different setup to now.

We have many services still running this configuration just fine, but that is more standard (injected and used in services, rather than used in the startup)

For more context I'm using this with https://github.com/AliBazzi/IdentityServer4.Contrib.RedisStore/blob/master/IdentityServer4.Contrib.RedisStore/Stores/PersistedGrantStore.cs

The redis setup (to make it inline with our other services) uses dependency injection and then I do a services.BuildProvider then pull the connection multiplexor from that, to pass it into the connection multiplexor parameter of the above package.

The error occurs before I pass it into that package, so I don't think it's that.

I tried v2.1.30, v2.2.88 and v2.5.43 and none of them are working.

Is there something happening under the hood with DI that I am not understanding properly?

NickCraver commented 2 years ago

Your config seems correct, at least it has Sentinel ports, but we're detecting Standalone. If you pass a logger to the connect method, we can see from that what it's detecting, or alternatively: connect to these endpoints and give me an INFO command - we need to see why the Sentinels don't appear to be Sentinels in your case.

JorTurFer commented 2 years ago

We have the same problem using redis-operator. Sometimes the connection is detected as Standalone and another times it's detected well. Nothing change in the redis cluster. This is the output from INFO command on each instance:

Redis Replica 0

# Server
redis_version:5.0.14
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:128ca9cb46c8c5e1
redis_mode:standalone
os:Linux 5.4.129-63.229.amzn2.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.3.1
process_id:1
run_id:6346cadb9a6b84d8582177d6abd43f8e489e252d
tcp_port:6379
uptime_in_seconds:798
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3700161
executable:/data/redis-server
config_file:/redis/redis.conf

# Clients
connected_clients:17
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

# Memory
used_memory:5711720
used_memory_human:5.45M
used_memory_rss:8732672
used_memory_rss_human:8.33M
used_memory_peak:5813400
used_memory_peak_human:5.54M
used_memory_peak_perc:98.25%
used_memory_overhead:2748470
used_memory_startup:791336
used_memory_dataset:2963250
used_memory_dataset_perc:60.22%
allocator_allocated:5789160
allocator_active:6377472
allocator_resident:9256960
total_system_memory:33190043648
total_system_memory_human:30.91G
used_memory_lua:100352
used_memory_lua_human:98.00K
used_memory_scripts:360
used_memory_scripts_human:360B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.10
allocator_frag_bytes:588312
allocator_rss_ratio:1.45
allocator_rss_bytes:2879488
rss_overhead_ratio:0.94
rss_overhead_bytes:-524288
mem_fragmentation_ratio:1.53
mem_fragmentation_bytes:3042920
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:33844
mem_clients_normal:418762
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:378
rdb_bgsave_in_progress:0
rdb_last_save_time:1647867043
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:1232896
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:415
total_commands_processed:9164
instantaneous_ops_per_sec:21
total_net_input_bytes:3858107
total_net_output_bytes:63545572
instantaneous_input_kbps:1.53
instantaneous_output_kbps:166.74
rejected_connections:0
sync_full:1
sync_partial_ok:1
sync_partial_err:0
expired_keys:239
expired_stale_perc:0.30
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:2400
keyspace_misses:383
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:292
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:2
slave0:ip=172.18.59.137,port=6379,state=online,offset=1805352207,lag=0
slave1:ip=172.18.50.187,port=6379,state=online,offset=1805351989,lag=0
master_replid:f836985efa8253328a259dd3f3a99d9dcd2abbc1
master_replid2:b9d60d21291814cdb534cc28cb18a666a36bb116
master_repl_offset:1805352888
second_repl_offset:1804563560
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1804304313
repl_backlog_histlen:1048576

# CPU
used_cpu_sys:0.650285
used_cpu_user:0.746188
used_cpu_sys_children:0.008539
used_cpu_user_children:0.053918

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=5071,expires=5070,avg_ttl=33884385

Redis Replica 1

# Server
redis_version:5.0.14
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:128ca9cb46c8c5e1
redis_mode:standalone
os:Linux 5.4.129-63.229.amzn2.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.3.1
process_id:1
run_id:825f2a747d695a5aff647ef9d307e5c8ae07f7f7
tcp_port:6379
uptime_in_seconds:883
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3700256
executable:/data/redis-server
config_file:/redis/redis.conf

# Clients
connected_clients:16
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

# Memory
used_memory:5629640
used_memory_human:5.37M
used_memory_rss:7983104
used_memory_rss_human:7.61M
used_memory_peak:5782232
used_memory_peak_human:5.51M
used_memory_peak_perc:97.36%
used_memory_overhead:2599324
used_memory_startup:791336
used_memory_dataset:3030316
used_memory_dataset_perc:62.63%
allocator_allocated:5688224
allocator_active:6242304
allocator_resident:9080832
total_system_memory:33190043648
total_system_memory_human:30.91G
used_memory_lua:40960
used_memory_lua_human:40.00K
used_memory_scripts:360
used_memory_scripts_human:360B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.10
allocator_frag_bytes:554080
allocator_rss_ratio:1.45
allocator_rss_bytes:2838528
rss_overhead_ratio:0.88
rss_overhead_bytes:-1097728
mem_fragmentation_ratio:1.43
mem_fragmentation_bytes:2416384
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:303524
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:716
rdb_bgsave_in_progress:0
rdb_last_save_time:1647867143
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:1122304
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:542
total_commands_processed:7600
instantaneous_ops_per_sec:9
total_net_input_bytes:3918683
total_net_output_bytes:1952604
instantaneous_input_kbps:0.92
instantaneous_output_kbps:1.08
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:8
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:362
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:slave
master_host:172.18.41.81
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:1805502910
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:f836985efa8253328a259dd3f3a99d9dcd2abbc1
master_replid2:b9d60d21291814cdb534cc28cb18a666a36bb116
master_repl_offset:1805502910
second_repl_offset:1804563560
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1804454335
repl_backlog_histlen:1048576

# CPU
used_cpu_sys:0.657987
used_cpu_user:0.750800
used_cpu_sys_children:0.016222
used_cpu_user_children:0.044535

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=5070,expires=5069,avg_ttl=0

Redis Replica 2

# Server
redis_version:5.0.14
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:128ca9cb46c8c5e1
redis_mode:standalone
os:Linux 5.4.129-63.229.amzn2.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.3.1
process_id:1
run_id:7d4b4362272e6e675f3d8c367a085f8d99741300
tcp_port:6379
uptime_in_seconds:1131
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3700566
executable:/data/redis-server
config_file:/redis/redis.conf

# Clients
connected_clients:18
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0

# Memory
used_memory:5724264
used_memory_human:5.46M
used_memory_rss:8179712
used_memory_rss_human:7.80M
used_memory_peak:6009600
used_memory_peak_human:5.73M
used_memory_peak_perc:95.25%
used_memory_overhead:2666964
used_memory_startup:791336
used_memory_dataset:3057300
used_memory_dataset_perc:61.98%
allocator_allocated:5742768
allocator_active:6324224
allocator_resident:9224192
total_system_memory:33190043648
total_system_memory_human:30.91G
used_memory_lua:74752
used_memory_lua_human:73.00K
used_memory_scripts:360
used_memory_scripts_human:360B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.10
allocator_frag_bytes:581456
allocator_rss_ratio:1.46
allocator_rss_bytes:2899968
rss_overhead_ratio:0.89
rss_overhead_bytes:-1044480
mem_fragmentation_ratio:1.44
mem_fragmentation_bytes:2518352
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:370140
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:653
rdb_bgsave_in_progress:0
rdb_last_save_time:1647867506
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:1040384
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:819
total_commands_processed:14124
instantaneous_ops_per_sec:8
total_net_input_bytes:17148051
total_net_output_bytes:5728228
instantaneous_input_kbps:0.62
instantaneous_output_kbps:2.26
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:91
keyspace_misses:688
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:283
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:slave
master_host:172.18.41.81
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:1805977846
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:f836985efa8253328a259dd3f3a99d9dcd2abbc1
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1805977846
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1804929271
repl_backlog_histlen:1048576

# CPU
used_cpu_sys:1.080469
used_cpu_user:0.889569
used_cpu_sys_children:0.007267
used_cpu_user_children:0.080729

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=5086,expires=5085,avg_ttl=0
JorTurFer commented 2 years ago

This is another redis cluster deployed also with that operator where we have the same problem:

Redis Replica 0:

# Server
redis_version:5.0.14
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:128ca9cb46c8c5e1
redis_mode:standalone
os:Linux 5.4.149-73.259.amzn2.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.3.1
process_id:1
run_id:414b722c518e9bbdd69ff4f0522f0507fec2413d
tcp_port:6379
uptime_in_seconds:12061
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3700840
executable:/data/redis-server
config_file:/redis/redis.conf

# Clients
connected_clients:7
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

# Memory
used_memory:2060640
used_memory_human:1.97M
used_memory_rss:4198400
used_memory_rss_human:4.00M
used_memory_peak:2236496
used_memory_peak_human:2.13M
used_memory_peak_perc:92.14%
used_memory_overhead:2008132
used_memory_startup:791336
used_memory_dataset:52508
used_memory_dataset_perc:4.14%
allocator_allocated:2061256
allocator_active:2326528
allocator_resident:5046272
total_system_memory:16594845696
total_system_memory_human:15.46G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:31457280
maxmemory_human:30.00M
maxmemory_policy:volatile-ttl
allocator_frag_ratio:1.13
allocator_frag_bytes:265272
allocator_rss_ratio:2.17
allocator_rss_bytes:2719744
rss_overhead_ratio:0.83
rss_overhead_bytes:-847872
mem_fragmentation_ratio:2.10
mem_fragmentation_bytes:2200664
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:16922
mem_clients_normal:151226
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1647859870
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:393216
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:4946
total_commands_processed:118376
instantaneous_ops_per_sec:2
total_net_input_bytes:6319137
total_net_output_bytes:7902417
instantaneous_input_kbps:0.14
instantaneous_output_kbps:0.18
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:1
keyspace_misses:7
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:276
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:1
slave0:ip=172.16.58.46,port=6379,state=online,offset=845470,lag=0
master_replid:292594df0b6ade54f6fa664e2d2a0eefb448409d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:845470
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:845470

# CPU
used_cpu_sys:9.644004
used_cpu_user:9.176633
used_cpu_sys_children:0.001046
used_cpu_user_children:0.000790

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1,expires=0,avg_ttl=0

Redis Replica 1:

# Server
redis_version:5.0.14
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:128ca9cb46c8c5e1
redis_mode:standalone
os:Linux 5.4.149-73.259.amzn2.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.3.1
process_id:1
run_id:68bd5c2473705be9b708001ea503ed1b581d3ae3
tcp_port:6379
uptime_in_seconds:12101
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3700879
executable:/data/redis-server
config_file:/redis/redis.conf

# Clients
connected_clients:8
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

# Memory
used_memory:2113248
used_memory_human:2.02M
used_memory_rss:4108288
used_memory_rss_human:3.92M
used_memory_peak:2197416
used_memory_peak_human:2.10M
used_memory_peak_perc:96.17%
used_memory_overhead:2040904
used_memory_startup:791336
used_memory_dataset:72344
used_memory_dataset_perc:5.47%
allocator_allocated:2101440
allocator_active:2371584
allocator_resident:5021696
total_system_memory:16594845696
total_system_memory_human:15.46G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:31457280
maxmemory_human:30.00M
maxmemory_policy:volatile-ttl
allocator_frag_ratio:1.13
allocator_frag_bytes:270144
allocator_rss_ratio:2.12
allocator_rss_bytes:2650112
rss_overhead_ratio:0.82
rss_overhead_bytes:-913408
mem_fragmentation_ratio:2.04
mem_fragmentation_bytes:2098912
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:200920
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1647859870
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:397312
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:7363
total_commands_processed:116965
instantaneous_ops_per_sec:1
total_net_input_bytes:6839306
total_net_output_bytes:10278602
instantaneous_input_kbps:0.10
instantaneous_output_kbps:0.14
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:6
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:234
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:slave
master_host:172.16.40.110
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:848205
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:292594df0b6ade54f6fa664e2d2a0eefb448409d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:848205
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:848205

# CPU
used_cpu_sys:8.825351
used_cpu_user:8.711314
used_cpu_sys_children:0.001062
used_cpu_user_children:0.000000

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1,expires=0,avg_ttl=0
JorTurFer commented 2 years ago

I'll try to replicate the problem in one testing cluster and debug internals.

JorTurFer commented 2 years ago

@tomhobson are you setting ConnectTimeout? In my case, the value was too low and that produced the wrong detection because it doesn't detect the server type

blattersturm commented 2 years ago

We're having the same issue since moving up from 2.2.20 to 2.5.61 using the same redis-ha chart.

As shown in the second log line, it seems to be that our configuration somehow resolves to 6379 as the same DNS name returns multiple hosts, despite us specifying 26379:

        public readonly static ConfigurationOptions RedisConfig = new ConfigurationOptions()
        {
            ServiceName = "mymaster",
            EndPoints =
            {
                { "redis-sentinel", 26379 }
            },
            TieBreaker = "",
            DefaultVersion = new Version(6, 0, 0),
            KeepAlive = 10,
            ResolveDns = true,
            ConnectRetry = 100,
            SyncTimeout = 5000,
            AllowAdmin = true,

nslookup output:

redis-sentinel.lambda.svc.cluster.local canonical name = redis-ha.redis-ha.svc.cluster.local

Name:   redis-ha.redis-ha.svc.cluster.local
Address: 10.42.1.139
Name:   redis-ha.redis-ha.svc.cluster.local
Address: 10.42.11.145
Name:   redis-ha.redis-ha.svc.cluster.local
Address: 10.42.15.148
Name:   redis-ha.redis-ha.svc.cluster.local
Address: 10.42.16.126
Name:   redis-ha.redis-ha.svc.cluster.local
Address: 10.42.5.40

Log entries:

18:55:41.0975: Connecting (sync) on .NET 5.0.16 (StackExchange.Redis: v2.5.61.22961)
18:55:41.0996: 10.43.104.6:6379,10.43.242.156:6379,10.43.1.229:6379,10.43.164.123:6379,10.43.113.244:6379,serviceName=mymaster,keepAlive=10,syncTimeout=5000,allowAdmin=True,version=6.0.0,resolveDns=True,connectRetry=100,defaultDatabase=5
18:55:41.0998: 10.43.104.6:6379/Interactive: Connecting...
18:55:41.0999: 10.43.104.6:6379: BeginConnectAsync
18:55:41.1001: 10.43.104.6:6379/Interactive: Connected
18:55:41.1002: 10.43.104.6:6379: Server handshake
18:55:41.1002: 10.43.104.6:6379: Auto-configuring...
18:55:41.1002: 10.43.242.156:6379/Interactive: Connecting...
18:55:41.1004: 10.43.242.156:6379: BeginConnectAsync
18:55:41.1004: 10.43.104.6:6379: Sending critical tracer (handshake): PING
18:55:41.1005: 10.43.104.6:6379/Interactive: Writing: PING
18:55:41.1006: 10.43.104.6:6379: Flushing outbound buffer
18:55:41.1006: 10.43.104.6:6379: OnEstablishingAsync complete
18:55:41.1006: 10.43.104.6:6379: Starting read
18:55:41.1006: 10.43.104.6:6379: Auto-configured (INFO) role: primary
18:55:41.1006: 10.43.1.229:6379/Interactive: Connecting...
18:55:41.1006: 10.43.104.6:6379: Auto-configured (INFO) version: 6.2.7
18:55:41.1006: 10.43.104.6:6379: Auto-configured (INFO) server-type: standalone
18:55:41.1006: 10.43.242.156:6379/Interactive: Connected
18:55:41.1007: 10.43.242.156:6379: Server handshake
18:55:41.1007: 10.43.242.156:6379: Auto-configuring...
18:55:41.1007: 10.43.242.156:6379: Sending critical tracer (handshake): PING
18:55:41.1007: 10.43.242.156:6379/Interactive: Writing: PING
18:55:41.1007: 10.43.242.156:6379: Flushing outbound buffer
18:55:41.1008: 10.43.242.156:6379: OnEstablishingAsync complete
18:55:41.1008: 10.43.242.156:6379: Starting read
18:55:41.1008: 10.43.1.229:6379: BeginConnectAsync
18:55:41.1009: 10.43.242.156:6379: Auto-configured (INFO) role: replica
18:55:41.1010: 10.43.242.156:6379: Auto-configured (INFO) version: 6.2.7
18:55:41.1010: 10.43.242.156:6379: Auto-configured (INFO) server-type: standalone
18:55:41.1010: 10.43.164.123:6379/Interactive: Connecting...
18:55:41.1011: 10.43.1.229:6379/Interactive: Connected
18:55:41.1011: 10.43.1.229:6379: Server handshake
18:55:41.1011: 10.43.1.229:6379: Auto-configuring...
18:55:41.1011: Response from 10.43.104.6:6379/Interactive / PING: SimpleString: PONG
18:55:41.1012: Response from 10.43.242.156:6379/Interactive / PING: SimpleString: PONG
18:55:41.1012: 10.43.1.229:6379: Sending critical tracer (handshake): PING
18:55:41.1012: 10.43.1.229:6379/Interactive: Writing: PING
18:55:41.1012: 10.43.1.229:6379: Flushing outbound buffer
18:55:41.1012: 10.43.1.229:6379: OnEstablishingAsync complete
18:55:41.1012: 10.43.1.229:6379: Starting read
18:55:41.1012: 10.43.164.123:6379: BeginConnectAsync
18:55:41.1014: 10.43.113.244:6379/Interactive: Connecting...
18:55:41.1016: 10.43.113.244:6379: BeginConnectAsync
18:55:41.1018: 5 unique nodes specified (without tiebreaker)
18:55:41.1025: 10.43.113.244:6379/Interactive: Connected
18:55:41.1025: 10.43.113.244:6379: Server handshake
18:55:41.1025: 10.43.113.244:6379: Auto-configuring...
18:55:41.1025: 10.43.1.229:6379: Auto-configured (INFO) role: replica
18:55:41.1026: 10.43.1.229:6379: Auto-configured (INFO) version: 6.2.7
18:55:41.1026: 10.43.1.229:6379: Auto-configured (INFO) server-type: standalone
18:55:41.1026: 10.43.113.244:6379: Sending critical tracer (handshake): PING
18:55:41.1026: 10.43.113.244:6379/Interactive: Writing: PING
18:55:41.1026: 10.43.113.244:6379: Flushing outbound buffer
18:55:41.1026: 10.43.113.244:6379: OnEstablishingAsync complete
18:55:41.1026: 10.43.113.244:6379: Starting read
18:55:41.1026: Response from 10.43.1.229:6379/Interactive / PING: SimpleString: PONG
18:55:41.1028: 10.43.104.6:6379: OnConnectedAsync already connected start
18:55:41.1028: 10.43.113.244:6379: Auto-configured (INFO) role: replica
18:55:41.1028: 10.43.113.244:6379: Auto-configured (INFO) version: 6.2.7
18:55:41.1028: 10.43.113.244:6379: Auto-configured (INFO) server-type: standalone
18:55:41.1029: Response from 10.43.113.244:6379/Interactive / PING: SimpleString: PONG
18:55:41.1029: 10.43.164.123:6379/Interactive: Connected
18:55:41.1029: 10.43.164.123:6379: Server handshake
18:55:41.1029: 10.43.164.123:6379: Auto-configuring...
18:55:41.1031: 10.43.164.123:6379: Sending critical tracer (handshake): PING
18:55:41.1031: 10.43.164.123:6379/Interactive: Writing: PING
18:55:41.1032: 10.43.164.123:6379: Flushing outbound buffer
18:55:41.1032: 10.43.164.123:6379: OnEstablishingAsync complete
18:55:41.1032: 10.43.164.123:6379: Starting read
18:55:41.1037: 10.43.104.6:6379/Interactive: Writing: PING
18:55:41.1047: 10.43.242.156:6379: OnConnectedAsync already connected start
18:55:41.1047: 10.43.242.156:6379/Interactive: Writing: PING
18:55:41.1048: 10.43.1.229:6379: OnConnectedAsync already connected start
18:55:41.1048: 10.43.1.229:6379/Interactive: Writing: PING
18:55:41.1049: 10.43.164.123:6379: OnConnectedAsync init (State=ConnectedEstablishing)
18:55:41.1049: 10.43.113.244:6379: OnConnectedAsync already connected start
18:55:41.1049: 10.43.113.244:6379/Interactive: Writing: PING
18:55:41.1050: Allowing 5 endpoint(s) 00:00:05 to respond...
18:55:41.1053: Awaiting 5 available task completion(s) for 5000ms, IOCP: (Busy=0,Free=1000,Min=6,Max=1000), WORKER: (Busy=1,Free=32766,Min=6,Max=32767), POOL: (Threads=7,QueuedItems=1,CompletedItems=189)
18:55:41.1069: Response from 10.43.104.6:6379/Interactive / PING: SimpleString: PONG
18:55:41.1069: Response from 10.43.113.244:6379/Interactive / PING: SimpleString: PONG
18:55:41.1069: Response from 10.43.242.156:6379/Interactive / PING: SimpleString: PONG
18:55:41.1069: 10.43.164.123:6379: Auto-configured (INFO) role: replica
18:55:41.1069: 10.43.164.123:6379: Auto-configured (INFO) version: 6.2.7
18:55:41.1069: 10.43.164.123:6379: Auto-configured (INFO) server-type: standalone
18:55:41.1070: Response from 10.43.164.123:6379/Interactive / PING: SimpleString: PONG
18:55:41.1070: 10.43.164.123:6379: OnConnectedAsync completed (From command: PING)
18:55:41.1071: Response from 10.43.1.229:6379/Interactive / PING: SimpleString: PONG
18:55:41.1079: 10.43.242.156:6379: OnConnectedAsync already connected end
18:55:41.1080: 10.43.1.229:6379: OnConnectedAsync already connected end
18:55:41.1080: 10.43.104.6:6379: OnConnectedAsync already connected end
18:55:41.1080: 10.43.113.244:6379: OnConnectedAsync already connected end
18:55:41.1080: All 5 available tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=6,Max=1000), WORKER: (Busy=1,Free=32766,Min=6,Max=32767), POOL: (Threads=7,QueuedItems=0,CompletedItems=201)
18:55:41.1080: Endpoint summary:
18:55:41.1080: 10.43.104.6:6379: Endpoint is ConnectedEstablished
18:55:41.1080: 10.43.242.156:6379: Endpoint is ConnectedEstablished
18:55:41.1080: 10.43.1.229:6379: Endpoint is ConnectedEstablished
18:55:41.1080: 10.43.164.123:6379: Endpoint is ConnectedEstablished
18:55:41.1080: 10.43.113.244:6379: Endpoint is ConnectedEstablished
18:55:41.1080: 10.43.104.6:6379: Returned with success as Standalone primary (Source: Already connected)
18:55:41.1081: 10.43.242.156:6379: Returned with success as Standalone replica (Source: Already connected)
18:55:41.1081: 10.43.1.229:6379: Returned with success as Standalone replica (Source: Already connected)
18:55:41.1081: 10.43.164.123:6379: Returned with success as Standalone replica (Source: From command: PING)
18:55:41.1081: 10.43.113.244:6379: Returned with success as Standalone replica (Source: Already connected)
18:55:41.1081: Election summary:
18:55:41.1081: Election: Single primary detected: 10.43.104.6:6379
18:55:41.1081: 10.43.104.6:6379: Clearing as RedundantPrimary
18:55:41.1090: Endpoint Summary:
18:55:41.1097: 10.43.104.6:6379: Standalone v6.2.7, primary; keep-alive: 00:00:10; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active
18:55:41.1115: 10.43.104.6:6379: int ops=6, qu=0, qs=0, qc=0, wr=0, socks=1; sub ops=2, qu=0, qs=0, qc=0, wr=0, subs=1, socks=1
18:55:41.1131: 10.43.104.6:6379: Circular op-count snapshot; int: 0+6=6 (0.60 ops/s; spans 10s); sub: 0+2=2 (0.20 ops/s; spans 10s)
18:55:41.1132: 10.43.242.156:6379: Standalone v6.2.7, replica; keep-alive: 00:00:10; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active
18:55:41.1132: 10.43.242.156:6379: int ops=6, qu=0, qs=0, qc=0, wr=0, socks=1; sub ops=2, qu=0, qs=0, qc=0, wr=0, subs=1, socks=1
18:55:41.1132: 10.43.242.156:6379: Circular op-count snapshot; int: 0+6=6 (0.60 ops/s; spans 10s); sub: 0+2=2 (0.20 ops/s; spans 10s)
18:55:41.1132: 10.43.1.229:6379: Standalone v6.2.7, replica; keep-alive: 00:00:10; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active
18:55:41.1132: 10.43.1.229:6379: int ops=6, qu=0, qs=0, qc=0, wr=0, socks=1; sub ops=2, qu=0, qs=0, qc=0, wr=0, subs=1, socks=1
18:55:41.1132: 10.43.1.229:6379: Circular op-count snapshot; int: 0+6=6 (0.60 ops/s; spans 10s); sub: 0+2=2 (0.20 ops/s; spans 10s)
18:55:41.1132: 10.43.164.123:6379: Standalone v6.2.7, replica; keep-alive: 00:00:10; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active
18:55:41.1132: 10.43.164.123:6379: int ops=5, qu=0,Unhandled exception. StackExchange.Redis.RedisConnectionException: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone
blattersturm commented 2 years ago

Right. We seem to use .Clone() in a few places to reuse the same ConfigurationOptions object for multiple ConnectionMultiplexer instances (some need a specialized DefaultDatabase), but it seems as of current versions the endpoint collection will get overridden with the discovered Redis primary/replica hosts and not the sentinel itself, even when creating from a cloned object only.

fvogl commented 2 years ago

Is anyone aware of a workaround which makes it possible to use the library together with Sentinel?

lord-kyron commented 2 years ago

Hey guys, is this issue resolved in the latest stable release, or it is still ongoing? We are using release 2.5.43 and facing the same problem with the redis-ha helm chart.

zhaidarbek commented 2 years ago

Had the same issue yesterday. Redeployed both application and RedisFailover (i'm using spotahome operator) but it didn't help. However reboot of the whole cluster (restarted all ec2 nodes) with subsequent redeploy of application and RedisFailover resolved the issue.

zhaidarbek commented 2 years ago

Update: this happened again, but this time i changed redis host from service fqdn to clusterIP and just redeployed the app. So i guess my issue is similar to this https://github.com/StackExchange/StackExchange.Redis/issues/1015

NickCraver commented 1 year ago

Hey all - I missed updating this issue in the list but this should be improved or resolved by changes in 2.6.66 (specifically #2242)

KendoSai commented 1 year ago

Hi @NickCraver, I have just upgraded my StackExchange.Redis from 2.1.30 to 2.6.80 and I also got the same error messsage

StackExchange.Redis.RedisConnectionException: 'Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone'

Here is my configuration and it's working perfectly with the 2.1.30 version. Do you have any ideas?

var readLazyConnection = new Lazy<ConnectionMultiplexer>(() =>
{
    ConfigurationOptions sentinelConfig = new ConfigurationOptions();
    sentinelConfig.TieBreaker = "";
    sentinelConfig.CommandMap = CommandMap.Sentinel;
    sentinelConfig.AbortOnConnectFail = false;
    sentinelConfig.SyncTimeout = 3000;
    sentinelConfig.Ssl = false;
    sentinelConfig.ConnectRetry = 3;
    sentinelConfig.KeepAlive = 180;
    sentinelConfig.ServiceName = instanceName;

    foreach (var item in connectionString.Split(';'))
    {
        sentinelConfig.EndPoints.Add(item);
    }
    ConnectionMultiplexer sentinelConnection = ConnectionMultiplexer.Connect(sentinelConfig);

    ConfigurationOptions masterConfig = new ConfigurationOptions();
    masterConfig.ServiceName = "mymaster";

    ConnectionMultiplexer masterConnection = sentinelConnection.GetSentinelMasterConnection(masterConfig);

    return masterConnection;
});
NickCraver commented 1 year ago

@KendoSai remove the CommandMap there - it's not needed and will cause issues in later versions where it's automated :)

KendoSai commented 1 year ago

@NickCraver : I removed the CommandMap but still got the same error. My connectionString is kind of 172.16.6.200:26379;172.16.6.201:26379;172.16.6.202:26379

NickCraver commented 1 year ago

@KendoSai you have 2 connect calls here - what are you trying to do overall?

To try and help clarify, the original call here:

ConnectionMultiplexer sentinelConnection = ConnectionMultiplexer.Connect(sentinelConfig);

Will return a connection to the servers that sentinel controls, not to Sentinel. It handles connecting to Sentinel, getting the connection of the current primary, returning that, and monitoring Sentinel for when that primary changes. I'm pointing that out because the name sentinelConnection implies a different assumption, more correctly it would be:

ConnectionMultiplexer currentPrimaryServerConnection = ConnectionMultiplexer.Connect(sentinelConfig);
KendoSai commented 1 year ago

Hi @NickCraver, thank you for clarifying that. I have managed to make my application works fine with StackExchange.Redis 2.6.80 by changing ConnectionMultiplexer.Connect(sentinelConfig); to ConnectionMultiplexer.SentinelConnect(sentinelConfig);

ConnectionMultiplexer currentPrimaryServerConnection = ConnectionMultiplexer.SentinelConnect(sentinelConfig);
ConfigurationOptions masterConfig = new ConfigurationOptions();
masterConfig.ServiceName = "mymaster";
ConnectionMultiplexer masterConnection = sentinelConnection.GetSentinelMasterConnection(masterConfig);
cuong21951 commented 1 year ago

I have the same problem when restart the service. My connection is using like public RedisClientWraper(string host, string pass, string serviceName = "") { this.Host = host; this.Pass = pass; this.ServiceName = serviceName;

        var sentinelConfig = $"{host},abortConnect=false";
        ConnectionMultiplexer currentPrimaryServerConnection = ConnectionMultiplexer.SentinelConnect(sentinelConfig);
        ConfigurationOptions masterConfig = new ConfigurationOptions();
        masterConfig.ServiceName = "mymaster";
        masterConfig.Password = pass;
        masterConfig.AbortOnConnectFail = false;
        redis = currentPrimaryServerConnection.GetSentinelMasterConnection(masterConfig);
        db = redis.GetDatabase();
daviddahan01 commented 1 year ago

Hi @cuong21951 , I have the same issue. have you solved it ? still getting "Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone" using V 2.6.9

var options = new ConfigurationOptions { AbortOnConnectFail = false, AllowAdmin = true, ClientName = "MapAPI", ConnectTimeout = 5000, // Password = this.AccessKey, ConnectRetry = 10, ServiceName = "mymaster", Proxy = Proxy.None, Ssl = false,
KeepAlive = 10, ResolveDns = true,
SyncTimeout = 5000,

            };

            foreach (var s in redisConfig.SentinelHosts)
            {
                options.EndPoints.Add(s);
            }
            var sentinelConnection = ConnectionMultiplexer.SentinelConnect(options);
            var masterConfig = new ConfigurationOptions { ServiceName = "mymaster" };
            var connection = sentinelConnection.GetSentinelMasterConnection(masterConfig);

            connection.ConnectionFailed += Connection_ConnectionFailed;
            connection.ConfigurationChanged += Conn_ConfigurationChanged;
            connection.ConnectionRestored += Connection_ConnectionRestored;

My service is running in docker . telnet to all sentinels and redis instances are ok. Thanks.

NickCraver commented 1 year ago

@daviddahan01 This is handled for you by the multiplexer, you need only connect to the sentinel instances to get a connection to the current primary standalone instance, so instead of this:

            var sentinelConnection = ConnectionMultiplexer.SentinelConnect(options);
            var masterConfig = new ConfigurationOptions { ServiceName = "mymaster" };
            var connection = sentinelConnection.GetSentinelMasterConnection(masterConfig);

You only need:

            var connection = ConnectionMultiplexer.Connect(options);
daviddahan01 commented 1 year ago

Hi @NickCraver , thanks for reply. I Modified it, got the same issue.......

Any idea ?

NickCraver commented 1 year ago

Can you show me current code and the error? The core bits are: connect to the sentinels, with a service name specified (so it knows which primary to return).

daviddahan01 commented 1 year ago

yes. i've 3 sentinel servers, configured: port 26379 dir /tmp sentinel announce-ip 172.31.21.191 sentinel announce-port 26379 sentinel monitor mymaster 172.31.28.74 6380 2 sentinel auth-pass mymaster redis_pwd sentinel down-after-milliseconds mymaster 30000 sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 180000 sentinel deny-scripts-reconfig yes

also have 2 redis instances (master/slave).

.net core application registering the following service:

services.AddSingleton(serviceProvider => { var config = serviceProvider.GetRequiredService(); var redisConfig = config.GetSection("Redis").Get();

            var logger = serviceProvider.GetRequiredService<ILogger<RedisDistributedSynchronizationProvider>>();
            var options = new ConfigurationOptions
            {
                AbortOnConnectFail = false,
                AllowAdmin = true,
                ClientName = "XXX",
                ConnectTimeout = 5000,                    
                ConnectRetry = 10,
                ServiceName = "mymaster",
                Proxy = Proxy.None,
                Ssl = false,                  
                KeepAlive = 10,
                ResolveDns = true,                    
                SyncTimeout = 5000,

            };

            foreach (var s in redisConfig.SentinelHosts)
            {
                options.EndPoints.Add(s);
            }                

            var connection = ConnectionMultiplexer.Connect(options);

            connection.ConnectionFailed += Connection_ConnectionFailed;
            connection.ConfigurationChanged += Conn_ConfigurationChanged;
            connection.ConnectionRestored += Connection_ConnectionRestored;

            return new RedisDistributedSynchronizationProvider(connection.GetDatabase());
        });

Error: [15:04:22 FTL] An error occurred starting the application System.AggregateException: One or more errors occurred. (Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone) ---> StackExchange.Redis.RedisConnectionException: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone at StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 166 at MapAPI.Extensions.ServiceCollectionExtensions.<>c.b0_9(IServiceProvider serviceProvider) in /src/MapAPI/Extensions/ServiceCollectionExtensions.cs:line 374 at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) at MapAPI.Services.HostApplicationLifetimeEventsHostedService.OnStarted() in /src/MapAPI/Services/HostApplicationLifetimeEventsHostedService.cs:line 66 at System.Threading.CancellationToken.<>c.b__12_0(Object obj) at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source) at System.Threading.CancellationTokenSource.CallbackNode.<>c.b9_0(Object s) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) --- End of inner exception stack trace --- at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException) at System.Threading.CancellationTokenSource.Cancel(Boolean throwOnFirstException) at Microsoft.Extensions.Hosting.Internal.ApplicationLifetime.NotifyStarted()

NickCraver commented 1 year ago

Are you sure the code you have is getting deployed? That call stack still shows a direct call to GetSentinelMasterConnection which doesn't match your code. It would be an exception I expect with your last example code, though.

daviddahan01 commented 1 year ago

hi @NickCraver , i found a firewwall issue. fixed it. but now, getting the following error : [16:13:34 FTL] An error occurred starting the application System.AggregateException: One or more errors occurred. (Timeout performing ROLE (5000ms), inst: 8, qu: 1, qs: 0, aw: False, bw: SpinningDown, last-in: 0, cur-in: 0, sync-ops: 1, async-ops: 0, serverEndpoint: 172.31.30.174:6381, conn-sec: n/a, mc: 1/1/0, mgr: 10 of 10 available, clientName: XXX, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=4,Free=32763,Min=2,Max=32767), POOL: (Threads=9,QueuedItems=1,CompletedItems=505), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)) ---> StackExchange.Redis.RedisTimeoutException: Timeout performing ROLE (5000ms), inst: 8, qu: 1, qs: 0, aw: False, bw: SpinningDown, last-in: 0, cur-in: 0, sync-ops: 1, async-ops: 0, serverEndpoint: 172.31.30.174:6381, conn-sec: n/a, mc: 1/1/0, mgr: 10 of 10 available, clientName: XXX, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=4,Free=32763,Min=2,Max=32767), POOL: (Threads=9,QueuedItems=1,CompletedItems=505), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server, T defaultValue) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1980 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/RedisBase.cs:line 62 at StackExchange.Redis.RedisServer.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/RedisServer.cs:line 742 at StackExchange.Redis.RedisServer.Role(CommandFlags flags) in //src/StackExchange.Redis/RedisServer.cs:line 428 at StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in //src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 236 at MapAPI.Extensions.ServiceCollectionExtensions.<>c.b0_9(IServiceProvider serviceProvider) in /src/MapAPI/Extensions/ServiceCollectionExtensions.cs:line 362 at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) at MapAPI.Services.HostApplicationLifetimeEventsHostedService.OnStarted() in /src/MapAPI/Services/HostApplicationLifetimeEventsHostedService.cs:line 66 at System.Threading.CancellationToken.<>c.b__12_0(Object obj) at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source) at System.Threading.CancellationTokenSource.CallbackNode.<>c.b9_0(Object s) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) --- End of inner exception stack trace --- at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException) at System.Threading.CancellationTokenSource.Cancel(Boolean throwOnFirstException) at Microsoft.Extensions.Hosting.Internal.ApplicationLifetime.NotifyStarted()

Im trying to investigate it now. Thanks. David.

NickCraver commented 1 year ago

Looks like you're still not connecting, maybe still firewall. 2 things you'll want to do:

  1. Don't set AbortOnConnectFail to false, you want it true so it throws when diagnosing your issue here.2
  2. Pass a logger to the Connect method (second arg) and that'll give you more details about the whole connect process here to help.
daviddahan01 commented 1 year ago

Hi @NickCraver , Thank's. i've just added the logger. seems sentinel is unable to auth against redis instances although it is configured with correct password. very strange.

this is the logs: 172.31.30.174:6381: Setting client name: XXX 08:10:59.6569: 172.31.28.74:6380/Interactive: Connected 08:10:59.6569: 172.31.28.74:6380: Server handshake 08:10:59.6569: 172.31.28.74:6380: Setting client name: XXX 08:10:59.6570: 172.31.28.74:6380: Auto-configuring... 08:10:59.6571: 172.31.30.174:6381: Auto-configuring... 08:10:59.6575: 172.31.28.74:6380: Sending critical tracer (handshake): ECHO 08:10:59.6576: 172.31.28.74:6380/Interactive: Writing: ECHO 08:10:59.6577: 172.31.30.174:6381: Sending critical tracer (handshake): ECHO 08:10:59.6577: 172.31.30.174:6381/Interactive: Writing: ECHO 08:10:59.6577: 172.31.30.174:6381: Flushing outbound buffer 08:10:59.6577: 172.31.30.174:6381: OnEstablishingAsync complete 08:10:59.6577: 172.31.30.174:6381: Starting read 08:10:59.6578: 172.31.28.74:6380: Flushing outbound buffer 08:10:59.6578: 172.31.28.74:6380: OnEstablishingAsync complete 08:10:59.6578: 172.31.28.74:6380: Starting read 08:10:59.6630: Response from 172.31.28.74:6380/Interactive / ECHO: Error: NOAUTH Authentication required. 08:10:59.6630: Response from 172.31.30.174:6381/Interactive / ECHO: Error: NOAUTH Authentication required. 08:10:59.6668: 172.31.28.74:6380: OnConnectedAsync completed (Disconnected) 08:10:59.6668: 172.31.30.174:6381: OnConnectedAsync completed (Disconnected) 08:10:59.6669: All 2 available tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), POOL: (Threads=3,QueuedItems=0,CompletedItems=187) 08:10:59.6669: Endpoint summary: 08:10:59.6669: 172.31.30.174:6381: Endpoint is (Interactive: Connecting, Subscription: Disconnected) 08:10:59.6670: 172.31.28.74:6380: Endpoint is (Interactive: Connecting, Subscription: Disconnected) 08:10:59.6670: Task summary: 08:10:59.6670: 172.31.30.174:6381: Returned, but incorrectly 08:10:59.6670: 172.31.28.74:6380: Returned, but incorrectly 08:10:59.6670: Election summary: 08:10:59.6670: Election: No primaries detected 08:10:59.6670: Endpoint Summary: 08:10:59.6670: 172.31.30.174:6381: Standalone v3.0.0, primary; keep-alive: 00:00:10; int: Connecting; sub: Disconnected; not in use: DidNotRespond 08:10:59.6670: 172.31.30.174:6381: int ops=8, qu=0, qs=0, qc=0, wr=0, socks=2; sub ops=3, qu=0, qs=0, qc=0, wr=0, socks=1 08:10:59.6670: 172.31.30.174:6381: Circular op-count snapshot; int: 0+8=8 (0.80 ops/s; spans 10s); sub: 0+3=3 (0.30 ops/s; spans 10s) 08:10:59.6670: 172.31.28.74:6380: Standalone v3.0.0, primary; keep-alive: 00:00:10; int: Connecting; sub: Disconnected; not in use: DidNotRespond 08:10:59.6670: 172.31.28.74:6380: int ops=8, qu=0, qs=0, qc=0, wr=0, socks=2; sub ops=3, qu=0, qs=0, qc=0, wr=0, socks=1 08:10:59.6670: 172.31.28.74:6380: Circular op-count snapshot; int: 0+8=8 (0.80 ops/s; spans 10s); sub: 0+3=3 (0.30 ops/s; spans 10s) 08:10:59.6670: Sync timeouts: 0; async timeouts: 0; fire and forget: 0; last heartbeat: -1s ago

( i've tested connection with redis-cli with that password - worked ok). i will try to remove auth to see if things are working, but i would need to set it up later in prod...

Best, David.

movergan commented 1 year ago

We have this problem when a password for connection to Redis is incorrect. The error is totally misleading. Steps-to-reproduce:

  1. Create a sentinel cluster with password "123abc"
  2. Try to connect from the app using a different password Actual Result: 
    "Error while connecting to Redis (endpoints: redis:26379)", "error": "StackExchange.Redis.RedisConnectionException: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone\n   at StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2390\

    Expected Result:

Any better log message pointing to the exact problem.

jakox1 commented 1 year ago

I have this problem as well. Have setup 3 sentinel replicas together with 3 redis replicas. They are all within the same kubernetes namespace so don't think there is a firewall issue here. I also don't think we use any password for the sentinel cluster. Currently using StackExchange.Redis version v2.6.111.64013 @daviddahan01 Did you manage to get it working?

I'll write a stacktrace of my problem if someone notices the problem Error in loader: ProcessCodePageCacheLoader. Attempt 1.At: /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs Method: StackExchange.Redis.ConnectionMultiplexer GetSentinelMasterConnection(StackExchange.Redis.ConfigurationOptions, System.IO.TextWriter) Line: 166 Message: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone Data: StackTrace:

StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 166

StackExchange.Redis.ConnectionMultiplexer.SentinelPrimaryConnect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 133

StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 673

StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 651

ConnectionFactory.GetConnection()
ConnectionFactory.GetDatabase()
AsyncStreamingRedisLoader2.InsertBatchAsync(UpdateType updateType, IDictionary2 entities)
AsyncStreamingDataLoader2.LoadData(UpdateType updateType, Nullable1 dateTime)
AsyncStreamingDataLoader2.MakeUpdate(Nullable1 dateTime)

Using connectionstring 'redis-sentinel-svc:5000,serviceName=mymaster,allowAdmin=true,asyncTimeout=10000,connectTimeout=100,connectRetry=0,abortConnect=false'

Pikabanga commented 2 months ago

I have this problem as well. Have setup 3 sentinel replicas together with 3 redis replicas. They are all within the same kubernetes namespace so don't think there is a firewall issue here. I also don't think we use any password for the sentinel cluster. Currently using StackExchange.Redis version v2.6.111.64013 @daviddahan01 Did you manage to get it working?

I'll write a stacktrace of my problem if someone notices the problem Error in loader: ProcessCodePageCacheLoader. Attempt 1.At: /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs Method: StackExchange.Redis.ConnectionMultiplexer GetSentinelMasterConnection(StackExchange.Redis.ConfigurationOptions, System.IO.TextWriter) Line: 166 Message: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone Data: StackTrace:

StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 166

StackExchange.Redis.ConnectionMultiplexer.SentinelPrimaryConnect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 133

StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 673

StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 651

ConnectionFactory.GetConnection() ConnectionFactory.GetDatabase() AsyncStreamingRedisLoader2.InsertBatchAsync(UpdateType updateType, IDictionary2 entities) AsyncStreamingDataLoader2.LoadData(UpdateType updateType, Nullable1 dateTime) AsyncStreamingDataLoader2.MakeUpdate(Nullable1 dateTime)

Using connectionstring 'redis-sentinel-svc:5000,serviceName=mymaster,allowAdmin=true,asyncTimeout=10000,connectTimeout=100,connectRetry=0,abortConnect=false'

Same problem here, using k8s.

StackExchange.Redis.RedisConnectionException: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone
   at StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 167
   at StackExchange.Redis.ConnectionMultiplexer.SentinelPrimaryConnect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 134
   at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 685

Connectionstring: sentinel-svc:5000,serviceName=mymaster,allowAdmin=true,connectTimeout=250,connectRetry=0,abortConnect=false

Skoucail commented 2 months ago

I have this problem as well. Have setup 3 sentinel replicas together with 3 redis replicas. They are all within the same kubernetes namespace so don't think there is a firewall issue here. I also don't think we use any password for the sentinel cluster. Currently using StackExchange.Redis version v2.6.111.64013 @daviddahan01 Did you manage to get it working? I'll write a stacktrace of my problem if someone notices the problem Error in loader: ProcessCodePageCacheLoader. Attempt 1.At: //src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs Method: StackExchange.Redis.ConnectionMultiplexer GetSentinelMasterConnection(StackExchange.Redis.ConfigurationOptions, System.IO.TextWriter) Line: 166 Message: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone Data: StackTrace: StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in //src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 166 StackExchange.Redis.ConnectionMultiplexer.SentinelPrimaryConnect(ConfigurationOptions configuration, TextWriter log) in //src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 133 StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 673 StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 651 ConnectionFactory.GetConnection() ConnectionFactory.GetDatabase() AsyncStreamingRedisLoader2.InsertBatchAsync(UpdateType updateType, IDictionary2 entities) AsyncStreamingDataLoader2.LoadData(UpdateType updateType, Nullable1 dateTime) AsyncStreamingDataLoader2.MakeUpdate(Nullable1 dateTime) Using connectionstring 'redis-sentinel-svc:5000,serviceName=mymaster,allowAdmin=true,asyncTimeout=10000,connectTimeout=100,connectRetry=0,abortConnect=false'

Same problem here, using k8s.

StackExchange.Redis.RedisConnectionException: Sentinel: The ConnectionMultiplexer is not a Sentinel connection. Detected as: Standalone
   at StackExchange.Redis.ConnectionMultiplexer.GetSentinelMasterConnection(ConfigurationOptions config, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 167
   at StackExchange.Redis.ConnectionMultiplexer.SentinelPrimaryConnect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs:line 134
   at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 685

Connectionstring: sentinel-svc:5000,serviceName=mymaster,allowAdmin=true,connectTimeout=250,connectRetry=0,abortConnect=false

Same issue, any solution?

Edit: I has a problem on my side (passing wrong password) What helped me understanding the issue was passing a logger to the ConnectionMultiplexer connect method.

Pikabanga commented 2 months ago

I have not been able to solve it yet. We do not have a password, and saw nothing out of the ordinary in the logs from the ConnectionMultiplexer after passing in our LoggerFactory in the configuration.

This issue claims that this error appears when you make a second connection with the ConnectionMultiplexer. However, I am unable to see how we are ever calling connect more than once. Everyhing are singletons, and we have our own connectionfactory, which initializes the connection in a thread safe Lazy (I have added logging before calling connect, I still need to check in the environment if we indeed only connect once).

So, I am still not sure if the issue is multiple connections or not.

I did some digging in the source code, and this exception occurs when ServerSelectionStrategy.ServerType is not Sentinel:

image

Now, where I got stuck was to figure out when exactly in the flow, this ServerType is set. I of course can find all parts where this property is set (ReconfigureAsync in ConnectionMultiplexer.cs and TryParseServerType and SetResultCore in ResultProcessor.cs), but I had difficulty understanding the order of events.

Could this be a race condition, where this is not set at the right time (the default value of the property is ServerType.Standalone)? Or is it like this other issue claims, that this correction is mutated incorrectly on following connections?

I could also mention that another team at my company have the same problem, and they are stuck like us. They are actually working on switching to a Redis Cluster instead of using Sentinel due to this.