search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.83k stars 9.12k forks source link

[bitnami/redis] repeated SSL Wrong Version Number on Sentinel Pod #7109

Closed panteparak closed 3 years ago

panteparak commented 3 years ago

Which chart: bitnami/redis (14.8.7)

Describe the bug On Sentinel Pod, a repeatedly logged SSL Wrong version number. Logs below

To Reproduce Steps to reproduce the behavior:

Deploy Redis with the following values.yaml

sentinel: 
  enabled: true
  livenessProbe:
    enabled: true
  readinessProbe:
    enabled: true
rbac:
  create: true
tls:
  enabled: true
  authClients: false
  autoGenerated: true

Expected behavior Able to connect without repeated errors.

Version of Helm and Kubernetes:

version.BuildInfo{Version:"v3.6.2", GitCommit:"ee407bdf364942bcb8e8c665f82e15aa28009b71", GitTreeState:"dirty", GoVersion:"go1.16.5"}
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Additional context

 19:07:10.33 WARN  ==> redis-sentinal-headless.redis-services.svc.cluster.local does not contain the IP of this pod: 10.233.82.130
 19:07:15.36 INFO  ==> Sentinels clean up done
1:X 31 Jul 2021 19:07:15.385 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 31 Jul 2021 19:07:15.385 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 31 Jul 2021 19:07:15.385 # Configuration loaded
1:X 31 Jul 2021 19:07:15.387 * monotonic clock: POSIX clock_gettime
1:X 31 Jul 2021 19:07:15.389 * Running mode=sentinel, port=26379.
1:X 31 Jul 2021 19:07:15.395 # Sentinel ID is 76992ad92a942284ab322ef40705454e15e6ffe7
1:X 31 Jul 2021 19:07:15.395 # +monitor master mymaster 10.233.82.130 6379 quorum 2
1:X 31 Jul 2021 19:07:37.019 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
1:X 31 Jul 2021 19:07:37.053 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=11)
...
marcosbc commented 3 years ago

Hi @panteparak, unfortunately I'm unable to reproduce your issue. Note that I'm deploying Redis and seeing no such errors.

If you are accessing Redis externally, it could be possible that the Redis client you are using is incompatible with the version used by the cluster, at least in terms of SSL compatibility. If so, have you tried with a different client?

panteparak commented 3 years ago

@marcosbc, i have done further investigation on my part. It seems that after the 2nd replica is deployed the SSL error starts to appear. Also I have updated my values.yaml config to include the enabled sentinel health check settings.

marcosbc commented 3 years ago

Unfortunately I'm still unable to reproduce this issue:

 10:26:19.19 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.17.227
 10:26:24.30 INFO  ==> Sentinels clean up done
1:X 03 Aug 2021 10:26:24.597 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 03 Aug 2021 10:26:24.597 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 03 Aug 2021 10:26:24.597 # Configuration loaded
1:X 03 Aug 2021 10:26:24.611 * monotonic clock: POSIX clock_gettime
1:X 03 Aug 2021 10:26:24.614 * Running mode=sentinel, port=26379.
1:X 03 Aug 2021 10:26:24.621 # Sentinel ID is c27c6143ab6ec0946242813d14fab911775063ea
1:X 03 Aug 2021 10:26:24.621 # +monitor master mymaster 10.30.17.227 6379 quorum 2
1:X 03 Aug 2021 10:27:32.159 # +reset-master master mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:27:34.817 * +slave slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:27:39.415 * +sentinel sentinel a58568a17ff98fda196151cd41750cc6a689268d 10.30.14.151 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:11.207 # +reset-master master mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:12.084 * +sentinel sentinel a58568a17ff98fda196151cd41750cc6a689268d 10.30.14.151 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:14.926 * +slave slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:18.388 * +sentinel sentinel 843216ed7e771a67cceb483e26036aedb1e7fcb9 10.30.12.16 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:29:15.021 # +sdown slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379
 10:27:01.89 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.14.151
 10:27:06.93 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.14.151
 10:27:11.95 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.14.151
 10:27:16.98 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.14.151
 10:27:22.00 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.14.151
 10:27:27.03 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.14.151
 10:27:32.14 INFO  ==> Cleaning sentinels in sentinel node: 10.30.17.227
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1
 10:27:37.16 INFO  ==> Sentinels clean up done
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1:X 03 Aug 2021 10:27:37.332 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 03 Aug 2021 10:27:37.332 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 03 Aug 2021 10:27:37.332 # Configuration loaded
1:X 03 Aug 2021 10:27:37.334 * monotonic clock: POSIX clock_gettime
1:X 03 Aug 2021 10:27:37.343 * Running mode=sentinel, port=26379.
1:X 03 Aug 2021 10:27:37.349 # Sentinel ID is a58568a17ff98fda196151cd41750cc6a689268d
1:X 03 Aug 2021 10:27:37.349 # +monitor master mymaster 10.30.17.227 6379 quorum 2
1:X 03 Aug 2021 10:27:37.353 * +slave slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:27:37.954 * +sentinel sentinel c27c6143ab6ec0946242813d14fab911775063ea 10.30.17.227 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:06.190 # +reset-master master mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:06.393 * +sentinel sentinel c27c6143ab6ec0946242813d14fab911775063ea 10.30.17.227 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:07.487 * +slave slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:18.388 * +sentinel sentinel 843216ed7e771a67cceb483e26036aedb1e7fcb9 10.30.12.16 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:29:07.502 # +sdown slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379
 10:28:00.90 WARN  ==> mbredis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.30.12.16
 10:28:05.94 INFO  ==> Cleaning sentinels in sentinel node: 10.30.14.151
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1
 10:28:11.19 INFO  ==> Cleaning sentinels in sentinel node: 10.30.17.227
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1
 10:28:16.21 INFO  ==> Sentinels clean up done
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1:X 03 Aug 2021 10:28:16.382 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 03 Aug 2021 10:28:16.382 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 03 Aug 2021 10:28:16.382 # Configuration loaded
1:X 03 Aug 2021 10:28:16.383 * monotonic clock: POSIX clock_gettime
1:X 03 Aug 2021 10:28:16.407 * Running mode=sentinel, port=26379.
1:X 03 Aug 2021 10:28:16.423 # Sentinel ID is 843216ed7e771a67cceb483e26036aedb1e7fcb9
1:X 03 Aug 2021 10:28:16.424 # +monitor master mymaster 10.30.17.227 6379 quorum 2
1:X 03 Aug 2021 10:28:16.430 * +slave slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:16.614 * +sentinel sentinel c27c6143ab6ec0946242813d14fab911775063ea 10.30.17.227 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:28:18.213 * +sentinel sentinel a58568a17ff98fda196151cd41750cc6a689268d 10.30.14.151 26379 @ mymaster 10.30.17.227 6379
1:X 03 Aug 2021 10:29:16.464 # +sdown slave 10.30.17.1:6379 10.30.17.1 6379 @ mymaster 10.30.17.227 6379

Note that I'm using the exact same deployment configuration that you included in the first post. I deployed it like this (chart version 14.8.7):

$ helm install mbredis -f my-values.yaml bitnami/redis

Could you let me know if I'm missing anything? I could not see any SSL errors even 10 minutes after the deployment.

panteparak commented 3 years ago

@marcosbc Ok, i've found the root cause of many repeated SSL errors on my sandbox cluster. There is a service that was trying to access redis but failed, so the ssl errors keeps coming up.

Closing this.