search

TykTechnologies / tyk

Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols
Other
9.54k stars 1.07k forks source link

[TT-11223]ERROR: Tyk PUMP not able to connect to Redis Sentinels #6002

Open arshavardan opened 7 months ago

arshavardan commented 7 months ago

Environment: Kubernetes

Using Tyk Pro-helm/tyk-pro --version 0.14.0 using Bitnami/Redis-18.6.0 (Sentinels Enabled) PUMP Configuration: env:

Legacy support for Redis Cluster driver. Driver dropped in v3.0.0.

        - name: REDIGOCLUSTER_SHARDCOUNT
          value: "128"
        - name: TYK_PMP_ANALYTICSSTORAGECONFIG_DATABASE
          value: "0"
        - name: TYK_PMP_ANALYTICSSTORAGECONFIG_REDISUSESSL
          value: "true"
        - name: TYK_PMP_ANALYTICSSTORAGECONFIG_ADDRS
          value: "redis-sentinels-headless.namespace.svc.cluster.local:26379"
        - name: TYK_PMP_ANALYTICSSTORAGECONFIG_ENABLECLUSTER
          value: "false"
        - name: TYK_PMP_ANALYTICSSTORAGECONFIG_REDISSSLINSECURESKIPVERIFY
          value: "true"
        - name: TYK_PMP_ANALYTICSSTORAGECONFIG_PASSWORD
          valueFrom:
            secretKeyRef:
              name: secrets-tyk-pro
              key: redisPass

and pump.conf: | is default as per the charts before we are using standalone redis, where we have given direct master pod DNS address, now we are giving redis-sentinels-headless service with port 26379.

Pump logs: time="Jan 31 05:44:23" level=debug msg="Unpacked vals: 0" prefix=redis time="Jan 31 05:44:23" level=debug msg="Getting raw key set: tyk-system-analytics_3_protobuf" prefix=redis time="Jan 31 05:44:23" level=debug msg="keyName is: tyk-system-analytics_3_protobuf" prefix=redis time="Jan 31 05:44:23" level=debug msg="Input key was: analytics-tyk-system-analytics_3_protobuf" prefix=redis time="Jan 31 05:44:23" level=debug msg="Fixed keyname is: analytics-tyk-system-analytics_3_protobuf" prefix=redis time="Jan 31 05:44:23" level=error msg="Multi command failed: read: connection reset by peer" prefix=redis time="Jan 31 05:44:23" level=debug msg="Storage Engine already initialized..." prefix=redis time="Jan 31 05:44:23" level=debug msg="Unpacked vals: 0" prefix=redis time="Jan 31 05:44:23" level=debug msg="Getting raw key set: tyk-system-analytics_4" prefix=redis time="Jan 31 05:44:23" level=debug msg="keyName is: tyk-system-analytics_4" prefix=redis time="Jan 31 05:44:23" level=debug msg="Input key was: analytics-tyk-system-analytics_4" prefix=redis time="Jan 31 05:44:23" level=debug msg="Fixed keyname is: analytics-tyk-system-analytics_4" prefix=redis time="Jan 31 05:44:23" level=error msg="Multi command failed: read: connection reset by peer" prefix=redis

These are the recurring logs in the tyk_pump: redis-master pod logs:

1:M 31 Jan 2024 05:12:31.526 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44116 laddr=:6379) 1:M 31 Jan 2024 05:12:33.139 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44122 laddr=:6379) 1:M 31 Jan 2024 05:12:33.159 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44128 laddr=:6379) 1:M 31 Jan 2024 05:12:33.183 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44142 laddr=:6379) 1:M 31 Jan 2024 05:12:33.200 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44158 laddr=:6379) 1:M 31 Jan 2024 05:12:33.211 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44160 laddr=:6379) 1:M 31 Jan 2024 05:12:33.222 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44166 laddr=:6379) 1:M 31 Jan 2024 05:12:33.233 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44168 laddr=:6379) 1:M 31 Jan 2024 05:12:33.251 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44184 laddr=:6379) 1:M 31 Jan 2024 05:12:33.262 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44186 laddr=:6379) 1:M 31 Jan 2024 05:12:33.287 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (addr=:44196 laddr=:6379)

if anyone faced this issue please provide appropriate solution! Thanks

arshavardan commented 7 months ago

Could this be causing by different openssl Versions between Redis and the Tyk Pump, below are the listed Openssl versions of both tyk-components and Redis: Dashboard: openssl 1.1.1w

Gateway: openssl 1.1.1w

mdcb: openssl 1.1.1w

pump: Openssl 1.1.1n

Redis: openssl 1.1.1w

You can see that pump openssl version is different from the others!