search

TykTechnologies / tyk-helm-chart

A Helm chart repository to install Tyk Pro (with Dashboard), Tyk Hybrid or Tyk Headless chart.
https://tyk.io
71 stars 55 forks source link

Reconnecting storage: Redis is either down or ws not configured #113

Closed anup-krai closed 1 year ago

anup-krai commented 3 years ago

We are installing Tyk gateway(oss) through helm chart but seeing some error in gateway pods. Below are the gateway pod logs -

time="Apr 07 13:41:46" level=info msg="Tyk API Gateway v3.1.2" prefix=main
time="Apr 07 13:41:46" level=warning msg="Insecure configuration allowed" config.allow_insecure_configs=true prefix=checkup
time="Apr 07 13:41:46" level=info msg="Starting Poller" prefix=host-check-mgr
time="Apr 07 13:41:46" level=info msg="PIDFile location set to: /mnt/tyk-gateway/tyk.pid" prefix=main
time="Apr 07 13:41:46" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
time="Apr 07 13:41:46" level=info msg="--> [REDIS] Creating single-node client"
time="Apr 07 13:41:46" level=info msg="--> Standard listener (http)" port=":9696" prefix=main
time="Apr 07 13:41:46" level=warning msg="Starting HTTP server on:0.0.0.0:9696" prefix=main
time="Apr 07 13:41:46" level=info msg="--> Standard listener (http)" port=":8080" prefix=main
time="Apr 07 13:41:46" level=warning msg="Starting HTTP server on:0.0.0.0:8080" prefix=main
time="Apr 07 13:41:46" level=info msg="Initialising distributed rate limiter" prefix=main
time="Apr 07 13:41:46" level=info msg="Tyk Gateway started (v3.1.2)" prefix=main
time="Apr 07 13:41:46" level=info msg="--> Listening on address: (open interface)" prefix=main
time="Apr 07 13:41:46" level=info msg="--> Listening on port: 8080" prefix=main
time="Apr 07 13:41:46" level=info msg="--> PID: 1" prefix=main
time="Apr 07 13:41:46" level=info msg="Loading policies" prefix=main
time="Apr 07 13:41:46" level=info msg="Policies found (1 total):" prefix=main
time="Apr 07 13:41:46" level=info msg="Starting gateway rate limiter notifications..."
time="Apr 07 13:41:46" level=info msg="Detected 0 APIs" prefix=main
time="Apr 07 13:41:46" level=warning msg="No API Definitions found, not reloading" prefix=main
time="Apr 07 13:41:56" level=error msg="Redis health check failed" error="storage: Redis is either down or ws not configured" liveness-check=true prefix=main
time="Apr 07 13:41:56" level=warning msg="Reconnecting storage: Redis is either down or ws not configured" prefix=pub-sub
time="Apr 07 13:42:06" level=error msg="Redis health check failed" error="storage: Redis is either down or ws not configured" liveness-check=true prefix=main
time="Apr 07 13:42:06" level=warning msg="Reconnecting storage: Redis is either down or ws not configured" prefix=pub-sub
time="Apr 07 13:42:16" level=error msg="Redis health check failed" error="storage: Redis is either down or ws not configured" liveness-check=true prefix=main
time="Apr 07 13:42:16" level=warning msg="Reconnecting storage: Redis is either down or ws not configured" prefix=pub-sub
time="Apr 07 13:42:26" level=error msg="Redis health check failed" error="storage: Redis is either down or ws not configured" liveness-check=true prefix=main
time="Apr 07 13:42:26" level=warning msg="Reconnecting storage: Redis is either down or ws not configured" prefix=pub-sub
time="Apr 07 13:42:26" level=info msg="--> [REDIS] Creating single-node client"
time="Apr 07 13:42:36" level=error msg="Redis health check failed" error="storage: Redis is either down or ws not configured" liveness-check=true prefix=main
time="Apr 07 13:42:36" level=warning msg="Reconnecting storage: Redis is either down or ws not configured" prefix=pub-sub

Below is the values.yaml file -

nameOverride: ""
fullnameOverride: "tyk-headless"

secrets:
  APISecret: "CHANGEME"
  OrgID: "1"

redis:
    shardCount: 128
    useSSL: true
    addrs:
      - "tyk-redis-master.tyk.svc.cluster.local:6379"
    pass: "somepassword"

mongo:
  enabled: false

gateway:
  kind: DaemonSet
  replicaCount: 1
  hostName: "gateway.tykbeta.com"
  tls: false
  containerPort: 8080
  tags: ""
  image:
    repository: tykio/tyk-gateway
    tag: v3.1.2
    pullPolicy: IfNotPresent
  service:
    type: NodePort
    port: 443
    externalTrafficPolicy: Local
    annotations: {}
  control:
    enabled: false
    containerPort: 9696
    port: 9696
    type: ClusterIP
    annotations: {}
  ingress:
    enabled: false
    annotations: {}
    path: /
    hosts:
      - tyk-gw.local
    tls: []
  resources: {}
  nodeSelector: {}
  tolerations:
    - key: node-role.kubernetes.io/master
      effect: NoSchedule
  affinity: {}
  extraEnvs: []

pump:
  enabled: false
  replicaCount: 1
  image:
    repository: tykio/tyk-pump-docker-pub
    tag: v1.2.0
    pullPolicy: IfNotPresent
  annotations: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}
  extraEnvs: []

rbac: true

We tried with both useSSL to true and false but getting same error in both. Are we missing anything here?

gernest commented 3 years ago

This has more to do with accessing redis. You can try to verify that you can indeed connect to redis on the tyk-redis-master.tyk.svc.cluster.local:6379

You need to have redis installed and working for tyk to operate. This will eventually go away though if the redis service is available and tyk has successfully established the connection.

can you please share the result of

kubectl get svc -n tyk
anup-krai commented 3 years ago

Below is the output -

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
gateway-svc-tyk-headless   NodePort    10.73.32.165    <none>        443:30716/TCP   127m
tyk-redis-headless         ClusterIP   None            <none>        6379/TCP        151m
tyk-redis-master           ClusterIP   10.73.71.169    <none>        6379/TCP        151m
tyk-redis-slave            ClusterIP   10.73.37.126    <none>        6379/TCP        151m
hervelemeur commented 3 years ago

Lost far too many hours on this warning message trying to resolve this redis "problem" which wasn't one.

There should be a following "Redis connected successfully" message in the gateway logs IMHO.

anup-krai commented 3 years ago

I checked all the gateway pod logs and all have same error and this is running for quite some time-

time="Apr 07 15:48:55" level=error msg="Redis health check failed" error="storage: Redis is either down or ws not configured" liveness-check=true prefix=main
time="Apr 07 15:48:56" level=warning msg="Reconnecting storage: Redis is either down or ws not configured" prefix=pub-sub
gernest commented 3 years ago

What happens when you make api calls to the gateway ?

This error can be sometime ignored, try creating api's and make api calls.

gernest commented 3 years ago

You can try to get more context from the logs by elevating log level to debug . In th gateway on values.yaml set extaenv like this

  extraEnvs: 
    - name: TYK_LOGLEVEL
      value: debug

This might help us to know what is going on.

anup-krai commented 3 years ago

I tried setting the loglevel but looks like its not getting set, still info/error logs are coming. Below is the gateway config on values.yaml

gateway:
  kind: DaemonSet
  replicaCount: 1
  hostName: "gateway.tykbeta.com"
  tls: false
  containerPort: 8080
  tags: ""
  image:
    repository: tykio/tyk-gateway
    tag: v3.1.2
    pullPolicy: IfNotPresent
  service:
    type: NodePort
    port: 443
    externalTrafficPolicy: Local
    annotations: {}
  control:
    enabled: false
    containerPort: 9696
    port: 9696
    type: ClusterIP
    annotations: {}
  ingress:
    enabled: false
    annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
    path: /
    hosts:
      - tyk-gw.local
    tls: []
    #  - secretName: chart-example-tls
    #    hosts:
    #      - chart-example.local
  extraEnvs: 
    - name: TYK_LOGLEVEL
      value: debug
  resources: {}
gernest commented 3 years ago

I asked, what you get after making calls to the gateway? Ignore the logs for now.

anup-krai commented 3 years ago

Sorry for the delayed response. We have installed tyk gateway though [https://github.com/TykTechnologies/tyk-oss-k8s-deployment#tyk-oss-deployment](Tyk OSS K8s Deployment) and redis error did not come.

But when we install through helm chat we are getting this error. Below is the response when we do an API call to the gateway -

404 page not found
sedkis commented 3 years ago

404 page not found

This isn't an error that the Gateway normally generates. What else could be creating that message?

zalbiraw commented 3 years ago

Sorry for the delayed response. We have installed tyk gateway though [https://github.com/TykTechnologies/tyk-oss-k8s-deployment#tyk-oss-deployment](Tyk OSS K8s Deployment) and redis error did not come.

But when we install through helm chat we are getting this error. Below is the response when we do an API call to the gateway -

404 page not found

What endpoint did you hit on the GW?

anup-krai commented 3 years ago

We are able to call gateway both through control port and service port successfully. Also we have enabled redis with TLS.

We are able to connect to redis through redis client successfully and also there are no errors in gateway. We also created few keys and those are also visible in redis.

But in redis pod I can see below logs -

1:M 21 Apr 2021 12:40:56.830 # Error accepting a client connection: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Is there any change we need to do in gateway configs?

ultd commented 3 years ago

Any progress on this? We're still running into the same error and the /hello route gives us following JSON response: 

{
    "status": "fail",
    "version": "v3.1.2",
    "description": "Tyk GW",
    "details": {
        "redis": {
            "status": "fail",
            "output": "storage: Redis is either down or ws not configured",
            "componentType": "datastore",
            "time": "2021-06-06T21:26:04Z"
        }
    }
}
ultd commented 3 years ago

I was able to solve this issue. Just use my shell script here and it should install & configure Tyk-CE and Tyk-Operator on a k8s cluster without any problem.

flyte commented 2 years ago

I was able to solve this issue

Could you please let us know what you did to solve the issue?

flyte commented 2 years ago

Of course I solve it 5 minutes after asking for help! I was missing the port from the addrs section:

redis:
  addrs:
    - my-redis-master:6379

now works :tada: