Upgrade to v15 crashes: Error from server: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused

FayeG2b commented 9 months ago

What version of redis operator are you using?

I'm upgrading from v11 to v15

k logs -p redis-operator-8494484f55-xlnj9
I0222 21:06:12.271390       1 request.go:665] Waited for 1.039555517s due to client-side throttling, not priority and fairness, request: GET:https://10.206.48.1:443/apis/postgres-operator.crunchydata.com/v1beta1?timeout=32s
{"level":"info","ts":1708635973.0753748,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1708635973.0758939,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1708635973.076478,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":1708635973.0765736,"msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
I0222 21:06:13.076661       1 leaderelection.go:248] attempting to acquire leader lease redis-operator/6cab913b.redis.opstreelabs.in...
I0222 21:06:29.910360       1 leaderelection.go:258] successfully acquired lease redis-operator/6cab913b.redis.opstreelabs.in
{"level":"info","ts":1708635989.9112737,"logger":"controller.redisreplication","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication","source":"kind source: *v1beta2.RedisReplication"}
{"level":"info","ts":1708635989.9113243,"logger":"controller.redisreplication","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication"}
{"level":"info","ts":1708635989.9114513,"logger":"controller.redis","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","source":"kind source: *v1beta2.Redis"}
{"level":"info","ts":1708635989.9114928,"logger":"controller.redis","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis"}
{"level":"info","ts":1708635989.9115448,"logger":"controller.redissentinel","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel","source":"kind source: *v1beta2.RedisSentinel"}
{"level":"info","ts":1708635989.9115677,"logger":"controller.redissentinel","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel"}
{"level":"info","ts":1708635989.9127183,"logger":"controller.rediscluster","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","source":"kind source: *v1beta2.RedisCluster"}
{"level":"info","ts":1708635989.9127526,"logger":"controller.rediscluster","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster"}
W0222 21:06:29.918174       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:29.918209       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.Redis: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:29.919669       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:29.919691       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.RedisCluster: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:30.909363       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:30.909416       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.RedisCluster: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:31.295004       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:31.295031       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.Redis: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:32.992515       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:32.992571       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.RedisCluster: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:33.972113       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:33.972139       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.Redis: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:38.156749       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:38.156808       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.RedisCluster: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:40.326339       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:40.326377       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.Redis: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:47.099123       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:47.099181       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.RedisCluster: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:06:52.450205       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:06:52.450226       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.Redis: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:07:07.612207       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:07:07.612271       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.RedisCluster: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:07:17.457688       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:07:17.457803       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.Redis: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:07:41.888662       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:07:41.888700       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.RedisCluster: failed to list *v1beta2.RedisCluster: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
W0222 21:08:01.122756       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
E0222 21:08:01.122790       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: Failed to watch *v1beta2.Redis: failed to list *v1beta2.Redis: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused
{"level":"error","ts":1708636109.911984,"logger":"controller.redisreplication","msg":"Could not wait for Cache to sync","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication","error":"failed to wait for redisreplication caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218"}
{"level":"info","ts":1708636109.9121125,"msg":"Stopping and waiting for non leader election runnables"}
{"level":"error","ts":1708636109.911997,"logger":"controller.redis","msg":"Could not wait for Cache to sync","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","error":"failed to wait for redis caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218"}
{"level":"error","ts":1708636109.9121382,"msg":"error received after stop sequence was engaged","error":"failed to wait for redis caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541"}
{"level":"error","ts":1708636109.912006,"logger":"controller.redissentinel","msg":"Could not wait for Cache to sync","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel","error":"failed to wait for redissentinel caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218"}
{"level":"error","ts":1708636109.9121892,"msg":"error received after stop sequence was engaged","error":"failed to wait for redissentinel caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541"}
{"level":"info","ts":1708636109.912122,"msg":"Stopping and waiting for leader election runnables"}
{"level":"error","ts":1708636109.9122558,"logger":"controller.rediscluster","msg":"Could not wait for Cache to sync","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","error":"failed to wait for rediscluster caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218"}
{"level":"info","ts":1708636109.9122794,"msg":"Stopping and waiting for caches"}
{"level":"error","ts":1708636109.912295,"msg":"error received after stop sequence was engaged","error":"failed to wait for rediscluster caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541"}
{"level":"info","ts":1708636109.9123614,"msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":1708636109.912505,"msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":1708636109.9125361,"msg":"error received after stop sequence was engaged","error":"leader election lost","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541"}
{"level":"error","ts":1708636109.9125364,"logger":"setup","msg":"problem running manager","error":"failed to wait for redisreplication caches to sync: timed out waiting for cache to be synced","stacktrace":"main.main\n\t/workspace/main.go:159\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255"}

redis-operator version: v15

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (kubectl version)?

kubectl version Output

$ kubectl version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.12-eks-5e0fdde

What did you do?

kubectl create ns redis-operator
kubectl create ns sre-test-redis

helm install redis-operator --version 0.11.2 ot-helm/redis-operator --namespace redis-operator
helm install redis-cluster ot-helm/redis-cluster --version 0.11.2 --set redisCluster.clusterSize=3  --namespace sre-test-redis

kubectl get rediscluster --all-namespaces --no-headers | while read ns name other; do
    kubectl patch rediscluster ${name} -n "${ns}" -p '{"metadata": {"annotations": {"rediscluster.opstreelabs.in/skip-reconcile": "true"}}}' --type merge
done

helm upgrade redis-operator --version 0.12.0 ot-helm/redis-operator --namespace redis-operator
kubectl replace -f "${REDIS_DIR}/redis-operator-0.12.0/crds/redis-cluster.yaml"
kubectl replace -f "${REDIS_DIR}/redis-operator-0.12.0/crds/redis.yaml"

helm upgrade redis-operator --version 0.13.1 ot-helm/redis-operator --namespace redis-operator
kubectl replace -f "${REDIS_DIR}/redis-operator-0.13.1/crds/redis-cluster.yaml"
kubectl replace -f "${REDIS_DIR}/redis-operator-0.13.1/crds/redis.yaml"

helm upgrade redis-operator --version 0.14.3 ot-helm/redis-operator --namespace redis-operator
kubectl replace -f "${REDIS_DIR}/redis-operator-0.14.3/crds/redis-cluster.yaml"
kubectl replace -f "${REDIS_DIR}/redis-operator-0.14.3/crds/redis.yaml"
kubectl create -f "${REDIS_DIR}/redis-operator-0.14.3/crds/redis-replication.yaml"
kubectl create -f "${REDIS_DIR}/redis-operator-0.14.3/crds/redis-sentinel.yaml"

helm upgrade redis-operator --version 0.15.9 ot-helm/redis-operator --namespace redis-operator
kubectl replace -f "${REDIS_DIR}/redis-operator-0.15.9/crds/redis-cluster.yaml"
kubectl replace -f "${REDIS_DIR}/redis-operator-0.15.9/crds/redis.yaml"
kubectl replace -f "${REDIS_DIR}/redis-operator-0.15.9/crds/redis-replication.yaml"
kubectl replace -f "${REDIS_DIR}/redis-operator-0.15.9/crds/redis-sentinel.yaml"

sleep 180

kubectl get rediscluster --all-namespaces
# ^^^ fails with error:
# Error from server: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused

What did you expect to see?

A list of rediscluster objects

What did you see instead?

Error from server: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused

FayeG2b commented 9 months ago

The IP address 10.206.44.45 is the IP address of the redis-operator pod

FayeG2b commented 9 months ago

Some progress. I did helm install redis-operator --version 0.15.9 ot-helm/redis-operator --namespace redis-operator --set redisOperator.webhook=true --set certmanager.enabled=true

and now I get a new error:

Error from server: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority

FayeG2b commented 9 months ago

More progress I'm now trying kubectl get rediscluster --all-namespaces --insecure-skip-tls-verify

and am getting this error:

Error from server: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": no endpoints available for service "webhook-service"

yushuobuxiu commented 9 months ago

I encountered the same problem as you.

leona commented 8 months ago

I was able to fix this by using v1beta2 instead of v1beta1. apiVersion: redis.redis.opstreelabs.in/v1beta2

FayeG2b commented 8 months ago

Good to hear.

I'm finding that I need to provide a X509.3 CA object (base64 encoded) into the webhook configuration of the CRD to get past this when upgrading redis-operator from v11 to v15 with already extant v1beta1 redisclusters deployed in k8s.
I also have to gut the redis-operator's helm chart of cert-manager otherwise it destroys my cluster's ability to use the main cert-manager's clusterissuer configurations.
Then I edit the helm chart so that the webhook asks for a clusterissuer cert rather than a issuer cert.

I'll post more details soon.

OT-CONTAINER-KIT / redis-operator

Upgrade to v15 crashes: Error from server: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=RedisCluster failed: Post "https://webhook-service.redis-operator.svc:443/convert?timeout=30s": dial tcp 10.206.44.45:9443: connect: connection refused #784