search

mittwald / kubernetes-replicator

Kubernetes controller for synchronizing secrets & config maps across namespaces
Apache License 2.0
834 stars 97 forks source link

fatal error: concurrent map iteration and map write dealing with short-living namespaces #216

Open elitvinov opened 1 year ago

elitvinov commented 1 year ago

Describe the bug

Hi, we have the following config to replicate a docker private registry key across ~500 namespaces.

apiVersion: v1
kind: Secret
metadata:
  name: docker-cred-secret
  annotations:
    replicator.v1.mittwald.de/replicate-to: "*"
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ${docker_config_json}

Namespaces are dynamically created and removed on regular basis. A pod running kubernetes-replicator is restarting hourly or more often due to

fatal error: concurrent map iteration and map write

To Reproduce Create the above spec Start 250 static namespaces Start creating followed by termination 250 more namespaces constantly. Some namespaces live 10 seconds or less, some - not more than 1h. Aim is having a process of minimum 5 simultaneously namespaces termination

Expected behavior kubernetes-replicator works fine without restarts

Environment:

Additional context

logs of failed pod:

time="2022-09-15T12:43:02Z" level=info msg="Secret replicator/docker-cred-secret to be replicated to: [*]" kind=Secret source=replicator/docker-cred-secret
time="2022-09-15T12:43:02Z" level=info msg="Checking if auto-ns-40888236/docker-cred-secret exists? false" kind=Secret source=replicator/docker-cred-secret target=auto-ns-40888236/docker-cred-secret
time="2022-09-15T12:43:02Z" level=debug msg="Creating a new secret secret auto-ns-40888236/docker-cred-secret" kind=Secret source=replicator/docker-cred-secret target=auto-ns-40888236/docker-cred-secret
time="2022-09-15T12:43:02Z" level=info msg="Replicated replicator/docker-cred-secret to: auto-ns-40888236" source=replicator/docker-cred-secret
fatal error: concurrent map iteration and map write

goroutine 5280 [running]:
runtime.throw(0x17f2bd1, 0x26)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/panic.go:1117 +0x72 fp=0xc0070db550 sp=0xc0070db520 pc=0x4361f2
runtime.mapiternext(0xc0070db8a0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/map.go:858 +0x54c fp=0xc0070db5d0 sp=0xc0070db550 pc=0x410aac
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).NamespaceAdded(0xc00003d130, 0xc000676160)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:153 +0x208 fp=0xc0070dbfb8 sp=0xc0070db5d0 pc=0x146a6e8
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).NamespaceAdded-fm(0xc000676160)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:151 +0x3d fp=0xc0070dbfd8 sp=0xc0070dbfb8 pc=0x14733fd
runtime.goexit()
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0070dbfe0 sp=0xc0070dbfd8 pc=0x46ba81
created by github.com/mittwald/kubernetes-replicator/replicate/common.(*NamespaceWatcher).create.func1.1
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/namespaces.go:40 +0x70

and so on (see a comment)

elitvinov commented 1 year ago 
time="2022-09-15T12:43:02Z" level=info msg="Secret replicator/docker-cred-secret to be replicated to: [*]" kind=Secret source=replicator/docker-cred-secret
time="2022-09-15T12:43:02Z" level=info msg="Checking if auto-ns-40888236/docker-cred-secret exists? false" kind=Secret source=replicator/docker-cred-secret target=auto-ns-40888236/docker-cred-secret
time="2022-09-15T12:43:02Z" level=debug msg="Creating a new secret secret auto-ns-40888236/docker-cred-secret" kind=Secret source=replicator/docker-cred-secret target=auto-ns-40888236/docker-cred-secret
time="2022-09-15T12:43:02Z" level=info msg="Replicated replicator/docker-cred-secret to: auto-ns-40888236" source=replicator/docker-cred-secret
fatal error: concurrent map iteration and map write

goroutine 5280 [running]:
runtime.throw(0x17f2bd1, 0x26)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/panic.go:1117 +0x72 fp=0xc0070db550 sp=0xc0070db520 pc=0x4361f2
runtime.mapiternext(0xc0070db8a0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/map.go:858 +0x54c fp=0xc0070db5d0 sp=0xc0070db550 pc=0x410aac
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).NamespaceAdded(0xc00003d130, 0xc000676160)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:153 +0x208 fp=0xc0070dbfb8 sp=0xc0070db5d0 pc=0x146a6e8
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).NamespaceAdded-fm(0xc000676160)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:151 +0x3d fp=0xc0070dbfd8 sp=0xc0070dbfb8 pc=0x14733fd
runtime.goexit()
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0070dbfe0 sp=0xc0070dbfd8 pc=0x46ba81
created by github.com/mittwald/kubernetes-replicator/replicate/common.(*NamespaceWatcher).create.func1.1
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/namespaces.go:40 +0x70

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7eff83617358, 0x72, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/netpoll.go:227 +0x55
internal/poll.(*pollDesc).wait(0xc000378818, 0x72, 0x0, 0x0, 0x17d3c23)
    /opt/hostedtoolcache/go/1.16.10/x64/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /opt/hostedtoolcache/go/1.16.10/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000378800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc000378800, 0x74edff6a613bba7e, 0x0, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000398678, 0x63231dd5, 0xc0003fdcc8, 0x490546)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000398678, 0xc0003fdd18, 0x18, 0xc000000180, 0x6c7bbb)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc0001b2460, 0x19b7100, 0xc000398678, 0x0, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/http/server.go:2961 +0x285
net/http.(*Server).ListenAndServe(0xc0001b2460, 0xc0001b2460, 0x8)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/http/server.go:2890 +0xba
net/http.ListenAndServe(...)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/http/server.go:3144
main.main()
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/main.go:103 +0x4ce

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x23577c0)
    /home/runner/go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
    /home/runner/go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:420 +0xdf

goroutine 7 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0003a1608, 0x275)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0003a15f8)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003a15e0, 0xc0003bc870, 0x0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/delta_fifo.go:525 +0x98
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003aba70)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:183 +0x42
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000486f90)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000486f90, 0x198d8a0, 0xc00049c000, 0xc00048a001, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000486f90, 0x3b9aca00, 0x0, 0xc0003c2601, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003aba70, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:154 +0x2e5
created by github.com/mittwald/kubernetes-replicator/replicate/common.(*NamespaceWatcher).create.func1
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/namespaces.go:70 +0x33a

goroutine 8 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0003a1568, 0x1493)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0003a1558)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003a1540, 0xc0003bc750, 0x0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/delta_fifo.go:525 +0x98
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003ab9e0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:183 +0x42
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000094f38)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002b9bf38, 0x198d8a0, 0xc0004020f0, 0xc00040a001, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000094f38, 0x3b9aca00, 0x0, 0xc000315901, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003ab9e0, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:154 +0x2e5
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).Run(0xc00003d130)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:146 +0x122
created by main.main
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/main.go:88 +0x24c

goroutine 9 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0003a16a8, 0x14f4)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0003a1698)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003a1680, 0xc0003bcab0, 0x0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/delta_fifo.go:525 +0x98
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003abb00)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:183 +0x42
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000482f38)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0004aff38, 0x198d8a0, 0xc00049c6f0, 0xc00048a301, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000482f38, 0x3b9aca00, 0x0, 0xc000315a01, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003abb00, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:154 +0x2e5
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).Run(0xc00003d1e0)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:146 +0x122
created by main.main
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/main.go:90 +0x272

goroutine 10 [sync.Cond.Wait, 15 minutes]:
sync.runtime_notifyListWait(0xc0003a1748, 0x1)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0003a1738)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003a1720, 0xc0003bcc60, 0x0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/delta_fifo.go:525 +0x98
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003abb90)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:183 +0x42
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000488f38)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000b05f38, 0x198d8a0, 0xc00049c360, 0xc00048a101, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000488f38, 0x3b9aca00, 0x0, 0xc000315b01, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003abb90, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:154 +0x2e5
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).Run(0xc00003d290)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:146 +0x122
created by main.main
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/main.go:92 +0x298

goroutine 11 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0003a17e8, 0x139)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0003a17d8)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003a17c0, 0xc0003bce10, 0x0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/delta_fifo.go:525 +0x98
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003abc20)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:183 +0x42
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000098f38)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003885f38, 0x198d8a0, 0xc0003bd0e0, 0xc0003c0101, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000098f38, 0x3b9aca00, 0x0, 0xc0003c2801, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003abc20, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:154 +0x2e5
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).Run(0xc00003d340)
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/generic-replicator.go:146 +0x122
created by main.main
    /home/runner/work/kubernetes-replicator/kubernetes-replicator/main.go:94 +0x2be

goroutine 17 [chan receive, 15 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc000092300, 0xc0003ab9e0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:129 +0xa5

goroutine 18 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc00041e000, 0xc0c0e524c76de459, 0xaec8ddea65, 0x23574a0, 0x19956c0, 0xc002a9ad00, 0xc002b99b68, 0xc0060e13e0, 0xc000092300, 0x0, ...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:468 +0x165
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00041e000, 0xc000092300, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:428 +0x738
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0004246c8)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002b99ec8, 0x198d880, 0xc00040e050, 0x1, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00041e000, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:220 +0x1ed
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00040a030, 0xc000410018)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:71 +0x65

goroutine 56 [select, 15 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc00041e000, 0xc000092300, 0xc000093d40, 0xc0060e13e0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:373 +0x125
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:367 +0x2ed

goroutine 33 [chan receive, 15 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc000092300, 0xc0003aba70)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:129 +0xa5

goroutine 34 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000498000, 0xc0c0e537de1bb527, 0xc0917fb334, 0x23574a0, 0x19956c0, 0xc000776d40, 0xc001a21b68, 0xc000330480, 0xc000092300, 0x0, ...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:468 +0x165
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000498000, 0xc000092300, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:428 +0x738
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000064ec8)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001a21ec8, 0x198d880, 0xc000494000, 0x1, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000498000, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:220 +0x1ed
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00048a010, 0xc00049a000)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:71 +0x65

goroutine 84 [select, 15 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc000498000, 0xc000092300, 0xc00042c3c0, 0xc000330480)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:373 +0x125
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:367 +0x2ed

goroutine 12 [chan receive, 15 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc000092300, 0xc0003abc20)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:129 +0xa5

goroutine 13 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0001b2540, 0xc0c0e53820a4bcd6, 0xc0cfa384e5, 0x23574a0, 0x19956c0, 0xc00a26a3c0, 0xc003889b68, 0xc000a16060, 0xc000092300, 0x0, ...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:468 +0x165
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0001b2540, 0xc000092300, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:428 +0x738
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00005f6c8)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003889ec8, 0x198d880, 0xc0003c4280, 0x1, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0001b2540, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:220 +0x1ed
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0003c0160, 0xc0003986a8)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:71 +0x65

goroutine 54 [select, 15 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0001b2540, 0xc000092300, 0xc000092ea0, 0xc000a16060)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:373 +0x125
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:367 +0x2ed

goroutine 36 [chan receive, 15 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc000092300, 0xc0003abb90)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:129 +0xa5

goroutine 37 [select, 6 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0004980e0, 0xc0c0e4fa9f1e528a, 0x87875efe86, 0x23574a0, 0x19956c0, 0xc000777600, 0xc00010fb68, 0xc000884180, 0xc000092300, 0x0, ...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:468 +0x165
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0004980e0, 0xc000092300, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:428 +0x738
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000064ec8)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00010fec8, 0x198d880, 0xc0004940f0, 0x1, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0004980e0, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:220 +0x1ed
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00048a1a0, 0xc00049a090)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:71 +0x65

goroutine 99 [select, 15 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0004980e0, 0xc000092300, 0xc000092900, 0xc000884180)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:373 +0x125
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:367 +0x2ed

goroutine 41 [chan receive, 15 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc000092300, 0xc0003abb00)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/controller.go:129 +0xa5

goroutine 42 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0004981c0, 0xc0c0e552a4ecbac4, 0xd9820326c7, 0x23574a0, 0x19956c0, 0xc0009637c0, 0xc003aefb68, 0xc0062d1980, 0xc000092300, 0x0, ...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:468 +0x165
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0004981c0, 0xc000092300, 0x0, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:428 +0x738
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000420ec8)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003aefec8, 0x198d880, 0xc0004942d0, 0x1, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0004981c0, 0xc000092300)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:220 +0x1ed
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00048a320, 0xc00049a138)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/wait/wait.go:71 +0x65

goroutine 82 [IO wait]:
internal/poll.runtime_pollWait(0x7eff83617270, 0x72, 0xffffffffffffffff)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/netpoll.go:227 +0x55
internal/poll.(*pollDesc).wait(0xc000444118, 0x72, 0xbe00, 0xbee2, 0xffffffffffffffff)
    /opt/hostedtoolcache/go/1.16.10/x64/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /opt/hostedtoolcache/go/1.16.10/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000444100, 0xc008e98000, 0xbee2, 0xbee2, 0x0, 0x0, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000444100, 0xc008e98000, 0xbee2, 0xbee2, 0x95f2, 0xc008e9a8eb, 0x5)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00042e038, 0xc008e98000, 0xbee2, 0xbee2, 0x0, 0x0, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0040d2e70, 0xc008e98000, 0xbee2, 0xbee2, 0x95f2, 0xc000448800, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc00044a5f8, 0x198bd00, 0xc0040d2e70, 0x40b6a5, 0x1629440, 0x1773600)
    /opt/hostedtoolcache/go/1.16.10/x64/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc00044a380, 0x198dd00, 0xc00042e038, 0x5, 0xc00042e038, 0x2877)
    /opt/hostedtoolcache/go/1.16.10/x64/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc00044a380, 0x0, 0x0, 0x3)
    /opt/hostedtoolcache/go/1.16.10/x64/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
    /opt/hostedtoolcache/go/1.16.10/x64/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc00044a380, 0xc000587000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/crypto/tls/conn.go:1276 +0x165
bufio.(*Reader).Read(0xc0004e8d20, 0xc000498498, 0x9, 0x9, 0xc005c964e0, 0xc006f02ca8, 0xc000487c68)
    /opt/hostedtoolcache/go/1.16.10/x64/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x198bba0, 0xc0004e8d20, 0xc000498498, 0x9, 0x9, 0x9, 0x0, 0x6c4f18073adb01, 0xc006f02c90)
    /opt/hostedtoolcache/go/1.16.10/x64/src/io/io.go:328 +0x87
io.ReadFull(...)
    /opt/hostedtoolcache/go/1.16.10/x64/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader(0xc000498498, 0x9, 0x9, 0x198bba0, 0xc0004e8d20, 0x0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000498460, 0xc002bb35c0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000487fa8, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:1821 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000481800)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:1743 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:695 +0x6c5

goroutine 4243 [sync.Cond.Wait, 6 minutes]:
sync.runtime_notifyListWait(0xc0065a7900, 0xc000000000)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0065a78f0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
golang.org/x/net/http2.(*pipe).Read(0xc0065a78e8, 0xc0069d1c00, 0x200, 0x200, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc0065a78c0, 0xc0069d1c00, 0x200, 0x200, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:2110 +0xaf
encoding/json.(*Decoder).refill(0xc00a88c000, 0x300000000000, 0x7eff82fd91a0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc00a88c000, 0x0, 0x0, 0x15ef480)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc00a88c000, 0x16198a0, 0xc0043f5ea8, 0x203000, 0x203000)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:63 +0x7c
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc006af4690, 0xc003cc8000, 0x400, 0x400, 0x40, 0x38, 0x1673ec0)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc006922eb0, 0x0, 0x1995440, 0xc000777640, 0xc000539dc8, 0x4672e5, 0xc00014ea80, 0x200000003, 0xc00014ea80)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0064a2560, 0x203000, 0x203000, 0x1, 0x0, 0xc000539e98, 0x4d53dc)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000777600)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:76 +0xea

goroutine 4733 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc006f02ca0, 0x42f)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc006f02c90)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
golang.org/x/net/http2.(*pipe).Read(0xc006f02c88, 0xc00699e001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc006f02c60, 0xc00699e001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:2110 +0xaf
encoding/json.(*Decoder).refill(0xc006f03e40, 0xa, 0x9)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc006f03e40, 0x0, 0x0, 0x15ef480)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc006f03e40, 0x16198a0, 0xc0040d3098, 0x203000, 0x203000)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:63 +0x7c
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc008ef7140, 0xc007df6a80, 0x4000, 0x4a80, 0x40, 0x38, 0x1673ec0)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002aa5900, 0x0, 0x1995440, 0xc002a9b080, 0x0, 0x0, 0x0, 0xc0004b3e64, 0x2)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00007dc80, 0xc0004b3e60, 0x0, 0x1, 0x1, 0x1, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002a9ad00)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:76 +0xea

goroutine 4956 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00634ab40, 0x26)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc00634ab30)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
golang.org/x/net/http2.(*pipe).Read(0xc00634ab28, 0xc0053c8001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc00634ab00, 0xc0053c8001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:2110 +0xaf
encoding/json.(*Decoder).refill(0xc004e85e40, 0xa, 0x9)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc004e85e40, 0x0, 0x0, 0x15ef480)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc004e85e40, 0x16198a0, 0xc0039426d8, 0x203002, 0x203002)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:63 +0x7c
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000cc5c20, 0xc004052800, 0x400, 0x400, 0x40, 0x38, 0x1673ec0)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc003502780, 0x0, 0x1995440, 0xc00a6a0a40, 0x0, 0x0, 0x0, 0xc002b9de64, 0x2)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00007c840, 0xc002b9de60, 0x0, 0x1, 0x1, 0x1, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00a26a3c0)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:76 +0xea

goroutine 59 [select, 15 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0004981c0, 0xc000092300, 0xc001157d40, 0xc0062d1980)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:373 +0x125
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:367 +0x2ed

goroutine 5028 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0092e7e80, 0x52)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0092e7e70)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
golang.org/x/net/http2.(*pipe).Read(0xc0092e7e68, 0xc0006de001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc0092e7e40, 0xc0006de001, 0xdff, 0xdff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:2110 +0xaf
encoding/json.(*Decoder).refill(0xc006d971e0, 0xa, 0x9)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc006d971e0, 0x0, 0x0, 0x15ef480)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc006d971e0, 0x16198a0, 0xc003f72990, 0x203002, 0x203002)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:63 +0x7c
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc006e17bc0, 0xc003ce9300, 0x1000, 0x1300, 0x40, 0x38, 0x1673ec0)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc006922910, 0x0, 0x1995440, 0xc00a26ab80, 0x0, 0x0, 0x0, 0xc000073e64, 0x2)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00658edc0, 0xc000073e60, 0x0, 0x1, 0x1, 0x1, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000776d40)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:105 +0xe5
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:76 +0xea

goroutine 5211 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00aaa14e0, 0x47)
    /opt/hostedtoolcache/go/1.16.10/x64/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc00aaa14d0)
    /opt/hostedtoolcache/go/1.16.10/x64/src/sync/cond.go:56 +0x99
golang.org/x/net/http2.(*pipe).Read(0xc00aaa14c8, 0xc006820001, 0x3fdff, 0x3fdff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/pipe.go:65 +0x97
golang.org/x/net/http2.transportResponseBody.Read(0xc00aaa14a0, 0xc006820001, 0x3fdff, 0x3fdff, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20210520170846-37e1c6afe023/http2/transport.go:2110 +0xaf
encoding/json.(*Decoder).refill(0xc00aaa1760, 0xa, 0x9)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:165 +0xeb
encoding/json.(*Decoder).readValue(0xc00aaa1760, 0x0, 0x0, 0x15ef480)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:140 +0x1ff
encoding/json.(*Decoder).Decode(0xc00aaa1760, 0x16198a0, 0xc0040d2cf0, 0x203000, 0x203000)
    /opt/hostedtoolcache/go/1.16.10/x64/src/encoding/json/stream.go:63 +0x7c
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002ba1110, 0xc006db0000, 0x40000, 0x50000, 0x40, 0x38, 0x1673ec0)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/util/framer/framer.go:152 +0x1a8
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000495680, 0x0, 0x1995440, 0xc002a9afc0, 0x0, 0x0, 0x0, 0xc0020fbe64, 0x2)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0003acf00, 0xc0020fbe60, 0x0, 0x1, 0x1, 0x1, 0x0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.22.4/rest/watch/decoder.go:49 +0x6e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0009637c0)
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:105 +0xe5
    created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
    /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.22.4/pkg/watch/streamwatcher.go:76 +0xea
mittwald-machine commented 1 year ago

There has not been any activity to this issue in the last 14 days. It will automatically be closed after 7 more days. Remove the stale label to prevent this.

martin-helmich commented 1 year ago

Thanks for the thorough report, and apologies for the late reaction. I'll investigate when I find the time. In the meantime, PRs are also welcome. 🙂

mFranz82 commented 1 month ago

We are facing the same problem with the latest version