search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.14k stars 176 forks source link

[BUG] `kbcli kubeblocks install` failed: kubeblocks pod CrashLoopBackOff #2243

Closed JashBook closed 1 year ago

JashBook commented 1 year ago

Describe the bug kbcli kubeblocks install failed: kubeblocks pod CrashLoopBackOff

kbcli version
Kubernetes: v1.24.10-eks-48e63af
KubeBlocks: 0.5.0-alpha.1
kbcli: 0.5.0-alpha.1

To Reproduce Steps to reproduce the behavior:

  1. install kubeblocks 
    kbcli kubeblocks install --version 0.5.0-alpha.1
  2. See error 
    kubectl get pod -n kb-system
NAME                                                     READY   STATUS             RESTARTS      AGE
kb-addon-alertmanager-webhook-adaptor-5bd87c4697-pqshr   2/2     Running            0             6m7s
kb-addon-grafana-77c7f45cd6-j2t5c                        3/3     Running            0             4m21s
kb-addon-prometheus-alertmanager-0                       2/2     Running            0             8m5s
kb-addon-prometheus-server-0                             2/2     Running            0             8m5s
kb-addon-snapshot-controller-5b7c5df74c-9zxqc            1/1     Running            0             7m14s
kubeblocks-664dfc5c44-gpvxc                              0/1     CrashLoopBackOff   6 (71s ago)   9m8s

    logs kubeblocks pod

    
kubectl logs kubeblocks-664dfc5c44-gpvxc -n kb-system
2023-03-27T10:59:05.751Z    INFO    setup   unable read in config, errors ignored
2023-03-27T10:59:05.751Z    INFO    setup   config file: 
2023-03-27T10:59:05.751Z    INFO    setup   config settings: map[alsologtostderr:false cert_dir:/tmp/k8s-webhook-server/serving-certs cm_namespace:kb-system cm_recon_retry_duration_ms:100 enable_debug_sysaccounts:false health_probe_bind_address::8081 kill_container_signal:SIGKILL kubeblocks_addon_sa_name:kubeblocks-addon-installer kubeblocks_image:registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks:0.5.0-alpha.1 kubeblocks_serviceaccount_name:kubeblocks kubeconfig: leader_elect:true log_backtrace_at::0 log_dir: logtostderr:false maxconcurrentreconciles_addon:4 maxconcurrentreconciles_clusterdef:4 maxconcurrentreconciles_clusterversion:4 maxconcurrentreconciles_dataprotection:4 metrics_bind_address::8080 pod_min_ready_seconds:10 probe_service_grpc_port:50001 probe_service_http_port:3501 probe_service_log_level:info stderrthreshold:2 v:0 vmodule: volumesnapshot:true zap_devel:false zap_encoder:console zap_log_level: zap_stacktrace_level: zap_time_encoding:iso8601]
2023-03-27T10:59:06.206Z    INFO    controller-runtime.metrics  Metrics server is starting to listen    {"addr": ":8080"}
2023-03-27T10:59:06.209Z    INFO    setup   starting manager
2023-03-27T10:59:06.209Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-03-27T10:59:06.209Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
I0327 10:59:06.210007       1 leaderelection.go:248] attempting to acquire leader lease kb-system/001c317f.kubeblocks.io...
I0327 10:59:23.519157       1 leaderelection.go:258] successfully acquired lease kb-system/001c317f.kubeblocks.io
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.StatefulSet"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Deployment"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Service"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ConfigMap"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PodDisruptionBudget"}
2023-03-27T10:59:23.519Z    INFO    Starting Controller {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "source": "kind source: *v1alpha1.ClusterVersion"}
2023-03-27T10:59:23.519Z    INFO    Starting Controller {"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "source": "kind source: *v1alpha1.ClusterDefinition"}
2023-03-27T10:59:23.519Z    INFO    Starting Controller {"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "backuppolicy", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "BackupPolicy", "source": "kind source: *v1alpha1.BackupPolicy"}
2023-03-27T10:59:23.519Z    INFO    Starting Controller {"controller": "backuppolicy", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "BackupPolicy"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "backuptool", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "BackupTool", "source": "kind source: *v1alpha1.BackupTool"}
2023-03-27T10:59:23.519Z    INFO    Starting Controller {"controller": "backuptool", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "BackupTool"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "backup", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "Backup", "source": "kind source: *v1alpha1.Backup"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "backup", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "Backup", "source": "kind source: *v1.Job"}
2023-03-27T10:59:23.519Z    INFO    Starting EventSource    {"controller": "backup", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "Backup", "source": "kind source: *v1.PartialObjectMetadata"}
2023-03-27T10:59:23.519Z    INFO    Starting Controller {"controller": "backup", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "Backup"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "cronjob", "controllerGroup": "batch", "controllerKind": "CronJob", "source": "kind source: *v1.CronJob"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "cronjob", "controllerGroup": "batch", "controllerKind": "CronJob", "source": "kind source: *v1.Job"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "cronjob", "controllerGroup": "batch", "controllerKind": "CronJob"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.OpsRequest"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "restorejob", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "RestoreJob", "source": "kind source: *v1alpha1.RestoreJob"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "restorejob", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "RestoreJob"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "source": "kind source: *v1alpha1.Addon"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1alpha1.ConfigConstraint"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1.ConfigMap"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Job"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2023-03-27T10:59:23.520Z    INFO    Starting EventSource    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-03-27T10:59:23.520Z    INFO    Starting Controller {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim"}
2023-03-27T10:59:23.521Z    INFO    Starting EventSource    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "source": "kind source: *v1.Event"}
2023-03-27T10:59:23.521Z    INFO    Starting EventSource    {"controller": "deployment", "controllerGroup": "apps", "controllerKind": "Deployment", "source": "kind source: *v1.Deployment"}
2023-03-27T10:59:23.521Z    INFO    Starting Controller {"controller": "event", "controllerGroup": "", "controllerKind": "Event"}
2023-03-27T10:59:23.521Z    INFO    Starting EventSource    {"controller": "deployment", "controllerGroup": "apps", "controllerKind": "Deployment", "source": "kind source: *v1.ReplicaSet"}
2023-03-27T10:59:23.521Z    INFO    Starting Controller {"controller": "deployment", "controllerGroup": "apps", "controllerKind": "Deployment"}
2023-03-27T10:59:23.521Z    INFO    Starting EventSource    {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "source": "kind source: *v1.StatefulSet"}
2023-03-27T10:59:23.521Z    INFO    Starting EventSource    {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "source": "kind source: *v1.Pod"}
2023-03-27T10:59:23.521Z    INFO    Starting Controller {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet"}
2023-03-27T10:59:23.621Z    INFO    Starting workers    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "worker count": 1}
2023-03-27T10:59:23.621Z    INFO    Starting workers    {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "worker count": 1}
2023-03-27T10:59:23.622Z    INFO    Starting workers    {"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "worker count": 4}
2023-03-27T10:59:23.622Z    INFO    Starting workers    {"controller": "backuptool", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "BackupTool", "worker count": 4}
2023-03-27T10:59:23.623Z    INFO    Starting workers    {"controller": "restorejob", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "RestoreJob", "worker count": 4}
2023-03-27T10:59:23.623Z    INFO    Starting workers    {"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "worker count": 4}
2023-03-27T10:59:23.623Z    INFO    Starting workers    {"controller": "cronjob", "controllerGroup": "batch", "controllerKind": "CronJob", "worker count": 1}
2023-03-27T10:59:23.621Z    INFO    Starting workers    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1}
2023-03-27T10:59:23.631Z    INFO    Starting workers    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "worker count": 1}
2023-03-27T10:59:23.631Z    INFO    Starting workers    {"controller": "deployment", "controllerGroup": "apps", "controllerKind": "Deployment", "worker count": 1}
2023-03-27T10:59:23.631Z    INFO    Starting workers    {"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "worker count": 4}
2023-03-27T10:59:23.631Z    INFO    Starting workers    {"controller": "backuppolicy", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "BackupPolicy", "worker count": 4}
2023-03-27T10:59:23.631Z    INFO    Starting workers    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "worker count": 1}
2023-03-27T10:59:23.632Z    INFO    Starting workers    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "worker count": 1}
2023-03-27T10:59:23.632Z    INFO    Starting workers    {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "worker count": 1}
2023-03-27T10:59:23.632Z    INFO    Starting workers    {"controller": "backup", "controllerGroup": "dataprotection.kubeblocks.io", "controllerKind": "Backup", "worker count": 4}
2023-03-27T10:59:23.632Z    INFO    Starting workers    {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "worker count": 1}
2023-03-27T10:59:23.644Z    INFO    KubeAPIWarningLogger    child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x4022200360 stack=[0x4022200000, 0x4042200000]
fatal error: stack overflow

runtime stack: runtime.throw({0x1a2aa41?, 0x2c68860?}) /usr/local/go/src/runtime/panic.go:1047 +0x40 fp=0x40008bde00 sp=0x40008bddd0 pc=0x485f0 runtime.newstack() /usr/local/go/src/runtime/stack.go:1103 +0x474 fp=0x40008bdfb0 sp=0x40008bde00 pc=0x62684 runtime.morestack() /usr/local/go/src/runtime/asm_arm64.s:316 +0x70 fp=0x40008bdfb0 sp=0x40008bdfb0 pc=0x76ba0

goroutine 676 [running]: github.com/apecloud/kubeblocks/controllers/extensions.(AddonReconciler).Event(0x4000634810?, {0x1d560c0?, 0x40010e09c0?}, {0x1a1b8f5?, 0x6?}, {0x1a25c3f?, 0xc?}, {0x40017c0860?, 0x19?}) /workspace/controllers/extensions/addon_controller.go:200 +0x80 fp=0x4022200360 sp=0x4022200360 pc=0x158d510 github.com/apecloud/kubeblocks/controllers/extensions.(AddonReconciler).Event(0x0?, {0x1d560c0?, 0x40010e09c0?}, {0x1a1b8f5?, 0x0?}, {0x1a25c3f?, 0x0?}, {0x40017c0860?, 0x0?}) /workspace/controllers/extensions/addon_controller.go:204 +0x3c fp=0x40222003c0 sp=0x4022200360 pc=0x158d4cc github.com/apecloud/kubeblocks/controllers/extensions.(AddonReconciler).Event(0x0?, {0x1d560c0?, 0x40010e09c0?}, {0x1a1b8f5?, 0x0?}, {0x1a25c3f?, 0x0?}, {0x40017c0860?, 0x0?}) /workspace/controllers/extensions/addon_controller.go:204 +0x3c fp=0x4022200420 sp=0x40222003c0 pc=0x158d4cc github.com/apecloud/kubeblocks/controllers/extensions.(AddonReconciler).Event(0x0?, {0x1d560c0?, 0x40010e09c0?}, {0x1a1b8f5?, 0x0?}, {0x1a25c3f?, 0x0?}, {0x40017c0860?, 0x0?}) ... /workspace/controllers/extensions/addon_controller.go:204 +0x3c fp=0x40222027c0 sp=0x4022202760 pc=0x158d4cc github.com/apecloud/kubeblocks/controllers/extensions.(AddonReconciler).Event(0x0?, {0x1d560c0?, 0x40010e09c0?}, {0x1a1b8f5?, 0x0?}, {0x1a25c3f?, 0x0?}, {0x40017c0860?, 0x0?}) /workspace/controllers/extensions/addon_controller.go:204 +0x3c fp=0x4022202820 sp=0x40222027c0 pc=0x158d4cc github.com/apecloud/kubeblocks/controllers/extensions.(AddonReconciler).Event(0x0?, {0x1d560c0?, 0x40010e09c0?}, {0x1a1b8f5?, 0x0?}, {0x1a25c3f?, 0x0?}, {0x40017c0860?, 0x0?}) /workspace/controllers/extensions/addon_controller.go:204 +0x3c fp=0x4022202880 sp=0x4022202820 pc=0x158d4cc ...additional frames elided... created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:231 +0x294

goroutine 1 [select]: runtime.gopark(0x400089fa98?, 0x2?, 0xc0?, 0x15?, 0x400089f934?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x40008c37c0 sp=0x40008c37a0 pc=0x4afb4 runtime.selectgo(0x40008c3a98, 0x400089f930, 0x40000aab80?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x40008c38d0 sp=0x40008c37c0 pc=0x5b0d4 sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start(0x4000096a80, {0x1d6d510, 0x40000aab40}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/manager/internal.go:509 +0x530 fp=0x40008c3ae0 sp=0x40008c38d0 pc=0xfda650 main.main() /workspace/cmd/manager/main.go:436 +0x27fc fp=0x40008c3f70 sp=0x40008c3ae0 pc=0x1599c8c runtime.main() /usr/local/go/src/runtime/proc.go:250 +0x254 fp=0x40008c3fd0 sp=0x40008c3f70 pc=0x4abe4 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40008c3fd0 sp=0x40008c3fd0 pc=0x78f94

goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000074fa0 sp=0x4000074f80 pc=0x4afb4 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:369 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:302 +0xb4 fp=0x4000074fd0 sp=0x4000074fa0 pc=0x4ae44 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000074fd0 sp=0x4000074fd0 pc=0x78f94 created by runtime.init.6 /usr/local/go/src/runtime/proc.go:290 +0x24

goroutine 3 [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000075770 sp=0x4000075750 pc=0x4afb4 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:369 runtime.bgsweep(0x0?) /usr/local/go/src/runtime/mgcsweep.go:297 +0x10c fp=0x40000757b0 sp=0x4000075770 pc=0x3675c runtime.gcenable.func1() /usr/local/go/src/runtime/mgc.go:178 +0x28 fp=0x40000757d0 sp=0x40000757b0 pc=0x2ab48 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40000757d0 sp=0x40000757d0 pc=0x78f94 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:178 +0x70

goroutine 4 [GC scavenge wait]: runtime.gopark(0x93b1f1e0c51?, 0x23b43a0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000075f50 sp=0x4000075f30 pc=0x4afb4 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:369 runtime.(*scavengerState).park(0x2d5e3c0) /usr/local/go/src/runtime/mgcscavenge.go:389 +0x5c fp=0x4000075f80 sp=0x4000075f50 pc=0x346dc runtime.bgscavenge(0x0?) /usr/local/go/src/runtime/mgcscavenge.go:622 +0xac fp=0x4000075fb0 sp=0x4000075f80 pc=0x34cac runtime.gcenable.func2() /usr/local/go/src/runtime/mgc.go:179 +0x28 fp=0x4000075fd0 sp=0x4000075fb0 pc=0x2aae8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000075fd0 sp=0x4000075fd0 pc=0x78f94 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:179 +0xb4

goroutine 17 [finalizer wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000070580 sp=0x4000070560 pc=0x4afb4 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:369 runtime.runfinq() /usr/local/go/src/runtime/mfinal.go:180 +0x128 fp=0x40000707d0 sp=0x4000070580 pc=0x29d68 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40000707d0 sp=0x40000707d0 pc=0x78f94 created by runtime.createfing /usr/local/go/src/runtime/mfinal.go:157 +0x94

goroutine 18 [GC worker (idle)]: runtime.gopark(0x93b2032d465?, 0x2?, 0x7f?, 0x10?, 0x0?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000070f40 sp=0x4000070f20 pc=0x4afb4 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1235 +0xec fp=0x4000070fd0 sp=0x4000070f40 pc=0x2cfcc runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000070fd0 sp=0x4000070fd0 pc=0x78f94 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1159 +0x28

goroutine 5 [GC worker (idle)]: runtime.gopark(0x2d94540?, 0x2?, 0x27?, 0x65?, 0x4000090000?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000074740 sp=0x4000074720 pc=0x4afb4 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1235 +0xec fp=0x40000747d0 sp=0x4000074740 pc=0x2cfcc runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40000747d0 sp=0x40000747d0 pc=0x78f94 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1159 +0x28

goroutine 19 [sleep]: runtime.gopark(0x936b9c28b78?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000071740 sp=0x4000071720 pc=0x4afb4 time.Sleep(0x6fc23ac00) /usr/local/go/src/runtime/time.go:195 +0x120 fp=0x4000071780 sp=0x4000071740 pc=0x75c80 sigs.k8s.io/controller-runtime/pkg/log.init.0.func1() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/log/log.go:63 +0x30 fp=0x40000717d0 sp=0x4000071780 pc=0xea33c0 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40000717d0 sp=0x40000717d0 pc=0x78f94 created by sigs.k8s.io/controller-runtime/pkg/log.init.0 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/log/log.go:62 +0x24

goroutine 238 [chan receive]: runtime.gopark(0x40000c1c98?, 0x40002da670?, 0x18?, 0x8f?, 0x24ae0c?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000778ee0 sp=0x4000778ec0 pc=0x4afb4 runtime.chanrecv(0x4000869f20, 0x0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x47c fp=0x4000778f70 sp=0x4000778ee0 pc=0x161bc runtime.chanrecv1(0x400082a0b0?, 0x2?) /usr/local/go/src/runtime/chan.go:442 +0x14 fp=0x4000778fa0 sp=0x4000778f70 pc=0x15d04 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:171 +0x38 fp=0x4000778fd0 sp=0x4000778fa0 pc=0xf96ed8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000778fd0 sp=0x4000778fd0 pc=0x78f94 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:170 +0x184

goroutine 33 [chan receive]: runtime.gopark(0x4000071ef8?, 0x75ccc?, 0x0?, 0x45?, 0x40004a4008?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000071ea0 sp=0x4000071e80 pc=0x4afb4 runtime.chanrecv(0x4000494000, 0x4000071f90, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x47c fp=0x4000071f30 sp=0x4000071ea0 pc=0x161bc runtime.chanrecv2(0x6fc23ac00?, 0x0?) /usr/local/go/src/runtime/chan.go:447 +0x14 fp=0x4000071f60 sp=0x4000071f30 pc=0x15d24 github.com/golang/glog.(*loggingT).flushDaemon(0x0?) /go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:882 +0x5c fp=0x4000071fb0 sp=0x4000071f60 pc=0x1244e7c github.com/golang/glog.init.0.func1() /go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:410 +0x28 fp=0x4000071fd0 sp=0x4000071fb0 pc=0x1243148 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000071fd0 sp=0x4000071fd0 pc=0x78f94 created by github.com/golang/glog.init.0 /go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:410 +0x200

goroutine 267 [chan receive]: runtime.gopark(0x0?, 0x0?, 0x40?, 0x6e?, 0x40007298a8?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x400076ae30 sp=0x400076ae10 pc=0x4afb4 runtime.chanrecv(0x4000869f20, 0x0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x47c fp=0x400076aec0 sp=0x400076ae30 pc=0x161bc runtime.chanrecv1(0x4000729860?, 0x1d6d510?) /usr/local/go/src/runtime/chan.go:442 +0x14 fp=0x400076aef0 sp=0x400076aec0 pc=0x15d04 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start(0x4000729860, {0x1d6d510, 0x4000592f40}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:247 +0x1d0 fp=0x400076af50 sp=0x400076aef0 pc=0xf96500 sigs.k8s.io/controller-runtime/pkg/manager.(runnableGroup).reconcile.func1(0x4000715ea0) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/manager/runnable_group.go:219 +0xd8 fp=0x400076afb0 sp=0x400076af50 pc=0xfdd6f8 sigs.k8s.io/controller-runtime/pkg/manager.(runnableGroup).reconcile.func2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/manager/runnable_group.go:222 +0x30 fp=0x400076afd0 sp=0x400076afb0 pc=0xfdd5e0 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400076afd0 sp=0x400076afd0 pc=0x78f94 created by sigs.k8s.io/controller-runtime/pkg/manager.(runnableGroup).reconcile /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/manager/runnable_group.go:203 +0x220

goroutine 37 [IO wait]: runtime.gopark(0x4000158658?, 0x40000f9628?, 0x38?, 0x96?, 0x2aae?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x40000f95a0 sp=0x40000f9580 pc=0x4afb4 runtime.netpollblock(0x0?, 0xffffffff?, 0xff?) /usr/local/go/src/runtime/netpoll.go:526 +0x160 fp=0x40000f95e0 sp=0x40000f95a0 pc=0x43a90 internal/poll.runtime_pollWait(0xffff6e341758, 0x72) /usr/local/go/src/runtime/netpoll.go:305 +0xa0 fp=0x40000f9610 sp=0x40000f95e0 pc=0x735a0 internal/poll.(pollDesc).wait(0x4000436500?, 0x4000820000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40000f9640 sp=0x40000f9610 pc=0xe79f8 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(FD).Read(0x4000436500, {0x4000820000, 0xa000, 0xa000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x1e8 fp=0x40000f96c0 sp=0x40000f9640 pc=0xe8d98 net.(netFD).Read(0x4000436500, {0x4000820000?, 0x400038e198?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x28 fp=0x40000f9710 sp=0x40000f96c0 pc=0x1ed328 net.(conn).Read(0x4000444000, {0x4000820000?, 0x40000f97e8?, 0x233f0?}) /usr/local/go/src/net/net.go:183 +0x34 fp=0x40000f9760 sp=0x40000f9710 pc=0x1fe1b4 crypto/tls.(atLeastReader).Read(0x40040b1758, {0x4000820000?, 0x0?, 0x2da858?}) /usr/local/go/src/crypto/tls/conn.go:787 +0x40 fp=0x40000f97b0 sp=0x40000f9760 pc=0x2da6c0 bytes.(Buffer).ReadFrom(0x400038e278, {0x1d4d880, 0x40040b1758}) /usr/local/go/src/bytes/buffer.go:202 +0x90 fp=0x40000f9810 sp=0x40000f97b0 pc=0x112470 crypto/tls.(Conn).readFromUntil(0x400038e000, {0x1d51c60?, 0x4000444000}, 0x74d6?) /usr/local/go/src/crypto/tls/conn.go:809 +0xd4 fp=0x40000f9850 sp=0x40000f9810 pc=0x2da8c4 crypto/tls.(Conn).readRecordOrCCS(0x400038e000, 0x0) /usr/local/go/src/crypto/tls/conn.go:616 +0xd8 fp=0x40000f9be0 sp=0x40000f9850 pc=0x2d8598 crypto/tls.(Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:582 crypto/tls.(Conn).Read(0x400038e000, {0x40003bb000, 0x1000, 0x46209c?}) /usr/local/go/src/crypto/tls/conn.go:1315 +0x178 fp=0x40000f9c50 sp=0x40000f9be0 pc=0x2dd658 bufio.(Reader).Read(0x400072e0c0, {0x400072a200, 0x9, 0x40040e0360?}) /usr/local/go/src/bufio/bufio.go:237 +0x1e8 fp=0x40000f9c90 sp=0x40000f9c50 pc=0x134118 io.ReadAtLeast({0x1d4d700, 0x400072e0c0}, {0x400072a200, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0xa8 fp=0x40000f9ce0 sp=0x40000f9c90 pc=0x84768 io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0x400072a200?, 0x9?, 0x40e0390?}, {0x1d4d700?, 0x400072e0c0?}) /go/pkg/mod/golang.org/x/net@v0.5.0/http2/frame.go:237 +0x58 fp=0x40000f9d30 sp=0x40000f9ce0 pc=0x45a4a8 golang.org/x/net/http2.(Framer).ReadFrame(0x400072a1c0) /go/pkg/mod/golang.org/x/net@v0.5.0/http2/frame.go:498 +0x84 fp=0x40000f9de0 sp=0x40000f9d30 pc=0x45aca4 golang.org/x/net/http2.(clientConnReadLoop).run(0x40000f9f88) /go/pkg/mod/golang.org/x/net@v0.5.0/http2/transport.go:2224 +0xfc fp=0x40000f9f40 sp=0x40000f9de0 pc=0x46d3bc golang.org/x/net/http2.(ClientConn).readLoop(0x4000710180) /go/pkg/mod/golang.org/x/net@v0.5.0/http2/transport.go:2119 +0x5c fp=0x40000f9fb0 sp=0x40000f9f40 pc=0x46c87c golang.org/x/net/http2.(Transport).newClientConn.func1() /go/pkg/mod/golang.org/x/net@v0.5.0/http2/transport.go:817 +0x28 fp=0x40000f9fd0 sp=0x40000f9fb0 pc=0x465a58 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40000f9fd0 sp=0x40000f9fd0 pc=0x78f94 created by golang.org/x/net/http2.(*Transport).newClientConn /go/pkg/mod/golang.org/x/net@v0.5.0/http2/transport.go:817 +0xa88

goroutine 365 [select]: runtime.gopark(0x400094dac0?, 0x3?, 0xef?, 0x66?, 0x400094d8d2?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x400094d6e0 sp=0x400094d6c0 pc=0x4afb4 runtime.selectgo(0x400094dac0, 0x400094d8cc, 0x400094d848?, 0x0, 0x400094d828?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x400094d7f0 sp=0x400094d6e0 pc=0x5b0d4 k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x2d5e100?}, {0x1d5ab20?, 0x40000aa500}, {0xffff6e20e550, 0x40005ed680}, {0x1d867b0?, 0x19e5480}, 0x0, ...) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:493 +0x13c fp=0x400094dbc0 sp=0x400094d7f0 pc=0xed3bec k8s.io/client-go/tools/cache.(Reflector).ListAndWatch(0x40008565a0, 0x400077fd40) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:329 +0x484 fp=0x400094de10 sp=0x400094dbc0 pc=0xed22b4 k8s.io/client-go/tools/cache.(Reflector).Run.func1() /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:223 +0x28 fp=0x400094de40 sp=0x400094de10 pc=0xed1d08 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40007fbed8?) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:157 +0x40 fp=0x400094de60 sp=0x400094de40 pc=0xc1c460 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000879680?, {0x1d517e0, 0x4000657680}, 0x1, 0x400077fd40) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:158 +0x90 fp=0x400094dee0 sp=0x400094de60 pc=0xc1c330 k8s.io/client-go/tools/cache.(Reflector).Run(0x40008565a0, 0x400077fd40) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:222 +0x174 fp=0x400094df60 sp=0x400094dee0 pc=0xed1ba4 k8s.io/client-go/tools/cache.(Reflector).Run-fm(0x0?)

:1 +0x30 fp=0x400094df80 sp=0x400094df60 pc=0xede780 k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:58 +0x28 fp=0x400094dfa0 sp=0x400094df80 pc=0xc1be98 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x400094dfd0 sp=0x400094dfa0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400094dfd0 sp=0x400094dfd0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 ... goroutine 732 [select]: runtime.gopark(0x400081af48?, 0x2?, 0x0?, 0x0?, 0x400081af04?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x400081ada0 sp=0x400081ad80 pc=0x4afb4 runtime.selectgo(0x400081af48, 0x400081af00, 0x400004ec01?, 0x1, 0x400081af28?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x400081aeb0 sp=0x400081ada0 pc=0x5b0d4 k8s.io/client-go/tools/cache.(*processorListener).pop(0x400128be00) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:876 +0xe0 fp=0x400081af80 sp=0x400081aeb0 pc=0xed9450 k8s.io/client-go/tools/cache.(*processorListener).pop-fm() :1 +0x28 fp=0x400081afa0 sp=0x400081af80 pc=0xedf0b8 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x400081afd0 sp=0x400081afa0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400081afd0 sp=0x400081afd0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 735 [select]: runtime.gopark(0x4000e29f48?, 0x2?, 0x22?, 0x66?, 0x4000e29f04?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000e29da0 sp=0x4000e29d80 pc=0x4afb4 runtime.selectgo(0x4000e29f48, 0x4000e29f00, 0x40013d0fa0?, 0x1, 0x756f6c636570612f?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x4000e29eb0 sp=0x4000e29da0 pc=0x5b0d4 k8s.io/client-go/tools/cache.(*processorListener).pop(0x400128be80) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:876 +0xe0 fp=0x4000e29f80 sp=0x4000e29eb0 pc=0xed9450 k8s.io/client-go/tools/cache.(*processorListener).pop-fm() :1 +0x28 fp=0x4000e29fa0 sp=0x4000e29f80 pc=0xedf0b8 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x4000e29fd0 sp=0x4000e29fa0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000e29fd0 sp=0x4000e29fd0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 739 [chan receive]: runtime.gopark(0x0?, 0x469530?, 0xa0?, 0x2?, 0x0?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000e2ace0 sp=0x4000e2acc0 pc=0x4afb4 runtime.chanrecv(0x40006b6f00, 0x4000e2ae28, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x47c fp=0x4000e2ad70 sp=0x4000e2ace0 pc=0x161bc runtime.chanrecv2(0x0?, 0xc21b28?) /usr/local/go/src/runtime/chan.go:447 +0x14 fp=0x4000e2ada0 sp=0x4000e2ad70 pc=0x15d24 k8s.io/client-go/tools/cache.(*processorListener).run.func1() /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:906 +0x48 fp=0x4000e2ae40 sp=0x4000e2ada0 pc=0xed96d8 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x25a0?) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:157 +0x40 fp=0x4000e2ae60 sp=0x4000e2ae40 pc=0xc1c460 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000e2af28?, {0x1d51800, 0x40014a6c90}, 0x1, 0x40006b7e00) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:158 +0x90 fp=0x4000e2aee0 sp=0x4000e2ae60 pc=0xc1c330 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x1d79c10?, 0x3b9aca00, 0x0, 0xe0?, 0x4000e2af18?) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:135 +0x80 fp=0x4000e2af30 sp=0x4000e2aee0 pc=0xc1c260 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:92 k8s.io/client-go/tools/cache.(*processorListener).run(0x40014ac080?) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:905 +0x68 fp=0x4000e2af80 sp=0x4000e2af30 pc=0xed9668 k8s.io/client-go/tools/cache.(*processorListener).run-fm() :1 +0x28 fp=0x4000e2afa0 sp=0x4000e2af80 pc=0xedf058 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x4000e2afd0 sp=0x4000e2afa0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000e2afd0 sp=0x4000e2afd0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 740 [select]: runtime.gopark(0x4000e2b748?, 0x2?, 0x17?, 0x6b?, 0x4000e2b704?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000e2b5a0 sp=0x4000e2b580 pc=0x4afb4 runtime.selectgo(0x4000e2b748, 0x4000e2b700, 0x30342d313735332d?, 0x1, 0x3436316662393732?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x4000e2b6b0 sp=0x4000e2b5a0 pc=0x5b0d4 k8s.io/client-go/tools/cache.(*processorListener).pop(0x40014ac080) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:876 +0xe0 fp=0x4000e2b780 sp=0x4000e2b6b0 pc=0xed9450 k8s.io/client-go/tools/cache.(*processorListener).pop-fm() :1 +0x28 fp=0x4000e2b7a0 sp=0x4000e2b780 pc=0xedf0b8 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x4000e2b7d0 sp=0x4000e2b7a0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000e2b7d0 sp=0x4000e2b7d0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 743 [chan receive]: runtime.gopark(0x3161623566393824?, 0x342d346136332d61?, 0x34?, 0x34?, 0x613336353530352d?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000e2cce0 sp=0x4000e2ccc0 pc=0x4afb4 runtime.chanrecv(0x40006b7260, 0x4000e2ce28, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x47c fp=0x4000e2cd70 sp=0x4000e2cce0 pc=0x161bc runtime.chanrecv2(0x4000e2cdd8?, 0x25db4?) /usr/local/go/src/runtime/chan.go:447 +0x14 fp=0x4000e2cda0 sp=0x4000e2cd70 pc=0x15d24 k8s.io/client-go/tools/cache.(*processorListener).run.func1() /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:906 +0x48 fp=0x4000e2ce40 sp=0x4000e2cda0 pc=0xed96d8 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x4540?) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:157 +0x40 fp=0x4000e2ce60 sp=0x4000e2ce40 pc=0xc1c460 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4000e2cf28?, {0x1d51800, 0x40014a6cc0}, 0x1, 0x4000914900) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:158 +0x90 fp=0x4000e2cee0 sp=0x4000e2ce60 pc=0xc1c330 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x622d393464383463?, 0x3b9aca00, 0x0, 0x35?, 0x3634343807323234?) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:135 +0x80 fp=0x4000e2cf30 sp=0x4000e2cee0 pc=0xc1c260 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:92 k8s.io/client-go/tools/cache.(*processorListener).run(0x40014ac100?) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:905 +0x68 fp=0x4000e2cf80 sp=0x4000e2cf30 pc=0xed9668 k8s.io/client-go/tools/cache.(*processorListener).run-fm() :1 +0x28 fp=0x4000e2cfa0 sp=0x4000e2cf80 pc=0xedf058 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x4000e2cfd0 sp=0x4000e2cfa0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000e2cfd0 sp=0x4000e2cfd0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 744 [select]: runtime.gopark(0x40014bf748?, 0x2?, 0x73?, 0x77?, 0x40014bf704?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x40014bf5a0 sp=0x40014bf580 pc=0x4afb4 runtime.selectgo(0x40014bf748, 0x40014bf700, 0x424342386b733032?, 0x1, 0x4e33464575555569?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x40014bf6b0 sp=0x40014bf5a0 pc=0x5b0d4 k8s.io/client-go/tools/cache.(*processorListener).pop(0x40014ac100) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:876 +0xe0 fp=0x40014bf780 sp=0x40014bf6b0 pc=0xed9450 k8s.io/client-go/tools/cache.(*processorListener).pop-fm() :1 +0x28 fp=0x40014bf7a0 sp=0x40014bf780 pc=0xedf0b8 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x40014bf7d0 sp=0x40014bf7a0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40014bf7d0 sp=0x40014bf7d0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 747 [select]: runtime.gopark(0x4000e26f48?, 0x2?, 0x3a?, 0x7b?, 0x4000e26f04?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000e26da0 sp=0x4000e26d80 pc=0x4afb4 runtime.selectgo(0x4000e26f48, 0x4000e26f00, 0x40013d12b0?, 0x1, 0x6520637072203a78?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x4000e26eb0 sp=0x4000e26da0 pc=0x5b0d4 k8s.io/client-go/tools/cache.(*processorListener).pop(0x40014ac180) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:876 +0xe0 fp=0x4000e26f80 sp=0x4000e26eb0 pc=0xed9450 k8s.io/client-go/tools/cache.(*processorListener).pop-fm() :1 +0x28 fp=0x4000e26fa0 sp=0x4000e26f80 pc=0xedf0b8 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x4000e26fd0 sp=0x4000e26fa0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000e26fd0 sp=0x4000e26fd0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 750 [sync.Cond.Wait]: runtime.gopark(0xa108083200123961?, 0x83a001006a185e8?, 0x8?, 0xa1?, 0x6d726f4e064a0140?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000e28590 sp=0x4000e28570 pc=0x4afb4 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:369 sync.runtime_notifyListWait(0x4000700710, 0x2) /usr/local/go/src/runtime/sema.go:517 +0x160 fp=0x4000e285e0 sp=0x4000e28590 pc=0x75290 sync.(*Cond).Wait(0x4000700700) /usr/local/go/src/sync/cond.go:70 +0xd0 fp=0x4000e28620 sp=0x4000e285e0 pc=0x80600 k8s.io/client-go/util/workqueue.(*Type).Get(0x40001195c0) /go/pkg/mod/k8s.io/client-go@v0.26.1/util/workqueue/queue.go:157 +0x90 fp=0x4000e28690 sp=0x4000e28620 pc=0xc219c0 k8s.io/client-go/util/workqueue.(*delayingType).Get(0x222c7d7b3a22706d?) :1 +0x30 fp=0x4000e286b0 sp=0x4000e28690 pc=0xc23970 k8s.io/client-go/util/workqueue.(*rateLimitingType).Get(0x733a66222c7d7b3a?) :1 +0x30 fp=0x4000e286d0 sp=0x4000e286b0 pc=0xc24ba0 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0x40007297c0, {0x1d6d510, 0x4000592f40}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:257 +0x4c fp=0x4000e28770 sp=0x4000e286d0 pc=0xf96f5c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235 +0x74 fp=0x4000e287d0 sp=0x4000e28770 pc=0xf96be4 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000e287d0 sp=0x4000e287d0 pc=0x78f94 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:231 +0x294 goroutine 751 [sync.Cond.Wait]: runtime.gopark(0x2d65333935376161?, 0x3062342d31393336?, 0x31?, 0x2d?, 0x3962383638323032?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000e28d90 sp=0x4000e28d70 pc=0x4afb4 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:369 sync.runtime_notifyListWait(0x4000700710, 0x3) /usr/local/go/src/runtime/sema.go:517 +0x160 fp=0x4000e28de0 sp=0x4000e28d90 pc=0x75290 sync.(*Cond).Wait(0x4000700700) /usr/local/go/src/sync/cond.go:70 +0xd0 fp=0x4000e28e20 sp=0x4000e28de0 pc=0x80600 k8s.io/client-go/util/workqueue.(*Type).Get(0x40001195c0) /go/pkg/mod/k8s.io/client-go@v0.26.1/util/workqueue/queue.go:157 +0x90 fp=0x4000e28e90 sp=0x4000e28e20 pc=0xc219c0 k8s.io/client-go/util/workqueue.(*delayingType).Get(0x6562756b2e663731?) :1 +0x30 fp=0x4000e28eb0 sp=0x4000e28e90 pc=0xc23970 k8s.io/client-go/util/workqueue.(*rateLimitingType).Get(0x31382d613438342d?) :1 +0x30 fp=0x4000e28ed0 sp=0x4000e28eb0 pc=0xc24ba0 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0x40007297c0, {0x1d6d510, 0x4000592f40}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:257 +0x4c fp=0x4000e28f70 sp=0x4000e28ed0 pc=0xf96f5c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235 +0x74 fp=0x4000e28fd0 sp=0x4000e28f70 pc=0xf96be4 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000e28fd0 sp=0x4000e28fd0 pc=0x78f94 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:231 +0x294 goroutine 752 [sync.Cond.Wait]: runtime.gopark(0x72384e4b4d6e7733?, 0x38596a5872654f72?, 0x6c?, 0x6f?, 0x374c584e4e32665a?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x40014bb590 sp=0x40014bb570 pc=0x4afb4 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:369 sync.runtime_notifyListWait(0x4000700710, 0x0) /usr/local/go/src/runtime/sema.go:517 +0x160 fp=0x40014bb5e0 sp=0x40014bb590 pc=0x75290 sync.(*Cond).Wait(0x4000700700) /usr/local/go/src/sync/cond.go:70 +0xd0 fp=0x40014bb620 sp=0x40014bb5e0 pc=0x80600 k8s.io/client-go/util/workqueue.(*Type).Get(0x40001195c0) /go/pkg/mod/k8s.io/client-go@v0.26.1/util/workqueue/queue.go:157 +0x90 fp=0x40014bb690 sp=0x40014bb620 pc=0xc219c0 k8s.io/client-go/util/workqueue.(*delayingType).Get(0x3062477a78706146?) :1 +0x30 fp=0x40014bb6b0 sp=0x40014bb690 pc=0xc23970 k8s.io/client-go/util/workqueue.(*rateLimitingType).Get(0x61586239524f4c55?) :1 +0x30 fp=0x40014bb6d0 sp=0x40014bb6b0 pc=0xc24ba0 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0x40007297c0, {0x1d6d510, 0x4000592f40}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:257 +0x4c fp=0x40014bb770 sp=0x40014bb6d0 pc=0xf96f5c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235 +0x74 fp=0x40014bb7d0 sp=0x40014bb770 pc=0xf96be4 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40014bb7d0 sp=0x40014bb7d0 pc=0x78f94 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:231 +0x294 goroutine 753 [chan receive]: runtime.gopark(0x4000a709c0?, 0x4001094480?, 0x8?, 0x8d?, 0xf94e0c?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x4000f68ce0 sp=0x4000f68cc0 pc=0x4afb4 runtime.chanrecv(0x40006b74a0, 0x4000f68e28, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x47c fp=0x4000f68d70 sp=0x4000f68ce0 pc=0x161bc runtime.chanrecv2(0x2b4b4d704f623274?, 0x19e6a40?) /usr/local/go/src/runtime/chan.go:447 +0x14 fp=0x4000f68da0 sp=0x4000f68d70 pc=0x15d24 k8s.io/client-go/tools/cache.(*processorListener).run.func1() /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:906 +0x48 fp=0x4000f68e40 sp=0x4000f68da0 pc=0xed96d8 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x4778000000003749?) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:157 +0x40 fp=0x4000f68e60 sp=0x4000f68e40 pc=0xc1c460 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x40014bbf28?, {0x1d51800, 0x40014a6d20}, 0x1, 0x40009149c0) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:158 +0x90 fp=0x4000f68ee0 sp=0x4000f68e60 pc=0xc1c330 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x756966306e695a61?, 0x3b9aca00, 0x0, 0x51?, 0x6f35697739313435?) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:135 +0x80 fp=0x4000f68f30 sp=0x4000f68ee0 pc=0xc1c260 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:92 k8s.io/client-go/tools/cache.(*processorListener).run(0x40014ac200?) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:905 +0x68 fp=0x4000f68f80 sp=0x4000f68f30 pc=0xed9668 k8s.io/client-go/tools/cache.(*processorListener).run-fm() :1 +0x28 fp=0x4000f68fa0 sp=0x4000f68f80 pc=0xedf058 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x4000f68fd0 sp=0x4000f68fa0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000f68fd0 sp=0x4000f68fd0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 goroutine 754 [select]: runtime.gopark(0x40014bc748?, 0x2?, 0x35?, 0x77?, 0x40014bc704?) /usr/local/go/src/runtime/proc.go:363 +0xe4 fp=0x40014bc5a0 sp=0x40014bc580 pc=0x4afb4 runtime.selectgo(0x40014bc748, 0x40014bc700, 0x40013d1510?, 0x1, 0x41433936672b3276?, 0x1) /usr/local/go/src/runtime/select.go:328 +0x6b4 fp=0x40014bc6b0 sp=0x40014bc5a0 pc=0x5b0d4 k8s.io/client-go/tools/cache.(*processorListener).pop(0x40014ac200) /go/pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/shared_informer.go:876 +0xe0 fp=0x40014bc780 sp=0x40014bc6b0 pc=0xed9450 k8s.io/client-go/tools/cache.(*processorListener).pop-fm() :1 +0x28 fp=0x40014bc7a0 sp=0x40014bc780 pc=0xedf0b8 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:75 +0x58 fp=0x40014bc7d0 sp=0x40014bc7a0 pc=0xc1c0e8 runtime.goexit() /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40014bc7d0 sp=0x40014bc7d0 pc=0x78f94 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:73 +0x84 ``` **Expected behavior** kubeblocks install success. **Screenshots** If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - OS: [e.g. iOS] - Browser [e.g. chrome, safari] - Version [e.g. 22] **Additional context** Add any other context about the problem here.
nashtsai commented 1 year ago

duplicated #2238