search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.5k stars 4.89k forks source link

Documentation errors prevent successful execution #19635

Closed PunkFleet closed 1 month ago

PunkFleet commented 2 months ago

What Happened?

I followed the official documentation to configure HA minikube: https://minikube.sigs.k8s.io/docs/tutorials/multi_control_plane_ha_clusters/

but I can't execute this command it , It has to enter a password here, and I've tried the host's password, tcuser, and it won't go through.

minikube ssh -p ha-demo -- 'sudo /var/lib/minikube/binaries/v1.28.4/kubectl --kubeconfig=/var/lib/minikube/kubeconfig logs -n kube-system pod/kube-vip-ha-demo'
Screenshot 2024-09-13 at 12 03 18 PM

Attach the log file

==> kindnet [9fce7c424401] <==
I0913 04:07:35.771272       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:07:35.771360       1 main.go:299] handling current node
I0913 04:07:35.771398       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:07:35.771407       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:07:35.771714       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:07:35.771731       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:07:45.771668       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:07:45.771833       1 main.go:299] handling current node
I0913 04:07:45.771859       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:07:45.771866       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:07:45.771999       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:07:45.772022       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:07:55.770894       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:07:55.771001       1 main.go:299] handling current node
I0913 04:07:55.771048       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:07:55.771076       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:07:55.771545       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:07:55.771580       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:08:05.774845       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:08:05.774915       1 main.go:299] handling current node
I0913 04:08:05.774953       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:08:05.774963       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:08:05.775321       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:08:05.775337       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:08:15.770900       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:08:15.770947       1 main.go:299] handling current node
I0913 04:08:15.770970       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:08:15.770977       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:08:15.771121       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:08:15.771145       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:08:25.771247       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:08:25.771325       1 main.go:299] handling current node
I0913 04:08:25.771369       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:08:25.771380       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:08:25.771810       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:08:25.771828       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:08:35.773609       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:08:35.773669       1 main.go:299] handling current node
I0913 04:08:35.773742       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:08:35.773751       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:08:35.773904       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:08:35.773915       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:08:45.772091       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:08:45.772156       1 main.go:299] handling current node
I0913 04:08:45.772181       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:08:45.772189       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:08:45.772327       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:08:45.772349       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:08:55.770790       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:08:55.770840       1 main.go:299] handling current node
I0913 04:08:55.770864       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:08:55.770871       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:08:55.771036       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:08:55.771057       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 
I0913 04:09:05.779934       1 main.go:295] Handling node with IPs: map[10.0.0.2:{}]
I0913 04:09:05.779988       1 main.go:299] handling current node
I0913 04:09:05.780012       1 main.go:295] Handling node with IPs: map[10.0.0.3:{}]
I0913 04:09:05.780020       1 main.go:322] Node minikube-m02 has CIDR [10.244.1.0/24] 
I0913 04:09:05.780191       1 main.go:295] Handling node with IPs: map[10.0.0.4:{}]
I0913 04:09:05.780209       1 main.go:322] Node minikube-m03 has CIDR [10.244.2.0/24] 

==> kube-apiserver [b7406b45bca4] <==
I0913 02:59:46.612727       1 shared_informer.go:313] Waiting for caches to sync for cluster_authentication_trust_controller
I0913 02:59:46.612758       1 remote_available_controller.go:411] Starting RemoteAvailability controller
I0913 02:59:46.612767       1 cache.go:32] Waiting for caches to sync for RemoteAvailability controller
I0913 02:59:46.612782       1 dynamic_serving_content.go:135] "Starting controller" name="aggregator-proxy-cert::/var/lib/minikube/certs/front-proxy-client.crt::/var/lib/minikube/certs/front-proxy-client.key"
I0913 02:59:46.612888       1 customresource_discovery_controller.go:292] Starting DiscoveryController
I0913 02:59:46.612983       1 apf_controller.go:377] Starting API Priority and Fairness config controller
I0913 02:59:46.612998       1 dynamic_cafile_content.go:160] "Starting controller" name="client-ca-bundle::/var/lib/minikube/certs/ca.crt"
I0913 02:59:46.613029       1 apiservice_controller.go:100] Starting APIServiceRegistrationController
I0913 02:59:46.613033       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0913 02:59:46.613033       1 dynamic_cafile_content.go:160] "Starting controller" name="request-header::/var/lib/minikube/certs/front-proxy-ca.crt"
I0913 02:59:46.613043       1 controller.go:78] Starting OpenAPI AggregationController
I0913 02:59:46.613079       1 system_namespaces_controller.go:66] Starting system namespaces controller
I0913 02:59:46.613238       1 controller.go:142] Starting OpenAPI controller
I0913 02:59:46.613255       1 controller.go:90] Starting OpenAPI V3 controller
I0913 02:59:46.613277       1 naming_controller.go:294] Starting NamingConditionController
I0913 02:59:46.613283       1 establishing_controller.go:81] Starting EstablishingController
I0913 02:59:46.613351       1 nonstructuralschema_controller.go:195] Starting NonStructuralSchemaConditionController
I0913 02:59:46.613388       1 apiapproval_controller.go:189] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0913 02:59:46.613415       1 crd_finalizer.go:269] Starting CRDFinalizer
I0913 02:59:46.613889       1 gc_controller.go:78] Starting apiserver lease garbage collector
I0913 02:59:46.614043       1 controller.go:119] Starting legacy_token_tracking_controller
I0913 02:59:46.614302       1 shared_informer.go:313] Waiting for caches to sync for configmaps
I0913 02:59:46.614321       1 local_available_controller.go:156] Starting LocalAvailability controller
I0913 02:59:46.614328       1 cache.go:32] Waiting for caches to sync for LocalAvailability controller
I0913 02:59:46.614403       1 crdregistration_controller.go:114] Starting crd-autoregister controller
I0913 02:59:46.614413       1 shared_informer.go:313] Waiting for caches to sync for crd-autoregister
I0913 02:59:46.637452       1 shared_informer.go:320] Caches are synced for node_authorizer
I0913 02:59:46.640029       1 shared_informer.go:320] Caches are synced for *generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]
I0913 02:59:46.640046       1 policy_source.go:224] refreshing policies
E0913 02:59:46.676732       1 controller.go:145] "Failed to ensure lease exists, will retry" err="namespaces \"kube-system\" not found" interval="200ms"
I0913 02:59:46.712868       1 shared_informer.go:320] Caches are synced for cluster_authentication_trust_controller
I0913 02:59:46.712890       1 cache.go:39] Caches are synced for RemoteAvailability controller
I0913 02:59:46.713080       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0913 02:59:46.713107       1 apf_controller.go:382] Running API Priority and Fairness config worker
I0913 02:59:46.713114       1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process
I0913 02:59:46.713165       1 handler_discovery.go:450] Starting ResourceDiscoveryManager
I0913 02:59:46.713720       1 controller.go:615] quota admission added evaluator for: namespaces
I0913 02:59:46.714430       1 shared_informer.go:320] Caches are synced for crd-autoregister
I0913 02:59:46.714448       1 shared_informer.go:320] Caches are synced for configmaps
I0913 02:59:46.714454       1 aggregator.go:171] initial CRD sync complete...
I0913 02:59:46.714459       1 autoregister_controller.go:144] Starting autoregister controller
I0913 02:59:46.714461       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0913 02:59:46.714464       1 cache.go:39] Caches are synced for autoregister controller
I0913 02:59:46.714450       1 cache.go:39] Caches are synced for LocalAvailability controller
I0913 02:59:46.879139       1 controller.go:615] quota admission added evaluator for: leases.coordination.k8s.io
I0913 02:59:47.620085       1 storage_scheduling.go:95] created PriorityClass system-node-critical with value 2000001000
I0913 02:59:47.623963       1 storage_scheduling.go:95] created PriorityClass system-cluster-critical with value 2000000000
I0913 02:59:47.623985       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
I0913 02:59:47.773966       1 controller.go:615] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0913 02:59:47.784599       1 controller.go:615] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0913 02:59:47.816874       1 alloc.go:330] "allocated clusterIPs" service="default/kubernetes" clusterIPs={"IPv4":"10.96.0.1"}
W0913 02:59:47.818811       1 lease.go:265] Resetting endpoints for master service "kubernetes" to [10.0.0.2]
I0913 02:59:47.819154       1 controller.go:615] quota admission added evaluator for: endpoints
I0913 02:59:47.820539       1 controller.go:615] quota admission added evaluator for: endpointslices.discovery.k8s.io
I0913 02:59:48.640105       1 controller.go:615] quota admission added evaluator for: serviceaccounts
I0913 03:00:01.169100       1 controller.go:615] quota admission added evaluator for: deployments.apps
I0913 03:00:01.181147       1 alloc.go:330] "allocated clusterIPs" service="kube-system/kube-dns" clusterIPs={"IPv4":"10.96.0.10"}
I0913 03:00:01.182912       1 controller.go:615] quota admission added evaluator for: replicasets.apps
I0913 03:00:01.187421       1 controller.go:615] quota admission added evaluator for: daemonsets.apps
I0913 03:00:01.192913       1 controller.go:615] quota admission added evaluator for: controllerrevisions.apps

==> kube-controller-manager [a9a0e786a509] <==
E0913 03:00:27.414971       1 range_allocator.go:246] "Unhandled Error" err="error syncing 'minikube-m03': failed to patch node CIDR: Node \"minikube-m03\" is invalid: [spec.podCIDRs: Invalid value: []string{\"10.244.3.0/24\", \"10.244.2.0/24\"}: may specify no more than one CIDR for each IP family, spec.podCIDRs: Forbidden: node updates may not change podCIDR except from \"\" to valid], requeuing" logger="UnhandledError"
I0913 03:00:27.415026       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:27.420031       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:27.893035       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:28.398617       1 node_lifecycle_controller.go:884] "Missing timestamp for Node. Assuming now as a timestamp" logger="node-lifecycle-controller" node="minikube-m03"
I0913 03:00:28.408029       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:28.548711       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:28.701904       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:28.724053       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:28.729372       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:28.753687       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:29.136139       1 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-6f6b679f8f" duration="43.631µs"
I0913 03:00:29.148518       1 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-6f6b679f8f" duration="5.56834ms"
I0913 03:00:29.148613       1 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-6f6b679f8f" duration="78.969µs"
I0913 03:00:29.374365       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:00:30.154466       1 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-6f6b679f8f" duration="38.588µs"
I0913 03:00:30.168661       1 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-6f6b679f8f" duration="4.651091ms"
I0913 03:00:30.169415       1 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-6f6b679f8f" duration="37.922µs"
I0913 03:00:31.560107       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:00:44.537764       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:01:58.742412       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:05:36.480166       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:05:50.741178       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:07:06.444439       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:10:42.550306       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:10:56.442309       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:12:12.124497       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:15:48.582602       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:16:02.787013       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:17:18.817923       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:20:54.099395       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:21:09.321009       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:22:25.465759       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:26:00.808279       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:26:15.889954       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:27:31.922111       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:31:07.164158       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:31:22.107465       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:32:38.236636       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:36:12.149213       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:36:27.634942       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:37:43.849545       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:41:18.708360       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:41:34.312655       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:42:49.244089       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:46:24.243159       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:46:39.827929       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:47:55.946749       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:51:30.622160       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:51:46.328330       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:53:02.386719       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 03:56:36.881599       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 03:56:51.901122       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 03:58:08.135074       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 04:01:43.583051       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 04:01:58.478123       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 04:03:15.112523       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"
I0913 04:06:50.470061       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube"
I0913 04:07:04.267240       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m02"
I0913 04:08:21.341158       1 range_allocator.go:241] "Successfully synced" logger="node-ipam-controller" key="minikube-m03"

==> kube-proxy [e939302a68a8] <==
I0913 03:00:02.495239       1 server_linux.go:66] "Using iptables proxy"
I0913 03:00:02.549985       1 server.go:677] "Successfully retrieved node IP(s)" IPs=["10.0.0.2"]
E0913 03:00:02.550019       1 server.go:234] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using `--nodeport-addresses primary`"
I0913 03:00:02.557264       1 server.go:243] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
I0913 03:00:02.557287       1 server_linux.go:169] "Using iptables Proxier"
I0913 03:00:02.558055       1 proxier.go:255] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses" ipFamily="IPv4"
I0913 03:00:02.558229       1 server.go:483] "Version info" version="v1.31.0"
I0913 03:00:02.558242       1 server.go:485] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0913 03:00:02.559160       1 config.go:104] "Starting endpoint slice config controller"
I0913 03:00:02.559192       1 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
I0913 03:00:02.559211       1 config.go:197] "Starting service config controller"
I0913 03:00:02.559215       1 shared_informer.go:313] Waiting for caches to sync for service config
I0913 03:00:02.559505       1 config.go:326] "Starting node config controller"
I0913 03:00:02.559509       1 shared_informer.go:313] Waiting for caches to sync for node config
I0913 03:00:02.659364       1 shared_informer.go:320] Caches are synced for service config
I0913 03:00:02.659399       1 shared_informer.go:320] Caches are synced for endpoint slice config
I0913 03:00:02.659549       1 shared_informer.go:320] Caches are synced for node config

==> kube-scheduler [83bc34aa1d57] <==
I0913 02:59:46.627324       1 server.go:169] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0913 02:59:46.628334       1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0913 02:59:46.628356       1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0913 02:59:46.628389       1 secure_serving.go:213] Serving securely on 127.0.0.1:10259
I0913 02:59:46.628410       1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
W0913 02:59:46.629894       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
W0913 02:59:46.629942       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0913 02:59:46.629946       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:kube-scheduler\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.629910       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0913 02:59:46.629972       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User \"system:kube-scheduler\" cannot list resource \"persistentvolumes\" in API group \"\" at the cluster scope" logger="UnhandledError"
E0913 02:59:46.629971       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User \"system:kube-scheduler\" cannot list resource \"persistentvolumeclaims\" in API group \"\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.629991       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
W0913 02:59:46.630000       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0913 02:59:46.630006       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:kube-scheduler\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError"
E0913 02:59:46.630009       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User \"system:kube-scheduler\" cannot list resource \"replicationcontrollers\" in API group \"\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.630038       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
W0913 02:59:46.630052       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0913 02:59:46.630055       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:kube-scheduler\" cannot list resource \"pods\" in API group \"\" at the cluster scope" logger="UnhandledError"
E0913 02:59:46.630059       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User \"system:kube-scheduler\" cannot list resource \"statefulsets\" in API group \"apps\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.630096       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0913 02:59:46.630102       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User \"system:kube-scheduler\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.630113       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
W0913 02:59:46.630115       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Namespace: namespaces is forbidden: User "system:kube-scheduler" cannot list resource "namespaces" in API group "" at the cluster scope
E0913 02:59:46.630119       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User \"system:kube-scheduler\" cannot list resource \"poddisruptionbudgets\" in API group \"policy\" at the cluster scope" logger="UnhandledError"
E0913 02:59:46.630122       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User \"system:kube-scheduler\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.630259       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.CSIStorageCapacity: csistoragecapacities.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csistoragecapacities" in API group "storage.k8s.io" at the cluster scope
W0913 02:59:46.630270       1 reflector.go:561] runtime/asm_arm64.s:1222: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0913 02:59:46.630278       1 reflector.go:158] "Unhandled Error" err="runtime/asm_arm64.s:1222: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps \"extension-apiserver-authentication\" is forbidden: User \"system:kube-scheduler\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"" logger="UnhandledError"
E0913 02:59:46.630284       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSIStorageCapacity: failed to list *v1.CSIStorageCapacity: csistoragecapacities.storage.k8s.io is forbidden: User \"system:kube-scheduler\" cannot list resource \"csistoragecapacities\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.630293       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0913 02:59:46.630298       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User \"system:kube-scheduler\" cannot list resource \"csinodes\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.630614       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0913 02:59:46.630627       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User \"system:kube-scheduler\" cannot list resource \"storageclasses\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError"
W0913 02:59:46.630673       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0913 02:59:46.630693       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:kube-scheduler\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope" logger="UnhandledError"
W0913 02:59:47.457106       1 reflector.go:561] runtime/asm_arm64.s:1222: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0913 02:59:47.457192       1 reflector.go:158] "Unhandled Error" err="runtime/asm_arm64.s:1222: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps \"extension-apiserver-authentication\" is forbidden: User \"system:kube-scheduler\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"" logger="UnhandledError"
W0913 02:59:47.529629       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0913 02:59:47.529783       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User \"system:kube-scheduler\" cannot list resource \"persistentvolumeclaims\" in API group \"\" at the cluster scope" logger="UnhandledError"
W0913 02:59:47.548724       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0913 02:59:47.548795       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User \"system:kube-scheduler\" cannot list resource \"storageclasses\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError"
W0913 02:59:47.568815       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0913 02:59:47.568869       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:kube-scheduler\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError"
W0913 02:59:47.608813       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0913 02:59:47.608869       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User \"system:kube-scheduler\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError"
W0913 02:59:47.710882       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0913 02:59:47.710906       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User \"system:kube-scheduler\" cannot list resource \"poddisruptionbudgets\" in API group \"policy\" at the cluster scope" logger="UnhandledError"
I0913 02:59:49.729393       1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E0913 03:00:27.413507       1 framework.go:1305] "Plugin Failed" err="Operation cannot be fulfilled on pods/binding \"kube-proxy-4k9dt\": pod kube-proxy-4k9dt is already assigned to node \"minikube-m03\"" plugin="DefaultBinder" pod="kube-system/kube-proxy-4k9dt" node="minikube-m03"
E0913 03:00:27.413644       1 schedule_one.go:348] "scheduler cache ForgetPod failed" err="pod 38a8dc15-f2cf-4598-9c15-08abc4c9534d(kube-system/kube-proxy-4k9dt) wasn't assumed so cannot be forgotten" pod="kube-system/kube-proxy-4k9dt"
E0913 03:00:27.413706       1 schedule_one.go:1057] "Error scheduling pod; retrying" err="running Bind plugin \"DefaultBinder\": Operation cannot be fulfilled on pods/binding \"kube-proxy-4k9dt\": pod kube-proxy-4k9dt is already assigned to node \"minikube-m03\"" pod="kube-system/kube-proxy-4k9dt"
I0913 03:00:27.413735       1 schedule_one.go:1070] "Pod has been assigned to node. Abort adding it back to queue." pod="kube-system/kube-proxy-4k9dt" node="minikube-m03"
E0913 03:00:27.414297       1 framework.go:1305] "Plugin Failed" err="Operation cannot be fulfilled on pods/binding \"kindnet-vfqjf\": pod kindnet-vfqjf is already assigned to node \"minikube-m03\"" plugin="DefaultBinder" pod="kube-system/kindnet-vfqjf" node="minikube-m03"
E0913 03:00:27.414348       1 schedule_one.go:348] "scheduler cache ForgetPod failed" err="pod 5d6e976e-cd1a-4248-b606-c2d86a17422f(kube-system/kindnet-vfqjf) wasn't assumed so cannot be forgotten" pod="kube-system/kindnet-vfqjf"
E0913 03:00:27.414543       1 schedule_one.go:1057] "Error scheduling pod; retrying" err="running Bind plugin \"DefaultBinder\": Operation cannot be fulfilled on pods/binding \"kindnet-vfqjf\": pod kindnet-vfqjf is already assigned to node \"minikube-m03\"" pod="kube-system/kindnet-vfqjf"
I0913 03:00:27.414685       1 schedule_one.go:1070] "Pod has been assigned to node. Abort adding it back to queue." pod="kube-system/kindnet-vfqjf" node="minikube-m03"
E0913 03:00:27.425369       1 framework.go:1305] "Plugin Failed" err="Operation cannot be fulfilled on pods/binding \"kindnet-pdxhq\": pod kindnet-pdxhq is already assigned to node \"minikube-m03\"" plugin="DefaultBinder" pod="kube-system/kindnet-pdxhq" node="minikube-m03"
E0913 03:00:27.425418       1 schedule_one.go:348] "scheduler cache ForgetPod failed" err="pod 7da90768-e0fd-4842-b755-f44148d3679d(kube-system/kindnet-pdxhq) wasn't assumed so cannot be forgotten" pod="kube-system/kindnet-pdxhq"
E0913 03:00:27.425456       1 schedule_one.go:1057] "Error scheduling pod; retrying" err="running Bind plugin \"DefaultBinder\": Operation cannot be fulfilled on pods/binding \"kindnet-pdxhq\": pod kindnet-pdxhq is already assigned to node \"minikube-m03\"" pod="kube-system/kindnet-pdxhq"
I0913 03:00:27.425466       1 schedule_one.go:1070] "Pod has been assigned to node. Abort adding it back to queue." pod="kube-system/kindnet-pdxhq" node="minikube-m03"

==> kubelet <==
Sep 13 03:00:00 minikube kubelet[2352]: I0913 03:00:00.922170    2352 manager.go:510] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Sep 13 03:00:00 minikube kubelet[2352]: I0913 03:00:00.922256    2352 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 13 03:00:00 minikube kubelet[2352]: I0913 03:00:00.922266    2352 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 13 03:00:00 minikube kubelet[2352]: I0913 03:00:00.922345    2352 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 13 03:00:01 minikube kubelet[2352]: E0913 03:00:01.020926    2352 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"kube-vip-minikube\" already exists" pod="kube-system/kube-vip-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: E0913 03:00:01.021378    2352 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"kube-apiserver-minikube\" already exists" pod="kube-system/kube-apiserver-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.024014    2352 kubelet_node_status.go:72] "Attempting to register node" node="minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.026865    2352 kubelet_node_status.go:111] "Node was previously registered" node="minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.026907    2352 kubelet_node_status.go:75] "Successfully registered node" node="minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.026920    2352 kuberuntime_manager.go:1633] "Updating runtime config through cri with podcidr" CIDR="10.244.0.0/24"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.027226    2352 kubelet_network.go:61] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.244.0.0/24"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.109669    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/host-path/6b697998bcaaa1805aa573d7faea4c14-ca-certs\") pod \"kube-apiserver-minikube\" (UID: \"6b697998bcaaa1805aa573d7faea4c14\") " pod="kube-system/kube-apiserver-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210869    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/40f5f661ab65f2e4bfe41ac2993c01de-usr-share-ca-certificates\") pod \"kube-controller-manager-minikube\" (UID: \"40f5f661ab65f2e4bfe41ac2993c01de\") " pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210912    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"k8s-certs\" (UniqueName: \"kubernetes.io/host-path/6b697998bcaaa1805aa573d7faea4c14-k8s-certs\") pod \"kube-apiserver-minikube\" (UID: \"6b697998bcaaa1805aa573d7faea4c14\") " pod="kube-system/kube-apiserver-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210932    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/6b697998bcaaa1805aa573d7faea4c14-usr-share-ca-certificates\") pod \"kube-apiserver-minikube\" (UID: \"6b697998bcaaa1805aa573d7faea4c14\") " pod="kube-system/kube-apiserver-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210940    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/40f5f661ab65f2e4bfe41ac2993c01de-etc-ca-certificates\") pod \"kube-controller-manager-minikube\" (UID: \"40f5f661ab65f2e4bfe41ac2993c01de\") " pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210946    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeconfig\" (UniqueName: \"kubernetes.io/host-path/40f5f661ab65f2e4bfe41ac2993c01de-kubeconfig\") pod \"kube-controller-manager-minikube\" (UID: \"40f5f661ab65f2e4bfe41ac2993c01de\") " pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210953    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-certs\" (UniqueName: \"kubernetes.io/host-path/2e9fd2533d444490ebe6c310253b855f-etcd-certs\") pod \"etcd-minikube\" (UID: \"2e9fd2533d444490ebe6c310253b855f\") " pod="kube-system/etcd-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210961    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/6b697998bcaaa1805aa573d7faea4c14-usr-local-share-ca-certificates\") pod \"kube-apiserver-minikube\" (UID: \"6b697998bcaaa1805aa573d7faea4c14\") " pod="kube-system/kube-apiserver-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210968    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/host-path/40f5f661ab65f2e4bfe41ac2993c01de-ca-certs\") pod \"kube-controller-manager-minikube\" (UID: \"40f5f661ab65f2e4bfe41ac2993c01de\") " pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210973    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"k8s-certs\" (UniqueName: \"kubernetes.io/host-path/40f5f661ab65f2e4bfe41ac2993c01de-k8s-certs\") pod \"kube-controller-manager-minikube\" (UID: \"40f5f661ab65f2e4bfe41ac2993c01de\") " pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210981    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/40f5f661ab65f2e4bfe41ac2993c01de-usr-local-share-ca-certificates\") pod \"kube-controller-manager-minikube\" (UID: \"40f5f661ab65f2e4bfe41ac2993c01de\") " pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210988    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flexvolume-dir\" (UniqueName: \"kubernetes.io/host-path/40f5f661ab65f2e4bfe41ac2993c01de-flexvolume-dir\") pod \"kube-controller-manager-minikube\" (UID: \"40f5f661ab65f2e4bfe41ac2993c01de\") " pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.210996    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeconfig\" (UniqueName: \"kubernetes.io/host-path/e039200acb850c82bb901653cc38ff6e-kubeconfig\") pod \"kube-scheduler-minikube\" (UID: \"e039200acb850c82bb901653cc38ff6e\") " pod="kube-system/kube-scheduler-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.211003    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeconfig\" (UniqueName: \"kubernetes.io/host-path/4ea92adf83a60f71813050688102e3c1-kubeconfig\") pod \"kube-vip-minikube\" (UID: \"4ea92adf83a60f71813050688102e3c1\") " pod="kube-system/kube-vip-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.211011    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-data\" (UniqueName: \"kubernetes.io/host-path/2e9fd2533d444490ebe6c310253b855f-etcd-data\") pod \"etcd-minikube\" (UID: \"2e9fd2533d444490ebe6c310253b855f\") " pod="kube-system/etcd-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.211049    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/6b697998bcaaa1805aa573d7faea4c14-etc-ca-certificates\") pod \"kube-apiserver-minikube\" (UID: \"6b697998bcaaa1805aa573d7faea4c14\") " pod="kube-system/kube-apiserver-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.901533    2352 apiserver.go:52] "Watching apiserver"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.907695    2352 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.908930    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/etcd-minikube" podStartSLOduration=0.908907584 podStartE2EDuration="908.907584ms" podCreationTimestamp="2024-09-13 03:00:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-13 03:00:01.908892999 +0000 UTC m=+1.038956789" watchObservedRunningTime="2024-09-13 03:00:01.908907584 +0000 UTC m=+1.038971374"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917454    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37a61afd-14a0-4ad2-a330-053f461103f2-config-volume\") pod \"coredns-6f6b679f8f-qcn76\" (UID: \"37a61afd-14a0-4ad2-a330-053f461103f2\") " pod="kube-system/coredns-6f6b679f8f-qcn76"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917546    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/857d8a89-0a6d-40ee-95cf-c3a668651866-xtables-lock\") pod \"kube-proxy-z8vss\" (UID: \"857d8a89-0a6d-40ee-95cf-c3a668651866\") " pod="kube-system/kube-proxy-z8vss"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917558    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbn6l\" (UniqueName: \"kubernetes.io/projected/857d8a89-0a6d-40ee-95cf-c3a668651866-kube-api-access-rbn6l\") pod \"kube-proxy-z8vss\" (UID: \"857d8a89-0a6d-40ee-95cf-c3a668651866\") " pod="kube-system/kube-proxy-z8vss"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917571    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08c60c8f-1497-4883-b914-7b3019ffaec8-config-volume\") pod \"coredns-6f6b679f8f-rzg2p\" (UID: \"08c60c8f-1497-4883-b914-7b3019ffaec8\") " pod="kube-system/coredns-6f6b679f8f-rzg2p"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917578    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-proxy\" (UniqueName: \"kubernetes.io/configmap/857d8a89-0a6d-40ee-95cf-c3a668651866-kube-proxy\") pod \"kube-proxy-z8vss\" (UID: \"857d8a89-0a6d-40ee-95cf-c3a668651866\") " pod="kube-system/kube-proxy-z8vss"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917716    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9f9j\" (UniqueName: \"kubernetes.io/projected/37a61afd-14a0-4ad2-a330-053f461103f2-kube-api-access-x9f9j\") pod \"coredns-6f6b679f8f-qcn76\" (UID: \"37a61afd-14a0-4ad2-a330-053f461103f2\") " pod="kube-system/coredns-6f6b679f8f-qcn76"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917725    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfthn\" (UniqueName: \"kubernetes.io/projected/08c60c8f-1497-4883-b914-7b3019ffaec8-kube-api-access-lfthn\") pod \"coredns-6f6b679f8f-rzg2p\" (UID: \"08c60c8f-1497-4883-b914-7b3019ffaec8\") " pod="kube-system/coredns-6f6b679f8f-rzg2p"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.917733    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/857d8a89-0a6d-40ee-95cf-c3a668651866-lib-modules\") pod \"kube-proxy-z8vss\" (UID: \"857d8a89-0a6d-40ee-95cf-c3a668651866\") " pod="kube-system/kube-proxy-z8vss"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.920408    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-controller-manager-minikube" podStartSLOduration=0.920398772 podStartE2EDuration="920.398772ms" podCreationTimestamp="2024-09-13 03:00:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-13 03:00:01.920229749 +0000 UTC m=+1.050293539" watchObservedRunningTime="2024-09-13 03:00:01.920398772 +0000 UTC m=+1.050462562"
Sep 13 03:00:01 minikube kubelet[2352]: I0913 03:00:01.934968    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-scheduler-minikube" podStartSLOduration=0.934958335 podStartE2EDuration="934.958335ms" podCreationTimestamp="2024-09-13 03:00:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-13 03:00:01.929312109 +0000 UTC m=+1.059375857" watchObservedRunningTime="2024-09-13 03:00:01.934958335 +0000 UTC m=+1.065022125"
Sep 13 03:00:01 minikube kubelet[2352]: E0913 03:00:01.936249    2352 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"kube-apiserver-minikube\" already exists" pod="kube-system/kube-apiserver-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: E0913 03:00:01.936373    2352 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"kube-controller-manager-minikube\" already exists" pod="kube-system/kube-controller-manager-minikube"
Sep 13 03:00:01 minikube kubelet[2352]: E0913 03:00:01.937731    2352 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"etcd-minikube\" already exists" pod="kube-system/etcd-minikube"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.018307    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9959e106-20c6-4fa0-b0dc-f8be702afc2c-lib-modules\") pod \"kindnet-z9cjw\" (UID: \"9959e106-20c6-4fa0-b0dc-f8be702afc2c\") " pod="kube-system/kindnet-z9cjw"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.018367    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-cfg\" (UniqueName: \"kubernetes.io/host-path/9959e106-20c6-4fa0-b0dc-f8be702afc2c-cni-cfg\") pod \"kindnet-z9cjw\" (UID: \"9959e106-20c6-4fa0-b0dc-f8be702afc2c\") " pod="kube-system/kindnet-z9cjw"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.018382    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/9959e106-20c6-4fa0-b0dc-f8be702afc2c-xtables-lock\") pod \"kindnet-z9cjw\" (UID: \"9959e106-20c6-4fa0-b0dc-f8be702afc2c\") " pod="kube-system/kindnet-z9cjw"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.018402    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnwd\" (UniqueName: \"kubernetes.io/projected/9959e106-20c6-4fa0-b0dc-f8be702afc2c-kube-api-access-smnwd\") pod \"kindnet-z9cjw\" (UID: \"9959e106-20c6-4fa0-b0dc-f8be702afc2c\") " pod="kube-system/kindnet-z9cjw"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.622933    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgcm\" (UniqueName: \"kubernetes.io/projected/e5cf0872-18a1-4fb8-813e-aa79b3785b95-kube-api-access-mlgcm\") pod \"storage-provisioner\" (UID: \"e5cf0872-18a1-4fb8-813e-aa79b3785b95\") " pod="kube-system/storage-provisioner"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.622959    2352 reconciler_common.go:245] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/host-path/e5cf0872-18a1-4fb8-813e-aa79b3785b95-tmp\") pod \"storage-provisioner\" (UID: \"e5cf0872-18a1-4fb8-813e-aa79b3785b95\") " pod="kube-system/storage-provisioner"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.965587    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/coredns-6f6b679f8f-rzg2p" podStartSLOduration=1.9655753219999998 podStartE2EDuration="1.965575322s" podCreationTimestamp="2024-09-13 03:00:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-13 03:00:02.965290325 +0000 UTC m=+2.095354073" watchObservedRunningTime="2024-09-13 03:00:02.965575322 +0000 UTC m=+2.095639112"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.983248    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-proxy-z8vss" podStartSLOduration=1.983236141 podStartE2EDuration="1.983236141s" podCreationTimestamp="2024-09-13 03:00:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-13 03:00:02.977827238 +0000 UTC m=+2.107890986" watchObservedRunningTime="2024-09-13 03:00:02.983236141 +0000 UTC m=+2.113299930"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.993075    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/storage-provisioner" podStartSLOduration=0.993066311 podStartE2EDuration="993.066311ms" podCreationTimestamp="2024-09-13 03:00:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-13 03:00:02.983387911 +0000 UTC m=+2.113451701" watchObservedRunningTime="2024-09-13 03:00:02.993066311 +0000 UTC m=+2.123130100"
Sep 13 03:00:02 minikube kubelet[2352]: I0913 03:00:02.993126    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/coredns-6f6b679f8f-qcn76" podStartSLOduration=1.9931235269999998 podStartE2EDuration="1.993123527s" podCreationTimestamp="2024-09-13 03:00:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-13 03:00:02.993110067 +0000 UTC m=+2.123173856" watchObservedRunningTime="2024-09-13 03:00:02.993123527 +0000 UTC m=+2.123187317"
Sep 13 03:00:16 minikube kubelet[2352]: I0913 03:00:16.060502    2352 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef894c41fbf983b6461282871d0bdde2bd91c9e42b800963879a845a587449c4"
Sep 13 03:00:16 minikube kubelet[2352]: I0913 03:00:16.068119    2352 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kindnet-z9cjw" podStartSLOduration=2.068929461 podStartE2EDuration="15.068109136s" podCreationTimestamp="2024-09-13 03:00:01 +0000 UTC" firstStartedPulling="2024-09-13 03:00:02.401985305 +0000 UTC m=+1.532049053" lastFinishedPulling="2024-09-13 03:00:15.40116498 +0000 UTC m=+14.531228728" observedRunningTime="2024-09-13 03:00:16.061032049 +0000 UTC m=+15.191095839" watchObservedRunningTime="2024-09-13 03:00:16.068109136 +0000 UTC m=+15.198172884"
Sep 13 03:00:17 minikube kubelet[2352]: I0913 03:00:17.074542    2352 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="631a94a15c5e67ed938dc164326e07b201d1b2754c066849bf2201e90753b04a"
Sep 13 03:00:29 minikube kubelet[2352]: I0913 03:00:29.128699    2352 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e068edaa9099ddf202694c3ee39ab1a33d617c6b049911732f12aa6afe4b50f0"
Sep 13 03:00:29 minikube kubelet[2352]: I0913 03:00:29.128728    2352 scope.go:117] "RemoveContainer" containerID="444cf11d5ef18dc5afc67f7b259080e231a46c77e34b95fd66127ad7b8255cc9"
Sep 13 03:00:30 minikube kubelet[2352]: I0913 03:00:30.148107    2352 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbda206eb34e76c5620e40e47bfcbbfc71d4f1a229c0804235abdaf5c5fc13b2"
Sep 13 03:00:30 minikube kubelet[2352]: I0913 03:00:30.148133    2352 scope.go:117] "RemoveContainer" containerID="7a2279b4b6d56496e4f9b7a1387397f58824265666da860053385f76ffe5915b"

Operating System

MacOs

Driver

Podman

spowelljr commented 1 month ago

Hi @PunkFleet, I'm not sure why it's asking for a password for sudo, the docker user doesn't even have a password set. If you try SSHing into the cluster with minikube ssh and then try executing the command does it result in the same thing? Do all sudo commands result in bringing up the password? ie. try sudo ls Also what version of minikube are you using?

spowelljr commented 1 month ago

I see another issue for this, looks like it might be specific to HA

https://github.com/kubernetes/minikube/issues/19701

spowelljr commented 1 month ago

I was able to reproduce this, it's weird because I can run some sudo commands and it doesn't ask for a password, but the command in the HA docs does. Bringing in @prezha

spowelljr commented 1 month ago

This isn't even specific to HA, very odd

spowelljr commented 1 month ago

I figured it out, it's because the path in the command doesn't match the Kubernetes version you're using, the doc says v1.28.4 but you're using Kubernetes v1.31.0. If you replace that version in your command it should work.

PunkFleet commented 1 month ago

I figured it out, it's because the path in the command doesn't match the Kubernetes version you're using, the doc says v1.28.4 but you're using Kubernetes v1.31.0. If you replace that version in your command it should work.

Thanks for sharing, I haven't tried it yet but I think it should fix the problem!

prezha commented 1 month ago

pr #19722 aims to improve the docs and use the appropriate kubernetes version automatically