search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
28.04k stars 2.35k forks source link

Streaming server stopped unexpectedly #3712

Closed pratikbin closed 2 years ago

pratikbin commented 3 years ago

Environmental Info: K3s Version: v1.21.3+k3s1

Node(s) CPU architecture, OS, and Version: Linux ctos 5.9.16-1-MANJARO #1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020 x86_64 GNU/Linux, 16GB, i5 10thGen,

Cluster Configuration: Single Node with Docker runtime

Describe the bug: Want to run k3s with docker runtime 1st try: curl -sfL https://get.k3s.io | sh -s - --docker

+ /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Failed to get unit file state for nm-cloud-setup.service: No such file or directory
time="2021-07-26T18:24:31.781674773+05:30" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
time="2021-07-26T18:24:31.783699262+05:30" level=info msg="Cluster bootstrap already complete"
time="2021-07-26T18:24:31.796546141+05:30" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2021-07-26T18:24:31.796590717+05:30" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2021-07-26T18:24:31.796707207+05:30" level=info msg="Database tables and indexes are up to date"
time="2021-07-26T18:24:31.797840342+05:30" level=info msg="Kine listening on unix://kine.sock"
time="2021-07-26T18:24:31.797997181+05:30" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0726 18:24:31.799809  444609 server.go:656] external host was not specified, using 10.0.8.87
I0726 18:24:31.800003  444609 server.go:195] Version: v1.21.3+k3s1
I0726 18:24:31.803454  444609 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0726 18:24:31.804490  444609 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 18:24:31.804513  444609 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0726 18:24:31.805735  444609 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 18:24:31.805750  444609 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0726 18:24:31.829180  444609 instance.go:283] Using reconciler: lease
I0726 18:24:31.866028  444609 rest.go:130] the default service ipfamily for this cluster is: IPv4
W0726 18:24:32.209699  444609 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0726 18:24:32.221974  444609 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0726 18:24:32.227045  444609 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0726 18:24:32.233763  444609 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0726 18:24:32.237310  444609 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0726 18:24:32.243942  444609 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
W0726 18:24:32.243967  444609 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
I0726 18:24:32.265841  444609 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 18:24:32.265867  444609 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
time="2021-07-26T18:24:32.279983217+05:30" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
time="2021-07-26T18:24:32.280032517+05:30" level=info msg="Waiting for API server to become available"
time="2021-07-26T18:24:32.280542302+05:30" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
time="2021-07-26T18:24:32.281109525+05:30" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
time="2021-07-26T18:24:32.282438094+05:30" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
time="2021-07-26T18:24:32.282471747+05:30" level=info msg="To join node to cluster: k3s agent -s https://10.0.8.87:6443 -t ${NODE_TOKEN}"
time="2021-07-26T18:24:32.283503789+05:30" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
time="2021-07-26T18:24:32.283534390+05:30" level=info msg="Run: k3s kubectl"
time="2021-07-26T18:24:32.339345596+05:30" level=info msg="Cluster-Http-Server 2021/07/26 18:24:32 http: TLS handshake error from 127.0.0.1:52210: remote error: tls: bad certificate"
time="2021-07-26T18:24:32.343567560+05:30" level=info msg="Cluster-Http-Server 2021/07/26 18:24:32 http: TLS handshake error from 127.0.0.1:52216: remote error: tls: bad certificate"
time="2021-07-26T18:24:32.351620225+05:30" level=info msg="certificate CN=ctos signed by CN=k3s-server-ca@1627303950: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 12:54:32 +0000 UTC"
time="2021-07-26T18:24:32.354491937+05:30" level=info msg="certificate CN=system:node:ctos,O=system:nodes signed by CN=k3s-client-ca@1627303950: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 12:54:32 +0000 UTC"
time="2021-07-26T18:24:32.358118198+05:30" level=info msg="Module overlay was already loaded"
time="2021-07-26T18:24:32.358143447+05:30" level=info msg="Module nf_conntrack was already loaded"
time="2021-07-26T18:24:32.358151771+05:30" level=info msg="Module br_netfilter was already loaded"
time="2021-07-26T18:24:32.358160063+05:30" level=info msg="Module iptable_nat was already loaded"
time="2021-07-26T18:24:32.365779524+05:30" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2021-07-26T18:24:32.367500734+05:30" level=info msg="Handling backend connection request [ctos]"
time="2021-07-26T18:24:32.368359468+05:30" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --network-plugin=cni --node-labels= --pod-infra-container-image=rancher/pause:3.1 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
time="2021-07-26T18:24:32.368867068+05:30" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Flag --network-plugin has been deprecated, will be removed along with dockershim.
W0726 18:24:32.369161  444609 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
E0726 18:24:32.383711  444609 node.go:161] Failed to retrieve node info: nodes "ctos" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
I0726 18:24:32.393649  444609 server.go:436] "Kubelet version" kubeletVersion="v1.21.3+k3s1"
I0726 18:24:32.423114  444609 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
W0726 18:24:32.423122  444609 manager.go:159] Cannot detect current cgroup on cgroup v2
I0726 18:24:32.501501  444609 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I0726 18:24:32.501713  444609 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I0726 18:24:32.501790  444609 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
I0726 18:24:32.501830  444609 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I0726 18:24:32.501844  444609 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
I0726 18:24:32.501855  444609 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
I0726 18:24:32.501947  444609 kubelet.go:307] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
I0726 18:24:32.501989  444609 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
I0726 18:24:32.502013  444609 client.go:97] "Start docker client with request timeout" timeout="2m0s"
I0726 18:24:32.509025  444609 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
I0726 18:24:32.509049  444609 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
I0726 18:24:32.515251  444609 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
I0726 18:24:32.521560  444609 docker_service.go:264] "Docker Info" dockerInfo=&{ID:G5RL:4X7T:W5YM:4BWC:LGDL:VYV7:O7QU:A425:2U33:MLLS:NFGR:T2IO Containers:22 ContainersRunning:1 ContainersPaused:0 ContainersStopped:21 Images:229 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:45 SystemTime:2021-07-26T18:24:32.515685567+05:30 LoggingDriver:json-file CgroupDriver:systemd CgroupVersion:2 NEventsListener:0 KernelVersion:5.9.16-1-MANJARO OperatingSystem:Manjaro Linux OSVersion: OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc003715c00 NCPU:8 MemTotal:16432685056 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ctos Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:36cc874494a56a253cd181a1a685b44b58a2e34a.m Expected:36cc874494a56a253cd181a1a685b44b58a2e34a.m} RuncCommit:{ID:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 Expected:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: DefaultAddressPools:[] Warnings:[]}
E0726 18:24:32.521597  444609 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"cgroupfs\" is different from docker cgroup driver: \"systemd\""
k3s.service: Main process exited, code=exited, status=1/FAILURE

2nd try: add kubelet flag curl -sfL https://get.k3s.io | sh -s - --kubelet-arg 'cgroup-driver=systemd' --docker

+ /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Failed to get unit file state for nm-cloud-setup.service: No such file or directory
time="2021-07-26T19:43:45.773788090+05:30" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
time="2021-07-26T19:43:45.773936226+05:30" level=info msg="Cluster bootstrap already complete"
time="2021-07-26T19:43:45.784963404+05:30" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2021-07-26T19:43:45.784988646+05:30" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2021-07-26T19:43:45.785072907+05:30" level=info msg="Database tables and indexes are up to date"
time="2021-07-26T19:43:45.786106017+05:30" level=info msg="Kine listening on unix://kine.sock"
time="2021-07-26T19:43:45.786303643+05:30" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0726 19:43:45.787752  602560 server.go:656] external host was not specified, using 10.0.8.87
I0726 19:43:45.787933  602560 server.go:195] Version: v1.21.3+k3s1
I0726 19:43:45.792464  602560 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0726 19:43:45.792966  602560 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 19:43:45.792980  602560 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0726 19:43:45.793822  602560 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 19:43:45.793832  602560 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0726 19:43:45.806814  602560 instance.go:283] Using reconciler: lease
I0726 19:43:45.860030  602560 rest.go:130] the default service ipfamily for this cluster is: IPv4
W0726 19:43:46.128420  602560 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0726 19:43:46.138016  602560 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0726 19:43:46.141459  602560 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0726 19:43:46.147181  602560 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0726 19:43:46.150571  602560 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0726 19:43:46.156566  602560 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
W0726 19:43:46.156587  602560 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
I0726 19:43:46.167785  602560 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 19:43:46.167803  602560 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
time="2021-07-26T19:43:46.176753451+05:30" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
time="2021-07-26T19:43:46.176808565+05:30" level=info msg="Waiting for API server to become available"
time="2021-07-26T19:43:46.177179338+05:30" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
time="2021-07-26T19:43:46.177559790+05:30" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
time="2021-07-26T19:43:46.178555112+05:30" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
time="2021-07-26T19:43:46.178581003+05:30" level=info msg="To join node to cluster: k3s agent -s https://10.0.8.87:6443 -t ${NODE_TOKEN}"
time="2021-07-26T19:43:46.179294860+05:30" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
time="2021-07-26T19:43:46.179319588+05:30" level=info msg="Run: k3s kubectl"
time="2021-07-26T19:43:46.219669890+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:43:46 http: TLS handshake error from 127.0.0.1:53082: remote error: tls: bad certificate"
time="2021-07-26T19:43:46.222848739+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:43:46 http: TLS handshake error from 127.0.0.1:53088: remote error: tls: bad certificate"
time="2021-07-26T19:43:46.230245755+05:30" level=info msg="certificate CN=ctos signed by CN=k3s-server-ca@1627303950: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:13:46 +0000 UTC"
time="2021-07-26T19:43:46.232510407+05:30" level=info msg="certificate CN=system:node:ctos,O=system:nodes signed by CN=k3s-client-ca@1627303950: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:13:46 +0000 UTC"
time="2021-07-26T19:43:46.236051185+05:30" level=info msg="Module overlay was already loaded"
time="2021-07-26T19:43:46.236072017+05:30" level=info msg="Module nf_conntrack was already loaded"
time="2021-07-26T19:43:46.236079584+05:30" level=info msg="Module br_netfilter was already loaded"
time="2021-07-26T19:43:46.236087911+05:30" level=info msg="Module iptable_nat was already loaded"
time="2021-07-26T19:43:46.242127052+05:30" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2021-07-26T19:43:46.243832486+05:30" level=info msg="Handling backend connection request [ctos]"
time="2021-07-26T19:43:46.244772268+05:30" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --network-plugin=cni --node-labels= --pod-infra-container-image=rancher/pause:3.1 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
time="2021-07-26T19:43:46.245470654+05:30" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Flag --network-plugin has been deprecated, will be removed along with dockershim.
W0726 19:43:46.245789  602560 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
E0726 19:43:46.258742  602560 node.go:161] Failed to retrieve node info: nodes "ctos" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
I0726 19:43:46.273577  602560 server.go:436] "Kubelet version" kubeletVersion="v1.21.3+k3s1"
I0726 19:43:46.288772  602560 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
W0726 19:43:46.288783  602560 manager.go:159] Cannot detect current cgroup on cgroup v2
I0726 19:43:46.368300  602560 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I0726 19:43:46.368445  602560 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I0726 19:43:46.368497  602560 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
I0726 19:43:46.368525  602560 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I0726 19:43:46.368533  602560 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
I0726 19:43:46.368539  602560 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
I0726 19:43:46.368592  602560 kubelet.go:307] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
I0726 19:43:46.368619  602560 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
I0726 19:43:46.368629  602560 client.go:97] "Start docker client with request timeout" timeout="2m0s"
I0726 19:43:46.373900  602560 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
I0726 19:43:46.373920  602560 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
I0726 19:43:46.381650  602560 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
I0726 19:43:46.387693  602560 docker_service.go:264] "Docker Info" dockerInfo=&{ID:G5RL:4X7T:W5YM:4BWC:LGDL:VYV7:O7QU:A425:2U33:MLLS:NFGR:T2IO Containers:22 ContainersRunning:1 ContainersPaused:0 ContainersStopped:21 Images:229 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:45 SystemTime:2021-07-26T19:43:46.382166814+05:30 LoggingDriver:json-file CgroupDriver:systemd CgroupVersion:2 NEventsListener:0 KernelVersion:5.9.16-1-MANJARO OperatingSystem:Manjaro Linux OSVersion: OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0002cfe30 NCPU:8 MemTotal:16432685056 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ctos Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:36cc874494a56a253cd181a1a685b44b58a2e34a.m Expected:36cc874494a56a253cd181a1a685b44b58a2e34a.m} RuncCommit:{ID:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 Expected:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: DefaultAddressPools:[] Warnings:[]}
I0726 19:43:46.387713  602560 docker_service.go:277] "Setting cgroupDriver" cgroupDriver="systemd"
E0726 19:43:46.387880  602560 docker_service.go:416] "Streaming server stopped unexpectedly" err="listen tcp 10.43.164.71:0: bind: cannot assign requested address"
k3s.service: Main process exited, code=exited, status=1/FAILURE

3rd disable service lb and traefik curl -sfL https://get.k3s.io | sh -s - --kubelet-arg 'cgroup-driver=systemd' --docker --disable traefik --disable servicelb

+ /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Failed to get unit file state for nm-cloud-setup.service: No such file or directory
time="2021-07-26T19:45:13.963861889+05:30" level=info msg="Starting k3s v1.21.3+k3s1 (1d1f220f)"
time="2021-07-26T19:45:13.964070011+05:30" level=info msg="Cluster bootstrap already complete"
time="2021-07-26T19:45:13.975286718+05:30" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2021-07-26T19:45:13.975317362+05:30" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2021-07-26T19:45:13.975402264+05:30" level=info msg="Database tables and indexes are up to date"
time="2021-07-26T19:45:13.976423984+05:30" level=info msg="Kine listening on unix://kine.sock"
time="2021-07-26T19:45:13.976586440+05:30" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0726 19:45:13.977932  606291 server.go:656] external host was not specified, using 10.0.8.87
I0726 19:45:13.978128  606291 server.go:195] Version: v1.21.3+k3s1
I0726 19:45:13.980607  606291 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0726 19:45:13.982441  606291 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 19:45:13.982628  606291 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0726 19:45:13.984569  606291 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 19:45:13.984589  606291 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0726 19:45:13.999510  606291 instance.go:283] Using reconciler: lease
I0726 19:45:14.053824  606291 rest.go:130] the default service ipfamily for this cluster is: IPv4
W0726 19:45:14.322164  606291 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0726 19:45:14.331918  606291 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0726 19:45:14.335625  606291 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0726 19:45:14.342944  606291 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0726 19:45:14.345444  606291 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0726 19:45:14.350097  606291 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
W0726 19:45:14.350111  606291 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
I0726 19:45:14.361069  606291 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0726 19:45:14.361151  606291 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
time="2021-07-26T19:45:14.371727177+05:30" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
time="2021-07-26T19:45:14.371787457+05:30" level=info msg="Waiting for API server to become available"
time="2021-07-26T19:45:14.372280189+05:30" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
time="2021-07-26T19:45:14.372726800+05:30" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
time="2021-07-26T19:45:14.373706046+05:30" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
time="2021-07-26T19:45:14.373731794+05:30" level=info msg="To join node to cluster: k3s agent -s https://10.0.8.87:6443 -t ${NODE_TOKEN}"
time="2021-07-26T19:45:14.374453070+05:30" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
time="2021-07-26T19:45:14.374474378+05:30" level=info msg="Run: k3s kubectl"
time="2021-07-26T19:45:14.414400347+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:45:14 http: TLS handshake error from 127.0.0.1:53724: remote error: tls: bad certificate"
time="2021-07-26T19:45:14.418621169+05:30" level=info msg="Cluster-Http-Server 2021/07/26 19:45:14 http: TLS handshake error from 127.0.0.1:53730: remote error: tls: bad certificate"
time="2021-07-26T19:45:14.426465276+05:30" level=info msg="certificate CN=ctos signed by CN=k3s-server-ca@1627303950: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:15:14 +0000 UTC"
time="2021-07-26T19:45:14.428272397+05:30" level=info msg="certificate CN=system:node:ctos,O=system:nodes signed by CN=k3s-client-ca@1627303950: notBefore=2021-07-26 12:52:30 +0000 UTC notAfter=2022-07-26 14:15:14 +0000 UTC"
time="2021-07-26T19:45:14.431628901+05:30" level=info msg="Module overlay was already loaded"
time="2021-07-26T19:45:14.431651641+05:30" level=info msg="Module nf_conntrack was already loaded"
time="2021-07-26T19:45:14.431659785+05:30" level=info msg="Module br_netfilter was already loaded"
time="2021-07-26T19:45:14.431665993+05:30" level=info msg="Module iptable_nat was already loaded"
time="2021-07-26T19:45:14.438055331+05:30" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2021-07-26T19:45:14.439675149+05:30" level=info msg="Handling backend connection request [ctos]"
time="2021-07-26T19:45:14.440586097+05:30" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --network-plugin=cni --node-labels= --pod-infra-container-image=rancher/pause:3.1 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
time="2021-07-26T19:45:14.441207253+05:30" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ctos --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Flag --network-plugin has been deprecated, will be removed along with dockershim.
W0726 19:45:14.441501  606291 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
E0726 19:45:14.454113  606291 node.go:161] Failed to retrieve node info: nodes "ctos" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
I0726 19:45:14.467739  606291 server.go:436] "Kubelet version" kubeletVersion="v1.21.3+k3s1"
I0726 19:45:14.491377  606291 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
W0726 19:45:14.491452  606291 manager.go:159] Cannot detect current cgroup on cgroup v2
I0726 19:45:14.579771  606291 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I0726 19:45:14.579941  606291 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I0726 19:45:14.580010  606291 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
I0726 19:45:14.580048  606291 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I0726 19:45:14.580062  606291 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
I0726 19:45:14.580072  606291 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
I0726 19:45:14.580141  606291 kubelet.go:307] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
I0726 19:45:14.580179  606291 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
I0726 19:45:14.580195  606291 client.go:97] "Start docker client with request timeout" timeout="2m0s"
I0726 19:45:14.585680  606291 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
I0726 19:45:14.585699  606291 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
I0726 19:45:14.591373  606291 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
I0726 19:45:14.598480  606291 docker_service.go:264] "Docker Info" dockerInfo=&{ID:G5RL:4X7T:W5YM:4BWC:LGDL:VYV7:O7QU:A425:2U33:MLLS:NFGR:T2IO Containers:22 ContainersRunning:1 ContainersPaused:0 ContainersStopped:21 Images:229 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:45 SystemTime:2021-07-26T19:45:14.591902881+05:30 LoggingDriver:json-file CgroupDriver:systemd CgroupVersion:2 NEventsListener:0 KernelVersion:5.9.16-1-MANJARO OperatingSystem:Manjaro Linux OSVersion: OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000a320e0 NCPU:8 MemTotal:16432685056 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ctos Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:36cc874494a56a253cd181a1a685b44b58a2e34a.m Expected:36cc874494a56a253cd181a1a685b44b58a2e34a.m} RuncCommit:{ID:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 Expected:b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: DefaultAddressPools:[] Warnings:[]}
I0726 19:45:14.598501  606291 docker_service.go:277] "Setting cgroupDriver" cgroupDriver="systemd"
E0726 19:45:14.598740  606291 docker_service.go:416] "Streaming server stopped unexpectedly" err="listen tcp 10.43.164.71:0: bind: cannot assign requested address"
k3s.service: Main process exited, code=exited, status=1/FAILURE

Steps To Reproduce:

Expected behavior: it should run

Backporting not sure

brandond commented 3 years ago

I don't think we've ever tested on Manjaro; it sounds like their Docker installation might be customized somehow, in a way that's not compatible with Kubernetes?

I'm particularly confused by this line: E0726 19:43:46.387880 602560 docker_service.go:416] "Streaming server stopped unexpectedly" err="listen tcp 10.43.164.71:0: bind: cannot assign requested address"

It sounds like the kubelet is unable to set up docker networking. Do you have other things already running in Docker on this node?

pratikbin commented 3 years ago

Yeah one container is running.

pratikbin commented 3 years ago

Well I tried after clearing docker containers, still same last error @brandond FYI, I didn't tweak anything in docker.

This could be?

W0726 18:24:32.423122  444609 manager.go:159] Cannot detect current cgroup on cgroup v2
brandond commented 3 years ago

I'm not sure, sorry. I don't think I've seen anyone else try to get it working with the Docker runtime on Manjaro. Can I ask why you're using Docker instead of containerd?

pratikbin commented 3 years ago

Long story short, I'm having doubt that concourse worker behaving weird/having issues in containerd, so wanted to test that in docker with a bunch of other things.

brandond commented 3 years ago

You might try again with one of the distros we test against - RHEL/CentOS, Ubuntu, etc.

mrwormhole commented 2 years ago

I have run into this same problem on Debian 11, but

curl -sfL https://get.k3s.io | sh -s - --kubelet-arg 'cgroup-driver=systemd' --docker

did work for me thanks @pratikbalar

Note: you don't actually need docker to run k3s in your nodes, it is deprecated

stale[bot] commented 2 years ago

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.