Open ygouzerh opened 2 years ago
ygouzerh
commented
2 years ago Actually I found the issue, we need to put --wait, k3d was terminating the command to early.
You can close this ticket when you want, as it might be interesting maybe to includes it in the documentation or increase the timeout time.
EDIT : False alarm, the issue is still pending
ygouzerh
commented
2 years ago Actually I found the issue, we need to put
--wait, k3d was terminating the command to early.You can close this ticket when you want, as it might be interesting maybe to includes it in the documentation or increase the timeout time.
Actually, I realized that one server keeps restarting, with the following logs:
E1010 10:44:51.326517 7 dynamic_cafile_content.go:166] "Failed to watch CA file, will retry later" err="error creating fsnotify watcher: too many open files"
time="2022-10-10T10:44:51Z" level=info msg="Stopped tunnel to 127.0.0.1:6443"
time="2022-10-10T10:44:51Z" level=info msg="Connecting to proxy" url="wss://192.168.128.3:6443/v1-k3s/connect"
time="2022-10-10T10:44:51Z" level=info msg="Connecting to proxy" url="wss://192.168.128.2:6443/v1-k3s/connect"
time="2022-10-10T10:44:51Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2022-10-10T10:44:51Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
W1010 10:44:51.335516 7 lease.go:234] Resetting endpoints for master service "kubernetes" to [192.168.128.2 192.168.128.3 192.168.128.4]
time="2022-10-10T10:44:51Z" level=info msg="Connecting to proxy" url="wss://192.168.128.4:6443/v1-k3s/connect"
time="2022-10-10T10:44:51Z" level=info msg="Handling backend connection request [k3d-cryptoblk-kentro-01-server-2]"
time="2022-10-10T10:44:51Z" level=info msg="Handling backend connection request [k3d-cryptoblk-kentro-01-server-1]"
time="2022-10-10T10:44:51Z" level=info msg="Handling backend connection request [k3d-cryptoblk-kentro-01-server-0]"
W1010 10:44:51.393843 7 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
I1010 10:44:51.394249 7 server.go:644] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
I1010 10:44:51.394691 7 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I1010 10:44:51.394760 7 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName:/k3s SystemCgroupsName: KubeletCgroupsName:/k3s KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:false CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
I1010 10:44:51.394781 7 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I1010 10:44:51.394793 7 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=true
I1010 10:44:51.394836 7 state_mem.go:36] "Initialized new in-memory state store"
I1010 10:44:51.399001 7 kubelet.go:376] "Attempting to sync node with API server"
I1010 10:44:51.399037 7 kubelet.go:267] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
I1010 10:44:51.399069 7 kubelet.go:278] "Adding apiserver pod source"
I1010 10:44:51.399094 7 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
E1010 10:44:51.399163 7 file_linux.go:61] "Unable to read config path" err="unable to create inotify: too many open files" path="/var/lib/rancher/k3s/agent/pod-manifests"
I1010 10:44:51.399927 7 kuberuntime_manager.go:239] "Container runtime initialized" containerRuntime="containerd" version="v1.6.6-k3s1" apiVersion="v1"
E1010 10:44:51.400113 7 plugins.go:604] "Error initializing dynamic plugin prober" err="error initializing watcher: too many open files"
I1010 10:44:51.400303 7 server.go:1177] "Started kubelet"
I1010 10:44:51.400402 7 server.go:150] "Starting to listen" address="0.0.0.0" port=10250
E1010 10:44:51.400566 7 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
E1010 10:44:51.400595 7 kubelet.go:1298] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
I1010 10:44:51.401182 7 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
I1010 10:44:51.401261 7 volume_manager.go:289] "Starting Kubelet Volume Manager"
I1010 10:44:51.401312 7 desired_state_of_world_populator.go:145] "Desired state populator starts to run"
I1010 10:44:51.401999 7 server.go:410] "Adding debug handlers to kubelet server"
E1010 10:44:51.404186 7 manager.go:279] Registration of the raw container factory failed: inotify_init: too many open files
E1010 10:44:51.404245 7 kubelet.go:1361] "Failed to start cAdvisor" err="inotify_init: too many open files"I found in the Github Actions that you bypass this error when running the E2E test, using :
sudo prlimit --pid $$ --nofile=1048576:1048576
sudo sysctl fs.inotify.max_user_instances=1280
sudo sysctl fs.inotify.max_user_watches=655360I added that inside the script executing my k3d create ..., and it works ! (Doesn't work if I execute it before running the script)
I tried multiple time, it still works, good news !
As before, you can close this ticket when you prefer, in case you want to put it in the documentation
What did you do
I was trying to create a cluster using :
k3d cluster create --server 3 --trace cluster-dev, unfortunately impossible to create.However, when running
k3d cluster create cluster-dev, it works well.Screenshots or terminal output
Output of
k3d cluster create --server 3 --trace cluster-dev:Output of
k3d cluster create dev-cluster:However, deleting this cluster will throw the warning :
But it seems well created :
Output of
docker volume ls | grep k3dWhich OS & Architecture
Which version of
k3dWhich version of docker
Output of docker version :