Closed ggari-robotnik closed 5 months ago
extended logs:
β Joining worker nodes π
Deleted nodes: ["fluidos-provider-control-plane" "fluidos-provider-worker" "fluidos-provider-worker2"]
ERROR: failed to create cluster: failed to join node with kubeadm: command "docker exec --privileged fluidos-provider-worker kubeadm join --config /kind/kubeadm.conf --skip-phases=preflight --v=6" failed with error: exit status 1
Command Output: I0423 07:52:39.553625 147 join.go:412] [preflight] found NodeName empty; using OS hostname as NodeName
I0423 07:52:39.553667 147 joinconfiguration.go:76] loading configuration from "/kind/kubeadm.conf"
I0423 07:52:39.554179 147 controlplaneprepare.go:225] [download-certs] Skipping certs download
I0423 07:52:39.554188 147 join.go:529] [preflight] Discovering cluster-info
I0423 07:52:39.554195 147 token.go:80] [discovery] Created cluster-info discovery client, requesting info from "fluidos-provider-control-plane:6443"
I0423 07:52:39.559846 147 round_trippers.go:553] GET https://fluidos-provider-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s 200 OK in 5 milliseconds
I0423 07:52:39.560000 147 token.go:223] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0423 07:52:45.430412 147 round_trippers.go:553] GET https://fluidos-provider-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s 200 OK in 1 milliseconds
I0423 07:52:45.430547 147 token.go:223] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0423 07:52:51.514020 147 round_trippers.go:553] GET https://fluidos-provider-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s 200 OK in 60 milliseconds
I0423 07:52:51.514640 147 token.go:105] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "fluidos-provider-control-plane:6443"
I0423 07:52:51.514650 147 discovery.go:52] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I0423 07:52:51.514668 147 join.go:543] [preflight] Fetching init configuration
I0423 07:52:51.514671 147 join.go:589] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I0423 07:52:51.569941 147 round_trippers.go:553] GET https://fluidos-provider-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s 200 OK in 54 milliseconds
I0423 07:52:51.595056 147 round_trippers.go:553] GET https://fluidos-provider-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy?timeout=10s 200 OK in 24 milliseconds
I0423 07:52:51.595779 147 kubelet.go:74] attempting to download the KubeletConfiguration from ConfigMap "kubelet-config"
I0423 07:52:51.596690 147 round_trippers.go:553] GET https://fluidos-provider-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config?timeout=10s 200 OK in 0 milliseconds
I0423 07:52:51.597589 147 interface.go:432] Looking for default routes with IPv4 addresses
I0423 07:52:51.597594 147 interface.go:437] Default route transits interface "eth0"
I0423 07:52:51.597646 147 interface.go:209] Interface eth0 is up
I0423 07:52:51.597667 147 interface.go:257] Interface "eth0" has 3 addresses :[10.250.3.7/24 fc00:f853:ccd:e793::7/64 fe80::42:aff:fefa:307/64].
I0423 07:52:51.597674 147 interface.go:224] Checking addr 10.250.3.7/24.
I0423 07:52:51.597678 147 interface.go:231] IP found 10.250.3.7
I0423 07:52:51.597682 147 interface.go:263] Found valid IPv4 address 10.250.3.7 for interface "eth0".
I0423 07:52:51.597684 147 interface.go:443] Found active IP 10.250.3.7
I0423 07:52:51.600934 147 kubelet.go:121] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I0423 07:52:51.601376 147 kubelet.go:136] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I0423 07:52:51.601602 147 loader.go:395] Config loaded from file: /etc/kubernetes/bootstrap-kubelet.conf
I0423 07:52:51.601827 147 kubelet.go:157] [kubelet-start] Checking for an existing Node in the cluster with name "fluidos-provider-worker" and status "Ready"
I0423 07:52:51.603024 147 round_trippers.go:553] GET https://fluidos-provider-control-plane:6443/api/v1/nodes/fluidos-provider-worker?timeout=10s 404 Not Found in 1 milliseconds
I0423 07:52:51.603200 147 kubelet.go:172] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
I0423 07:52:52.749423 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:52:53.750744 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:52:54.773399 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:52:55.749371 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:52:56.749488 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:52:57.749413 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:52:58.749529 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:52:59.750232 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:00.746713 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:01.749449 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:02.749550 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:03.749501 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:04.749418 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:05.751458 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:06.749398 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:07.749537 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:08.749501 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:09.749703 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:10.749546 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:11.749353 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:12.751393 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:13.749397 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:14.749429 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:15.750852 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:16.750729 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:17.749712 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:18.749937 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:19.749664 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:20.749970 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:21.747228 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:22.749394 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:23.753549 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:24.749676 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:25.749343 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:26.749494 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:27.749944 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:28.749567 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:29.749353 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:30.749400 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
[kubelet-check] Initial timeout of 40s passed.
I0423 07:53:31.749716 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0423 07:53:32.749852 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:33.749875 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:34.746694 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:35.750947 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:36.750302 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0423 07:53:37.749875 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:38.751449 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:39.749906 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:40.750025 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:41.749769 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:42.760600 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:43.749988 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:44.749579 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:45.750437 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:46.749992 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0423 07:53:47.749398 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:48.749421 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:49.749394 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:50.749513 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:51.746708 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:52.750075 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:53.751487 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:54.749472 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:55.750379 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:56.750250 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:57.746710 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:58.751121 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:53:59.749430 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:00.749445 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:01.746598 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:02.749505 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:03.749411 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:04.750684 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:05.749517 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:06.747092 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0423 07:54:07.749456 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:08.751458 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:09.749711 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:10.750543 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:11.750569 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:12.749363 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:13.749661 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:14.749542 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:15.749574 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:16.749842 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:17.749413 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:18.750094 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:19.749394 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:20.749480 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:21.747113 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:22.750420 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:23.749554 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:24.749566 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:25.749621 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:26.749982 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:27.749971 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:28.749967 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:29.749705 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:30.750638 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:31.746618 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:32.749373 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:33.750543 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:34.749481 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:35.749497 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:36.750967 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:37.749403 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:38.751443 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:39.749413 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:40.750721 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:41.749895 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:42.749382 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:43.749350 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:44.749970 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:45.749332 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0423 07:54:46.750561 147 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
timed out waiting for the condition
error execution phase kubelet-start
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
cmd/kubeadm/app/cmd/phases/workflow/runner.go:260
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1
cmd/kubeadm/app/cmd/join.go:179
github.com/spf13/cobra.(*Command).execute
vendor/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
vendor/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
vendor/github.com/spf13/cobra/command.go:992
k8s.io/kubernetes/cmd/kubeadm/app.Run
cmd/kubeadm/app/kubeadm.go:50
main.main
cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:250
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Hi @ggari-robotnik!
It looks like you're experiencing problems with Kubernetes in general rather than with the installation scripts.
This could be related more to kind
which is known to have configuration issues.
I suggest to try the following solutions:
swapoff -a
(it probably needs sudo
privileges) Let us know if you can solve the problem this way.
Thanks, Francesco
@fracappa That works! Here is the things that I did:
sudo swapoff -a
sudo sysctl fs.inotify.max_user_instances=512
sudo sysctl fs.inotify.max_user_watches=524288
If you need relaunch the setup.sh
kind delete clusters fluidos-consumer
This could be added to setup.sh or the guide.
Can you put some notes about it notes about it in the installation?
I think I won't be last one that will have this problems
Well done @ggari-robotnik!
We're going to put them as a side note since it looks like it's a common problem, even if it's not related directly with FLUIDOS technology.
We're keeping this issue open until we do it.
Thanks for your feedback!
I'm having problems to setup the kind testbed. the second cluster cannot be created
My system:
My procedure:
The results