Weave network not deleted by 'kubeadm reset'

[cheburakshu]

I used kubeadm join with token and ip address of master. Immediately I had to reset it. But, kubeadm does not delete the weave network.

FYI - I did not install weave on node. I installed only on master as per the kubeadm install guide.

How to recover the node?

root@ip-172-20-0-207:~# kubeadm reset
[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml", assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
root@ip-172-20-0-207:~# ifconfig weave
weave     Link encap:Ethernet  HWaddr 06:26:f0:aa:3b:39  
          inet addr:10.40.0.0  Bcast:0.0.0.0  Mask:255.240.0.0
          inet6 addr: fe80::426:f0ff:feaa:3b39/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1376  Metric:1
          RX packets:48 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3044 (3.0 KB)  TX bytes:690 (690.0 B)

root@ip-172-20-0-207:~# 

[bboreham]

Easiest thing is to reboot the machine.

You can also download the weave script and run weave reset

[cheburakshu]

Reboot the machine? I can install with a command but need to reboot for an uninstall? Not easy! I don't think it is a works for me.

Also, you have the below note posted on the website. Are you suggesting to download weave on master or node? Is there a linux command to delete the weave network directly?

Note: If using the Weave CNI Plugin from a prior full install of Weave Net with your cluster, you must first uninstall it before applying the Weave-kube addon. Shut down Kubernetes, and on all nodes perform the following:

weave reset Remove any separate provisions you may have made to run Weave at boot-time, e.g. systemd units rm /opt/cni/bin/weave-* Then relaunch Kubernetes and install the addon as described above.

[cheburakshu]

Found - sudo ip link delete weave rm /opt/cni/bin/weave-*

Need to test if this doesn't break anything else.

[bboreham]

Are you suggesting to download weave on master or node?

That would be on every master and every node.

Is there a linux command to delete the weave network directly?

The weave netdev you are pointing out is a Linux bridge, and as you have seen you can delete a bridge with the ip command. However Weave Net also sets up an Open vSwitch datapath device, named datapath, that cannot be deleted with ip. Also a couple more ancillary devices.

You also have the option to ignore these devices: if you have no software using them then nothing bad will happen.

What may be more important is the CNI configuration file installed in /etc/cni/net.d - Kubelet will continue to see this and try to talk to the weave-kube daemonset, unless you change Kubelet's network plugin setting or install another CNI config. Even weave reset does not remove the CNI files.

The root problem, as I see it, is that Kubernetes does not tell the individual pods that the entire daemonset is being deleted: there is no hook we can act on to uninstall when required. I created a new issue https://github.com/kubernetes/kubernetes/issues/44911 to see if anyone has any ideas over there.

If any pods have been attached to Weave Net then deleting the bridge will remove their network access, and Kubernetes has no way to be told that it should do something to recover. Again, a reboot is the simplest way to get back to a stable state.

[bboreham]

Ah, I forgot you used kubeadm reset - it clears out /etc/cni/net.d so you're all right on that point. But more generally a weave-kube-uninstall should remove that file.

[bboreham]

2888 also referred to difficulties cleaning down weave-kube

[cheburakshu]

Thanks @bboreham for you detailed reply. I will summarize the problem that I have, the options that I have and few clarification I need.

Requirement:

1. The kubernetes master that the node joined is no more available (or the node doesn't want to be a part of master anymore). The node may want to join another master or function standalone. All kubernetes components (excluding binaries like kubeadm) need to be rolled back. kubeadm is used for install/uninstall of master & nodes.

What I did for this :

1. Ran kubeadm-reset on node since the node wanted to leave.

Issues:

1. The port 10250 still in use and kubelet is running.
2. Weave-net is still active.

Things that need to be done still:

1. kubelet is to be killed. lsof -Pni 10250 or ps -fu $LOGNAME |grep -i kubelet and kill -9 $PID
2. Weave needs to be uninstalled, so that the network bridge and all associated components go away.
3. The node should be available to join any other master using other network policies (Flannel etc.)

Weave Recommendation :

1. Per weave recommendation,

The recommended way of using Weave with Kubernetes is via the new Kubernetes Addon. The instructions below remain valid however, and are still the recommended method for integrating with Mesos.

2. Per reference Kubernetes Addon documentation,

Weave Net can be installed onto your CNI-enabled Kubernetes cluster with a single command:

kubectl apply -f https://git.io/weave-kube-1.6

After a few seconds, a Weave Net pod should be running on each Node and any further pods you create will be automatically attached to the Weave network.

Note: If using the Weave CNI Plugin from a prior full install of Weave Net with your cluster, you must first uninstall it before applying the Weave-kube addon. Shut down Kubernetes, and on all nodes perform the following:

weave reset Remove any separate provisions you may have made to run Weave at boot-time, e.g. systemd units rm /opt/cni/bin/weave-* Then relaunch Kubernetes and install the addon as described above.

Your Recommendation:

You can also download the weave script and run weave reset

Are you suggesting to download weave on master or node? <= My question

That would be on every master and every node. <= Your reply

Conflict of recommendations:

1. You said to download the binary on every master and every node while the Weavenet recommendation is exactly the opposite where it says to delete weavenet (including components in etc folder) before even getting started with weavenet on kubernetes. It seems to me that they are mutually exclusive - either use weavenet full install or use the addon and both cannot co-exist.
2. You want to install the binary everywhere but weavenet says to run the install command only on the master and the other nodes will automatically join the network. Why even full-install weavenet on the node when Point 1 says otherwise.
3. Say I want to delete weave on node and join another master - a. I will first install weave- full install (don't know what will happen to the already existing binaries that got downloaded during kubeadm join), b. Run weave reset to delete the network c. Delete all the components of full install (as per weave recommendation) d. Run kubeadm join

Clarification

If Kubernetes master dies, the k8s API server also dies. So, the node needs to be instructed by kubeadm reset or a weave standalone binary (if any) to do the cleanup, as there is no way for the node to receive instruction from master. This needs to be an activity on the individual node rather than acting on a master hook.

The root problem, as I see it, is that Kubernetes does not tell the individual pods that the entire daemonset is being deleted: there is no hook we can act on to uninstall when required

Instead of having the install, delete, install loop as described in (Conflict of recommendations, point 3), can't there be a standalone binary that gets downloaded during the join process that does the cleanup activity?

Where can I find this file/command? Is it a standalone binary and can be run on a node without a need for a full install or is it part of the full install?

But more generally a weave-kube-uninstall should remove that file.

The perspective is of a thoroughly confused used when using weave. Sorry!

[bboreham]

can't there be a standalone binary that gets downloaded during the join process that does the cleanup activity?

There is no hook in Kubernetes or Kubeadm to execute such a binary when required. I tagged this requirement onto https://github.com/kubernetes/kubernetes/issues/35183.

If you want to download and run it yourself, it's:

sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave

then

weave reset

This is running outside of Kubernetes, with no knowledge of Kubernetes. It gets the job done, that is all. It is not a "conflict", it's just different.

[cheburakshu]

@bboreham Thank you very much for your understanding and patient reply.. Appreciate it!

[klizhentas]

@bboreham how would you feel if we add cleanup in this script on SIGTERM on Pod termination, removing links created here:

https://github.com/weaveworks/weave/blob/f17a2a00b19c3bcf19416f176002bdf562ae4dd9/prog/weave-kube/launch.sh#L33

This will have a positive impact on the https://github.com/kubernetes/community/pull/483/files

Let me know!

[klizhentas]

will be properly fixed by https://github.com/kubernetes/community/pull/483/files

[bboreham]

@klizhentas as I said earlier the problem is that Kubernetes does not tell the individual pods that the entire daemonset is being deleted; we do not want to uninstall (hence break the network for all other pods) on deletion of an individual Weave Net pod because it may be a restart for upgrade.

https://github.com/kubernetes/community/pull/541 would give us a "reason for termination" which is what we need to act correctly. "deferContainers" are a solution to a different problem.

[bboreham]

This got worse with Weave Net 2.0, because it creates a data file on the host, and if you restart then the old contents of that file can disrupt operation. Maybe #3022 could help, indirectly.