IP not released when pause container is deleted by accident

[clivez]

A deployment with replicas 4 is created, using danmnet 'test-net1' only.

[root@clive-alp-control-01 case]# kubectl get po -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP            NODE                  NOMINATED NODE   READINESS GATES
busybox-7b449fd984-2hgwf   1/1     Running   0          6s    10.10.10.12   clive-alp-worker-01   <none>           <none>
busybox-7b449fd984-6v2cx   1/1     Running   0          7s    10.10.10.10   clive-alp-worker-03   <none>           <none>
busybox-7b449fd984-786ck   1/1     Running   0          6s    10.10.10.13   clive-alp-worker-02   <none>           <none>
busybox-7b449fd984-dgr7n   1/1     Running   0          6s    10.10.10.11   clive-alp-worker-01   <none>           <none>

Then the pause container of one pod is forcely deleted.

[root@clive-alp-worker-01 ~]# docker ps | grep busybox-7b449fd984-2hgwf
c1f4317d0c86        bcmt-registry:5000/busybox                                     "top"                    About a minute ago   Up About a minute
           k8s_busybox_busybox-7b449fd984-2hgwf_default_43b900cb-4218-4d04-a218-0680192f0781_0
91aa3a98df8f        bcmt-registry:5000/gcr.io/google-containers/pause-amd64:3.1    "/pause"                 About a minute ago   Up About a minute
           k8s_POD_busybox-7b449fd984-2hgwf_default_43b900cb-4218-4d04-a218-0680192f0781_0
[root@clive-alp-worker-01 ~]#
[root@clive-alp-worker-01 ~]# docker rm -f 91aa3a98df8f
91aa3a98df8f

After that the pod got a new IP address, but the previous one was not released.

[root@clive-alp-control-01 case]# kubectl get po -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP            NODE                  NOMINATED NODE   READINESS GATES
busybox-7b449fd984-2hgwf   1/1     Running   1          96s   10.10.10.14   clive-alp-worker-01   <none>           <none>
busybox-7b449fd984-6v2cx   1/1     Running   0          97s   10.10.10.10   clive-alp-worker-03   <none>           <none>
busybox-7b449fd984-786ck   1/1     Running   0          96s   10.10.10.13   clive-alp-worker-02   <none>           <none>
busybox-7b449fd984-dgr7n   1/1     Running   0          96s   10.10.10.11   clive-alp-worker-01   <none>           <none>
[root@clive-alp-control-01 case]#
[root@clive-alp-control-01 case]# kubectl get danmnet test-net1 -o yaml
apiVersion: danm.k8s.io/v1
kind: DanmNet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"danm.k8s.io/v1","kind":"DanmNet","metadata":{"annotations":{},"name":"test-net1","namespace":"default"},"spec":{"NetworkID":"test-net1","NetworkType":"ipvlan","Options":{"allocation_pool":{"end":"10.10.10.100","start":"10.10.10.10"},"cidr":"10.10.10.0/24","host_device":"eth2","vxlan":101}}}
  creationTimestamp: "2020-02-14T07:44:09Z"
  generation: 6
  name: test-net1
  namespace: default
  resourceVersion: "70392995"
  selfLink: /apis/danm.k8s.io/v1/namespaces/default/danmnets/test-net1
  uid: 32e5ca89-a9ad-418c-b326-c9158bd72bd4
spec:
  NetworkID: test-net1
  NetworkType: ipvlan
  Options:
    alloc: gD4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAE=
    allocation_pool:
      end: 10.10.10.100
      start: 10.10.10.10
    cidr: 10.10.10.0/24
    host_device: eth2
    vxlan: 101
[root@clive-alp-control-01 case]#
[root@clive-alp-control-01 case]# kubectl get danmep -o yaml | grep 10.10.10
      Address: 10.10.10.13/24
      Address: 10.10.10.10/24
      Address: 10.10.10.11/24
      Address: 10.10.10.12/24
      Address: 10.10.10.14/24
[root@clive-alp-control-01 case]# kubectl get danmep -o yaml | grep Pod:
    Pod: busybox-7b449fd984-786ck
    Pod: busybox-7b449fd984-6v2cx
    Pod: busybox-7b449fd984-dgr7n
    Pod: busybox-7b449fd984-2hgwf
    Pod: busybox-7b449fd984-2hgwf

[Levovar]

we are not interfaceing with container runtime, and it is on purpose I don't consider "pause container deleted by accident" a valid production scenario. Nobody shall have access to the direct container runtime APIs in a production environment besides Kubelet.

[Levovar]

and just to be clear :) we also firmly consider interfacing with the container runtime out of scope as well. that is a dependency we don't want to introduce into any part of the DANM eco-system. hence closing this ticket with won't fix

[liurd]

Levo, Product is doing such testing - delete pause container, and this a real case will happen. "out of scope" it technical judgement, while business need a solution for such case.

[Levovar]

just because you created a test case for an imagined scenario, it doesn't mean it is a real-life production scenario. the business solution can -and should- be appliying approriate access control on your container runtime APIs, making sure no one can manually delete a running container manually deleting containers can have other side effects, not just networking related. dangling volume mounts, out of synch network device allocations etc.