search

cybozu-go / coil

CNI plugin for Kubernetes designed for scalability and extensibility
Apache License 2.0
164 stars 20 forks source link

Failed to create AddressBlock #152

Closed masa213f closed 3 years ago

masa213f commented 3 years ago

Describe the bug

A Cluster IP did not assign to a pod. The pod remained ContainerCreating status. 

$ kubectl get pod -n logging ingester-0 -o wide
NAME         READY   STATUS              RESTARTS   AGE     IP       NODE          NOMINATED NODE   READINESS GATES
ingester-0   0/1     ContainerCreating   0          5h28m   <none>   10.69.1.132   <none>           <none>

$ kubectl describe pod -n logging ingester-0
(...)

Events:
  Type     Reason                  Age                   From     Message
  ----     ------                  ----                  ----     -------
  Warning  FailedCreatePodSandBox  2m33s (x78 over 92m)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f4036648c160210acd505f6419a0c02b578454ac25762391be76b309a21c8716": failed to allocate address; aborting new block request: context deadline exceeded

A coil-controller logged the following messages. It seems that the creation of an AddressBlock was failed.

cybozu@gcp0-boot-0:~$ stern -n kube-system coil-controller-
(...)
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767715.798942,"logger":"blockrequest-reconciler","msg":"internal error","blockrequest":"req-default-10.69.1.132","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767715.7990007,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"coil.cybozu.com","reconcilerKind":"BlockRequest","controller":"blockrequest","name":"req-default-10.69.1.132","namespace":"","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767716.9866533,"logger":"pool-manager","msg":"failed to create AddressBlock","pool":"default","index":7,"node":"10.69.0.4","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767716.9867475,"logger":"blockrequest-reconciler","msg":"internal error","blockrequest":"req-default-10.69.0.4","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767716.9867713,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"coil.cybozu.com","reconcilerKind":"BlockRequest","controller":"blockrequest","name":"req-default-10.69.0.4","namespace":"","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767726.7400553,"logger":"pool-manager","msg":"failed to create AddressBlock","pool":"default","index":7,"node":"10.69.1.132","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767726.7401295,"logger":"blockrequest-reconciler","msg":"internal error","blockrequest":"req-default-10.69.1.132","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767726.7401562,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"coil.cybozu.com","reconcilerKind":"BlockRequest","controller":"blockrequest","name":"req-default-10.69.1.132","namespace":"","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767727.8927827,"logger":"pool-manager","msg":"failed to create AddressBlock","pool":"default","index":7,"node":"10.69.0.4","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767727.892883,"logger":"blockrequest-reconciler","msg":"internal error","blockrequest":"req-default-10.69.0.4","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"error","ts":1617767727.892939,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"coil.cybozu.com","reconcilerKind":"BlockRequest","controller":"blockrequest","name":"req-default-10.69.0.4","namespace":"","error":"addressblocks.coil.cybozu.com \"default-7\" already exists"}

Environments

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

masa213f commented 3 years ago

In this environment, 8 addressblocks were created from the default addresspool.

cybozu@gcp0-boot-0:~$ kubectl get addressblock | grep default
default-0           10.69.0.5     default           10.64.0.0/27
default-1           10.69.0.196   default           10.64.0.32/27
default-2           10.69.1.133   default           10.64.0.64/27
default-3           10.69.0.196   default           10.64.0.96/27
default-4           10.69.0.197   default           10.64.0.128/27
default-5           10.69.1.132   default           10.64.0.160/27
default-6           10.69.0.4     default           10.64.0.192/27
default-7           10.69.1.134   default           10.64.0.224/27

But coil-controller failed to count the addressblocks. According to the following logs, it is misjudged as 7.

cybozu@gcp0-boot-0:~$ stern -n kube-system coil-controller-
(...)
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"info","ts":1617760489.8024213,"logger":"blockrequest-reconciler","msg":"allocated","blockrequest":"req-default-10.69.1.132","block":"defaul
t-5","index":5,"pool":"default"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"info","ts":1617760511.0823762,"logger":"pool-manager","msg":"created AddressBlock","pool":"default","index":6,"node":"10.69.0.4"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"info","ts":1617760511.082449,"logger":"blockrequest-reconciler","msg":"allocated","blockrequest":"req-default-10.69.0.4","block":"default-6
","index":6,"pool":"default"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"info","ts":1617760603.1189501,"logger":"pool-manager","msg":"created AddressBlock","pool":"default","index":7,"node":"10.69.1.134"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"info","ts":1617760603.119075,"logger":"blockrequest-reconciler","msg":"allocated","blockrequest":"req-default-10.69.1.134","block":"default
-7","index":7,"pool":"default"}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"info","ts":1617760603.1197767,"logger":"pool-manager","msg":"resynced block usage","pool":"default","blocks":7}
coil-controller-cc56ff6f-sw5rq coil-controller {"level":"info","ts":1617760603.1198223,"logger":"addresspool-reconciler","msg":"synchronized"}
masa213f commented 3 years ago

The List in pool.SyncBlocks might not list all resources due to the local cache. Use Manager.GetAPIReader() to list addressblocks, instead of Manager.GetClient().