Weave NPC ipset failed: ipset v6.32: The set is full, more elements cannot be added

neubi4 commented 6 years ago

What you expected to happen?

I have the following kubernetes network policy:

kind: NetworkPolicy
metadata:
  name: ingress-web
  labels:
    project: domon
spec:
  podSelector:
    matchLabels:
      project: domon
      service: zed
      helm-release: domon-zed
      app: web
  policyTypes:
    - Ingress
  ingress:
    - from:
      - namespaceSelector: {}
      ports:
        - protocol: TCP
          port: 443

I also have 10 (and more in the future) namespaces wich will match in this policy. I expected this to work.

It worked as i reduced the number of namespaces to under 8. 8 is the default size of a ipset list, and the ipset here has Header: size 8 set ipset list.

I think weave does not set a size for the ipset list, and therfor it is created with the default of 8, and the npc dies when trying to add a ninth entry to the set.

What happened?

weave-npc dies with the following log entrys:

INFO: 2018/04/27 12:32:06.427648 EVENT AddNetworkPolicy {"metadata":{"creationTimestamp":"2018-04-25T05:59:14Z","generation":1,"labels":{"helm-release":"domon-zed","project":"domon","service":"zed"},"name":"ingress-web","namespace":"domon-zed","resourceVersion":"3275743","selfLink":"/apis/networking.k8s.io/v1/namespaces/domon-zed/networkpolicies/ingress-web","uid":"c8025294-484d-11e8-a2ff-005056831273"},"spec":{"ingress":[{"from":[{"namespaceSelector":{}}],"ports":[{"port":443,"protocol":"TCP"}]}],"podSelector":{"matchLabels":{"app":"web","helm-release":"domon-zed","project":"domon","service":"zed"}},"policyTypes":["Ingress"]}}
INFO: 2018/04/27 12:32:06.430199 creating ipset: &npc.selectorSpec{key:"", selector:labels.internalSelector{}, dst:false, ipsetType:"list:set", ipsetName:"weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb", nsName:""}
INFO: 2018/04/27 12:32:06.431637 adding entry weave-5sf{9])}VR[BYoo^J;dfaG?aW to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of bc9c9548-4244-11e8-a2ff-005056831273
INFO: 2018/04/27 12:32:06.431674 added entry weave-5sf{9])}VR[BYoo^J;dfaG?aW to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of bc9c9548-4244-11e8-a2ff-005056831273
INFO: 2018/04/27 12:32:06.433182 adding entry weave-QmEsPzU.egMcDS05Cj7}JvPcW to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of b2c2af99-46ef-11e8-a2ff-005056831273
INFO: 2018/04/27 12:32:06.433224 added entry weave-QmEsPzU.egMcDS05Cj7}JvPcW to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of b2c2af99-46ef-11e8-a2ff-005056831273
INFO: 2018/04/27 12:32:06.434694 adding entry weave-kb*.Oh5tpLoOpLjt^k]$52M1f to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 67618ae7-3e3e-11e8-abe6-005056833d35
INFO: 2018/04/27 12:32:06.434727 added entry weave-kb*.Oh5tpLoOpLjt^k]$52M1f to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 67618ae7-3e3e-11e8-abe6-005056833d35
INFO: 2018/04/27 12:32:06.436561 adding entry weave-k?Z;25^M}|1s7P3|H9i;*;MhG to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of db26a752-3330-11e8-9530-00505683210f
INFO: 2018/04/27 12:32:06.436619 added entry weave-k?Z;25^M}|1s7P3|H9i;*;MhG to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of db26a752-3330-11e8-9530-00505683210f
INFO: 2018/04/27 12:32:06.438840 adding entry weave-_~[lE64J9!5Xy(JIqNGU+rVEU to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 3a560365-3beb-11e8-99ab-0050568362b8
INFO: 2018/04/27 12:32:06.438891 added entry weave-_~[lE64J9!5Xy(JIqNGU+rVEU to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 3a560365-3beb-11e8-99ab-0050568362b8
INFO: 2018/04/27 12:32:06.441794 adding entry weave-iuZcey(5DeXbzgRFs8Szo]+@p to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of dbb99d5a-3330-11e8-9530-00505683210f
INFO: 2018/04/27 12:32:06.442103 added entry weave-iuZcey(5DeXbzgRFs8Szo]+@p to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of dbb99d5a-3330-11e8-9530-00505683210f
INFO: 2018/04/27 12:32:06.443773 adding entry weave-#!K#HR!YHbK?|GUPj}eALLUSz to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 1231db9c-3989-11e8-99ab-0050568362b8
INFO: 2018/04/27 12:32:06.443805 added entry weave-#!K#HR!YHbK?|GUPj}eALLUSz to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 1231db9c-3989-11e8-99ab-0050568362b8
INFO: 2018/04/27 12:32:06.445315 adding entry weave-YN)GDEqg?_S}OweFSRNER#Y9$ to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 195c10a9-416a-11e8-99ab-0050568362b8
INFO: 2018/04/27 12:32:06.445345 added entry weave-YN)GDEqg?_S}OweFSRNER#Y9$ to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of 195c10a9-416a-11e8-99ab-0050568362b8
INFO: 2018/04/27 12:32:06.446835 adding entry weave-4vtqMI+kx/2]jD%_c0S%thO%V to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of dc95ec2f-3330-11e8-9530-00505683210f
INFO: 2018/04/27 12:32:06.446875 added entry weave-4vtqMI+kx/2]jD%_c0S%thO%V to weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb of dc95ec2f-3330-11e8-9530-00505683210f
FATA: 2018/04/27 12:32:06.448420 add network policy: ipset [add weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb weave-4vtqMI+kx/2]jD%_c0S%thO%V] failed: ipset v6.32: The set is full, more elements cannot be added.

How to reproduce it?

Create more than 10 namespaces and use the networkpolicy above.

Versions:

$ weave version 2.3.0 $ docker version

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:09:15 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:13:03 2018
  OS/Arch:      linux/amd64
  Experimental: false

$ uname -a Linux zed-lmon-node1 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux $ kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

brb commented 6 years ago

Thanks for the issue.

Indeed, the default size of an ipset of the list:set type is 8. We can change it by passing "size N" when creating the ipset.

bboreham commented 6 years ago

I tried this on a 4.13 kernel, and the set size doesn't seem so fixed:

Name: weave-!KbcPP7#:Z;q|kV;:5)S@i1Yb
Type: list:set
Revision: 3
Header: size 8 comment
Size in memory: 1302
References: 1
Number of entries: 13
Members:
weave-N)8V}/RNBk}Hbq6iU7s^%D(4B comment "namespace: namespace1"
weave-zSTsK3R(n!XSc0to@fOZqySyo comment "namespace: namespace3"
weave-O.^I?XH/+TR*#ChsvaNf}0IM2 comment "namespace: namespace5"
weave-aRm4{a[@0CNQauQyb!yH.ZF)7 comment "namespace: namespace6"
weave-3[yUxF6(WCeM;/38{TEz^0goM comment "namespace: namespace7"
weave-nQg#GCV@k/*}hIE%67X9K6QJt comment "namespace: namespace8"
weave-iuZcey(5DeXbzgRFs8Szo]+@p comment "namespace: kube-system"
weave-k?Z;25^M}|1s7P3|H9i;*;MhG comment "namespace: default"
weave-CLJ9y.p0@5^Qv.m=.A4UUhG/o comment "namespace: namespace10"
weave-c;1_zMt}g|7Hn%!|sEo^5xSBv comment "namespace: namespace4"
weave-?}8a?)Oi$epp:oB@JKrB0;:)b comment "namespace: namespace9"
weave-4vtqMI+kx/2]jD%_c0S%thO%V comment "namespace: kube-public"
weave-6uV3q:VWP##e_D(zA6WcoNKN6 comment "namespace: namespace2"

brb commented 6 years ago

the set size doesn't seem so fixed

From man 8 ipset:

       size value
              The  size  of the list, the default is 8. The parameter is ignored since ipset
              version 6.24.

So, we can assume that a set of the list:set type can grow dynamically since 6.24 (released 4 years ago).

bboreham commented 6 years ago

OK, so do we have any theories about the problem the OP reported? I wondered if it is different in Red Hat (inferred from OP reporting kernel 3.10.0)

brb commented 6 years ago

I've just checked with CentOS 7 (kernel 3.10), and the problem exists as reported.

neubi4 commented 6 years ago

Thanks for your help.

In our case its CentOS Linux release 7.4.1708 (Core) with kernel 3.10.0.

ipset version is

# ipset --version
ipset v6.29, protocol version: 6

but man 8 ipset only says

       size value
              The size of the list, the default is 8.

brb commented 6 years ago

Fixed by #3305

weaveworks / weave