search

moby / libnetwork

networking for containers
Apache License 2.0
2.16k stars 881 forks source link

OOM-kill in dockerd 20.10, tries to allocate 2³² IPAM subnets #2683

Closed ulidtko closed 1 month ago

ulidtko commented 4 months ago

Very briefly. With IPv6 enabled, and this entry in default-address-pools (daemon.json):

                , {"base":"fd0d:0ce5::/48", "size": 80}

Dockerd fails to start, crashes with fatal error: runtime: out of memory.

Crash log: ubQ795Mx.txt

As can be easily found in the crash log, OOM happens while in ipamutils.splitNetworks:

https://github.com/moby/libnetwork/blob/3797618f9a38372e8107d8c06f6ae199e1133ae8/ipamutils/utils.go#L92-L93

The config shown, {"base":"fd0d:0ce5::/48", "size": 80} — denotes 2³² /80 subnets. This is over 4 billion subnets.

Related issue: https://github.com/moby/moby/issues/40275

docker version 
Client:
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        20.10.21-0ubuntu1~22.04.3
 Built:             Thu Apr 27 05:57:17 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.21-0ubuntu1~22.04.3
  Built:            Thu Apr 27 05:37:25 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.7.12
  GitCommit:        
 runc:
  Version:          1.1.12-0ubuntu2~22.04.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:
akerouanton commented 1 month ago

Thanks for reporting. This was also reported in moby/moby, under https://github.com/moby/moby/issues/40275.

The following PR introduced a new subnet allocator which doesn't suffer from this issue. It was released in v27.0, on June 25.

Note that:

  1. Docker Engine v20.10 was EOL'd a year ago, and thus you might miss important security fixes.
  2. The code here is now part of moby/moby. This repo is mostly defunct, and new bug reports should be opened there. This repo can't be archived yet as we need to migrate issues and PRs.
ulidtko commented 1 month ago

Thanks so much for accurate response @akerouanton