Error response from daemon: unable to find "cpuacct" in controller set: unknown

TheAifam5 commented 5 years ago

[x] This is a bug report
[ ] This is a feature request
[x] I searched existing issues before opening this one

Expected behavior

The container is created and starts.

Actual behavior

After executing the command, throws and error, but the container exists and can not be started using docker start command because throws another error:

(Hashes in the output below fake/replaced)

Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/de6d5g46d4gh98d4gf6dsa6df4as6d46sa4d6a4s6d4as6d6as46das/merged: device or resource busy
Error: failed to start containers: asda4sd45as6546a

Steps to reproduce the behavior

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=<your password>" -p 1433:1433 --name mssql -d mcr.microsoft.com/mssql/server:latest

Output of docker version:

Client:
 Version:           18.09.0-ce
 API version:       1.39
 Go version:        go1.11.2
 Git commit:        4d60db472b
 Built:             Fri Nov  9 00:05:34 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.2
  Git commit:       4d60db472b
  Built:            Fri Nov  9 00:05:11 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
runc version: 079817cc26ec5292ac375bb9f47f373d33574949
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.20.0-3-ck-haswell
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.41GiB
Name: pussy-master
ID: XMIG:WGMU:7HCP:FWEN:7KS2:63YO:4RSV:YVLB:PK5V:R6TR:ON5F:LVTZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support

Additional environment details (AWS, VirtualBox, physical, etc.)

Clean docker installation.

Output of uname -a:

Linux ****-master 4.20.0-3-ck-haswell #1 SMP PREEMPT Fri Jan 4 17:46:51 EST 2019 x86_64 GNU/Linux

Output of ls -hl /sys/fs/cgroup/:

dr-xr-xr-x 5 root root  0 Jan  9 22:04 blkio
dr-xr-xr-x 5 root root  0 Jan  9 22:04 cpu
lrwxrwxrwx 1 root root  3 Jan  9 22:04 cpuacct -> cpu
dr-xr-xr-x 3 root root  0 Jan  9 22:04 cpuset
dr-xr-xr-x 5 root root  0 Jan  9 22:04 devices
dr-xr-xr-x 3 root root  0 Jan  9 22:04 freezer
dr-xr-xr-x 3 root root  0 Jan  9 22:04 hugetlb
dr-xr-xr-x 5 root root  0 Jan  9 22:04 memory
lrwxrwxrwx 1 root root 16 Jan  9 22:04 net_cls -> net_cls,net_prio
dr-xr-xr-x 3 root root  0 Jan  9 22:04 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Jan  9 22:04 net_prio -> net_cls,net_prio
dr-xr-xr-x 3 root root  0 Jan  9 22:04 perf_event
dr-xr-xr-x 5 root root  0 Jan  9 22:04 pids
dr-xr-xr-x 2 root root  0 Jan  9 22:04 rdma
dr-xr-xr-x 6 root root  0 Jan  9 22:04 systemd
dr-xr-xr-x 5 root root  0 Jan  9 22:04 unified

Output of mount | grep cgroup:

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)

OneEyedSpaceFish commented 5 years ago

I encountered the same issue just now after a recent update. I'm suspecting the systemd upgrade, but I'm not sure as I downgraded everything in an attempt to make docker work again asap.

Here is the list of the suspects:

[2019-01-09 09:12] [ALPM] upgraded iana-etc (20180913-1 -> 20181219-1)
[2019-01-09 09:12] [ALPM] upgraded libsystemd (239.370-1 -> 240.0-3)
[2019-01-09 09:12] [ALPM] upgraded libutil-linux (2.33-2 -> 2.33-3)
[2019-01-09 09:12] [ALPM] upgraded flashplugin (32.0.0.101-1 -> 32.0.0.114-1)
[2019-01-09 09:12] [ALPM] upgraded iproute2 (4.19.0-1 -> 4.20.0-1)
[2019-01-09 09:12] [ALPM] upgraded util-linux (2.33-2 -> 2.33-3)
[2019-01-09 09:12] [ALPM] upgraded systemd (239.370-1 -> 240.0-3)
[2019-01-09 09:12] [ALPM] upgraded libinput (1.12.4-1 -> 1.12.5-1)
[2019-01-09 09:12] [ALPM] upgraded libxslt (1.1.32+3+g32c88216-1 -> 1.1.33-1)
[2019-01-09 09:12] [ALPM] upgraded man-db (2.8.4-1 -> 2.8.5-1)
[2019-01-09 09:12] [ALPM] upgraded php (7.3.0-1 -> 7.3.1-1)
[2019-01-09 09:12] [ALPM] upgraded systemd-sysvcompat (239.370-1 -> 240.0-3)
[2019-01-09 09:12] [ALPM] upgraded xorg-xcursorgen (1.0.6-2 -> 1.0.7-1)
[2019-01-10 09:26] [ALPM] upgraded firefox (64.0-1 -> 64.0.2-1)
[2019-01-10 09:26] [ALPM] upgraded linux-ck-haswell (4.20-3 -> 4.20.1-1)
[2019-01-10 09:26] [ALPM] upgraded pacman-mirrorlist (20181205-1 -> 20190109-1)
[2019-01-10 09:26] [ALPM] upgraded pacman (5.1.1-1 -> 5.1.2-2)
[2019-01-10 09:26] [ALPM] upgraded yarn (1.12.3-1 -> 1.13.0-1)
[2019-01-10 09:32] [ALPM] upgraded phpstorm (2018.3.2-1 -> 2018.3.3-1)
[2019-01-10 09:32] [ALPM] upgraded phpstorm-jre (2018.3.2-1 -> 2018.3.3-1)

My first guess was linux-ck-haswell, so that's the package I downgraded first. Close, but no cigar as the issue persisted after the downgrade.

TheAifam5 commented 5 years ago

Ah yea, I updated systemd too :D

EDIT: systemd changelog about cgroup

        * CPUAccounting=yes no longer enables the CPU controller when using
          kernel 4.15+ and the unified cgroup hierarchy, as required accounting
          statistics are now provided independently from the CPU controller.

        * Support for disabling a particular cgroup controller within a sub-tree
          has been added through the DisableControllers= directive.

        * cgroup_no_v1=all on the kernel command line now also implies
          using the unified cgroup hierarchy, unless one explicitly passes
          systemd.unified_cgroup_hierarchy=0 on the kernel command line.

        * The new "MemoryMin=" unit file property may now be used to set the
          memory usage protection limit of processes invoked by the unit. This
          controls the cgroup v2 memory.min attribute. Similarly, the new
          "IODeviceLatencyTargetSec=" property has been added, wrapping the new
          cgroup v2 io.latency cgroup property for configuring per-service I/O
          latency.

        * systemd now supports the cgroup v2 devices BPF logic, as counterpart
          to the cgroup v1 "devices" cgroup controller.

TheAifam5 commented 5 years ago

Downgrading systemd to 239.370 fixes the problem.

OneEyedSpaceFish commented 5 years ago

@TheAifam5 Have you checked if this issue is present on systemd's side or dockers? We need to create a new issue appropriately, so this can be fixed.

TheAifam5 commented 5 years ago

@OneEyedSpaceFish Hard to say. Systemd & Linux Kernel mostly i think.

kubrickfr commented 5 years ago

FYI, https://github.com/containerd/cgroups/pull/77 fixes the issue for me on Arch & systemd 241.7-2.

vinibali commented 2 years ago

just for the record: i had pretty much the same error message with Arch Linux using the last Linux-CK kernel (5.12) with MuQSS. in my case MuQSS was the rootcause for this error, but I could make it work with adding this line to the kernel parameters:

systemd.unified_cgroup_hierarchy=0

thanks for that!

docker / for-linux