search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
748 stars 85 forks source link

Docker fails to start with unable to find net_prio in controller set unknown #1064

Open MitchGaines opened 4 years ago

MitchGaines commented 4 years ago

Expected behavior

Running docker run hello-world should create a container and execute the hello-world image

Actual behavior

Note: The first time I ran this, it successfully found and downloaded the image. Issue appears to be when it goes to start up the container. I'm connected via a serial interface because I found that its output may be relevant. 

root@raspberrypi4-64:~# docker run hello-world
[   38.961757] docker0: port 1(vethd673de4) entered blocking state
[   38.971062] docker0: port 1(vethd673de4) entered disabled state
[   38.977531] device vethd673de4 entered promiscuous mode
[   38.987233] IPv6: ADDRCONF(NETDEV_UP): vethd673de4: link is not ready
[   40.143584] eth0: renamed from veth13129e4
[   40.167330] IPv6: ADDRCONF(NETDEV_CHANGE): vethd673de4: link becomes ready
[   40.174556] docker0: port 1(vethd673de4) entered blocking state
[   40.180603] docker0: port 1(vethd673de4) entered forwarding state
[   40.187362] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
[   40.395791] docker0: port 1(vethd673de4) entered disabled state
[   40.402370] veth13129e4: renamed from eth0
[   40.463366] docker0: port 1(vethd673de4) entered disabled state
[   40.495725] device vethd673de4 left promiscuous mode
[   40.500855] docker0: port 1(vethd673de4) entered disabled state
docker: Error response from daemon: unable to find "net_prio" in controller set: unknown.
ERRO[0002] error waiting for container: context canceled

Steps to reproduce the behavior

I'm using a Raspberry Pi 4 Model B with a Yocto warrior build. Probably not relevant, but I'm running it with mender. Below is a copy of the relevant portion of my local.conf and bblayers.conf.

local.conf:

MACHINE ?= "raspberrypi4-64"

RPI_USE_U_BOOT = "1"
MENDER_PARTITION_ALIGNMENT = "4194304"
MENDER_BOOT_PART_SIZE_MB = "40"
IMAGE_INSTALL_append = " kernel-image kernel-devicetree"

MENDER_FEATURES_ENABLE_append = " mender-uboot mender-image-sd mender-image"
MENDER_FEATURES_DISABLE_append = " mender-grub mender-image-uefi"

# DOCKER
ENABLE_UART = "1"
CORE_IMAGE_EXTRA_INSTALL += "kernel-modules iperf3 bash"
IMAGE_INSTALL_append = " docker-ce"
INHERIT += "extrausers"
DISTRO_FEATURES_append = " systemd virtualization"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"

# PYTHON
IMAGE_INSTALL_append = " python3-pip python3"

bblayers.conf:

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ${TOPDIR}/../sources/poky/meta \
  ${TOPDIR}/../sources/poky/meta-poky \
  ${TOPDIR}/../sources/poky/meta-yocto-bsp \
  ${TOPDIR}/../sources/meta-raspberrypi \
  ${TOPDIR}/../sources/meta-mender/meta-mender-core \
  ${TOPDIR}/../sources/meta-mender/meta-mender-raspberrypi \
  ${TOPDIR}/../sources/meta-openembedded/meta-oe \
  ${TOPDIR}/../sources/meta-openembedded/meta-multimedia \
  ${TOPDIR}/../sources/meta-openembedded/meta-networking \
  ${TOPDIR}/../sources/meta-openembedded/meta-python \
  ${TOPDIR}/../sources/meta-openembedded/meta-perl \
  ${TOPDIR}/../sources/meta-openembedded/meta-filesystems \
  ${TOPDIR}/../sources/meta-virtualization \
  redacted \
  redacted \
"

I built using bitbake core-image-base. I've tried just about everything available online so the local.conf looks like a bit of a hodge podge just because I was trying everything.

When I run journalctl -fu docker:

root@raspberrypi4-64:~# journalctl -fu docker
-- Logs begin at Thu 1970-01-01 00:00:03 UTC. --
Jul 09 14:23:29 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:29.260061879Z" level=warning msg="failed to retrieve docker-init version: exec: \"docker-init\": executable file not found in $PATH"
Jul 09 14:23:29 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:29.410409046Z" level=info msg="Docker daemon" commit=f5e591e815841e04e910223fb63f1ef7689ae153 graphdriver(s)=overlay2 version=18.09.3-ce
Jul 09 14:23:29 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:29.412825676Z" level=info msg="Daemon has completed initialization"
Jul 09 14:23:29 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:29.503585342Z" level=info msg="API listen on /var/run/docker.sock"
Jul 09 14:23:29 raspberrypi4-64 systemd[1]: Started Docker Application Container Engine.
Jul 09 14:23:51 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:51.039768163Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ec27b3aaa072d9bb5859587f1f16b22c9b2af51cd11181ea4c00f9d24fa7c4ef/shim.sock" debug=false pid=454
Jul 09 14:23:52 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:52.179153810Z" level=info msg="shim reaped" id=ec27b3aaa072d9bb5859587f1f16b22c9b2af51cd11181ea4c00f9d24fa7c4ef
Jul 09 14:23:52 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:52.500834588Z" level=error msg="ec27b3aaa072d9bb5859587f1f16b22c9b2af51cd11181ea4c00f9d24fa7c4ef cleanup: failed to delete container from containerd: no such container"
Jul 09 14:23:52 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:23:52.501042477Z" level=error msg="Handler for POST /v1.39/containers/ec27b3aaa072d9bb5859587f1f16b22c9b2af51cd11181ea4c00f9d24fa7c4ef/start returned error: unable to find \"net_prio\" in controller set: unknown"
Jul 09 14:32:58 raspberrypi4-64 dockerd[269]: time="2020-07-09T14:32:58.438107172Z" level=warning msg="failed to retrieve docker-init version: exec: \"docker-init\": executable file not found in $PATH"

Output of docker version:

root@raspberrypi4-64:~# docker version
Client:
 Version:           18.09.3-ce
 API version:       1.39
 Go version:        go1.12.9
 Git commit:        f5e591e
 Built:             Wed Jul  8 21:15:09 2020
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.3-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.12.9
  Git commit:       f5e591e815841e04e910223fb63f1ef7689ae153
  Built:            Wed Jul  8 21:11:43 2020
  OS/Arch:          linux/arm64
  Experimental:     false

Output of docker info:

root@raspberrypi4-64:~# docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 1
Server Version: 18.09.3-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e.m
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d-dirty
init version: N/A (expected: )
Kernel Version: 4.19.88
Operating System: Poky (Yocto Project Reference Distro) 2.7.4 (warrior)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.718GiB
Name: raspberrypi4-64
ID: SZN3:V5S3:CWDG:T4GR:HUCJ:LWKD:UZYV:5P4M:B57S:BW2J:JUIO:B6ME
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Additional environment details (AWS, VirtualBox, physical, etc.)

Update 7/15/2020: I have further isolated the issue to docker-ce. Problem persists without mender layers existing. When I install a basic Ubuntu image and install docker.io, it works without issue. So, its not a hardware issue. This issue appears to be with docker-ce, as I expected.

AstraLuma commented 3 years ago

I'm getting this with Debian 10 on Odroid HC4 (arm64) with Docker 18.09.1