apparmor: config provided but apparmor not supported

vsoch commented 5 years ago

[x] This is a bug report
[ ] This is a feature request
[ ] I searched existing issues before opening this one

Expected behavior

docker run hello-world
# works

Actual behavior

$ docker run hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"apply apparmor profile: apparmor: config provided but apparmor not supported\"": unknown.
ERRO[0000] error waiting for container: context canceled

Steps to reproduce the behavior

I can't really give you something to reproduce this because it's likely specific to my system - all I can say is that I hadn't changed anything, docker was working on evening, and the next morning is completely broken. I literally can't do anything. This is an urgent issue because I'm not able to do work for my job.

Output of docker version:

$ docker --version
Docker version 18.09.0, build 4d60db4

Output of docker info:

Containers: 66
 Running: 0
 Paused: 0
 Stopped: 66
Images: 1173
Server Version: 18.09.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1954
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 9397a6f62eb4252f01d98502105c49cfcd9feafe
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-39-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 19.47GiB
Name: vanessa-ThinkPad-T460s
ID: RHSP:4QXO:KZCS:O34F:RWEO:JLMN:EWF3:6K33:M2NR:UW4M:V234:Z2G3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: vanessa
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

This is Ubuntu 16.04, I've used Docker on it for years, no issues, and now it's just completely broken and I don't see a logical cause other than maybe system updates? Please help. :(

vsoch commented 5 years ago

In case it's helpful

$ dmesg | grep apparmor | grep docker
[   12.284751] audit: type=1400 audit(1543253652.680:40): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=3360 comm="apparmor_parser"

apparmor module is loaded.
39 profiles are loaded.
39 profiles are in enforce mode.
   /sbin/dhclient
   /snap/core/4917/usr/lib/snapd/snap-confine
   /snap/core/4917/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /snap/core/5897/usr/lib/snapd/snap-confine
   /snap/core/5897/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/evince
   /usr/bin/evince-previewer
   /usr/bin/evince-previewer//sanitized_helper
   /usr/bin/evince-thumbnailer
   /usr/bin/evince-thumbnailer//sanitized_helper
   /usr/bin/evince//sanitized_helper
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/cups/backend/cups-pdf
   /usr/lib/lightdm/lightdm-guest-session
   /usr/lib/lightdm/lightdm-guest-session//chromium
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/lib/telepathy/mission-control-5
   /usr/lib/telepathy/telepathy-*
   /usr/lib/telepathy/telepathy-*//pxgsettings
   /usr/lib/telepathy/telepathy-*//sanitized_helper
   /usr/lib/telepathy/telepathy-ofono
   /usr/sbin/cups-browsed
   /usr/sbin/cupsd
   /usr/sbin/cupsd//third_party
   /usr/sbin/ippusbxd
   /usr/sbin/tcpdump
   docker-default
   snap-update-ns.core
   snap-update-ns.hello-world
   snap.core.hook.configure
   snap.hello-world.env
   snap.hello-world.evil
   snap.hello-world.hello-world
   snap.hello-world.sh
   webbrowser-app
   webbrowser-app//oxide_helper
0 profiles are in complain mode.
5 processes have profiles defined.
5 processes are in enforce mode.
   /sbin/dhclient (27671) 
   /usr/lib/telepathy/mission-control-5 (5840) 
   /usr/sbin/cups-browsed (913) 
   /usr/sbin/cupsd (840) 
   /usr/sbin/cupsd (921) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

vsoch commented 5 years ago

Holy lord I fixed it! I don't know what did the trick, but I tried installing it different ways (via docker-engine, via docker-ce, and then back to the way I installed it years ago and all failed. I thought it might be related to an outdated runc (I have that in my $GOPATH) so I pulled latest from master and reinstalled. None of those worked. So I again purged everything:

$ sudo apt remove --purge docker*

And then for kicks and giggles, just tried this:

$ sudo apt install docker.io

And... hello-world seems to be working again! I am curious: 1. why do you think this could happen in the first place (something I can avoid in the future?) and what the difference is between all these different docker packages?

kargal commented 5 years ago

I had the same issue today after upgrade to docker-ce 18.09. When I reverted to docker-ce 18.06 everything is working fine again.

vsoch commented 5 years ago

Phew! Glad to hear! I still wonder if we might learn what was going on...

schu commented 5 years ago

Same issue here (currently with Docker version 18.09.1).

I thought it might be related to an outdated runc ...

Updating runc did the trick for me. I was on 1.0.0-rc4+dev (a618ab5a0186905949ee463dbb762c3d23e12a80) and did update to 1.0.0-rc6. @vsoch thanks for the pointer!

jomach commented 5 years ago

same issue here but on suse

nvidia-docker run --rm -it --security-opt apparmor=unconfined debian:jessie bash -i docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"apply apparmor profile: apparmor: config provided but apparmor not supported\"": unknown.

widgetpl commented 5 years ago

the same here:

$ sudo docker run -d -p 3000:3000 grafana/grafana
25a8e839175ce6559cbc7d0461bba4f77195af7eaa309ec578b6b9113ab70b13
docker: Error response from daemon: OCI runtime create failed: container_linux.go:336: starting container process caused "process_linux.go:402: container init caused \"apply apparmor profile: apparmor: config provided but apparmor not supported\"": unknown.

$ uname -a
Linux mike-dell 4.15.0-39-generic #42~16.04.1-Ubuntu SMP Wed Oct 24 17:09:54 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ runc --version
runc version 1.0.0-rc5+dev
commit: 2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7
spec: 1.0.0

northtyphoon commented 5 years ago

Hit the same issue after upgrade from 18.06.3 to 18.09.3.

PrimarchAlpharius commented 5 years ago

Same issue on 18.09.3, Ubuntu 18.04

alejandrox1 commented 5 years ago

For anyone that comes across this issue. I came across the same error after updating to latest docker-ce on ubuntu 18.04. My solution was to completely remove anything docker related from my machine (see this very useful stackoverflow answer). Then remove runc - in my case apt get remove --purge runc was not enough, I had to delete the binary from $GOPATH/bin/runc and /usr/local/sbin/runc which was the location pointed at by which runc.

After this, I just followed the installation instructions as normal and everything worked.

wasmup commented 5 years ago

Thank you @alejandrox1 I deleted runc twice:

sudo rm /usr/local/sbin/runc
sudo rm /usr/sbin/runc

Then it worked by reinstalling, The Error message was not informative:

$docker run  hello-world

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"apply apparmor profile: apparmor: config provided but apparmor not supported\"": unknown.
ERRO[0000] error waiting for container: context canceled 

$docker info

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.09.5

Silvanoc commented 5 years ago

It appears to be some incompatibility with certain runC versions. You might have a version of runC being taken by containerd due to you PATH configuration that is not consistent with the containerd version that you have.

By running whereis runc you can see which instances of runC are available in you system and visible through PATH.

Of course you can also use something like find / -name runc -executable, but will take much longer and the previous whereis might have been enough.

Anyway, find out if you have unexpected runC versions available in your system and clean them up, if that's the case.

vcvitaly commented 4 years ago

In my case I had 2 versions of runc

/usr/bin/runc provided by containerd.io package

vcvitaly@vc-pc:~/docker$ dpkg -S /usr/bin/runc
containerd.io: /usr/bin/runc

and /usr/local/sbin/runc which I build from sources the day before. I removed it and everything started working again

docker / for-linux