auto-generated seccomp profile doesn't work

[AkihiroSuda]

$ sudo ./docker-slim p nginx:alpine
docker-slim[profile]: info=http.probe message='using default probe'
docker-slim[profile]: state=started
docker-slim[profile]: info=params target=nginx:alpine
docker-slim[profile]: state=image.inspection.start
docker-slim[profile]: info=image id=sha256:55ceb2abad47854d182034514057b038a6e50e9d029ba152bd713d535ed39603 size.bytes=21137785 size.human=21 MB
docker-slim[profile]: state=image.inspection.done
docker-slim[profile]: state=container.inspection.start
docker-slim[profile]: info=container status=created id=adb10d2f8e69cda92a5f35b92d4f21f70c6268c4c053f99170aadce284add001
docker-slim[profile]: info=cmd.startmonitor status=sent
docker-slim[profile]: info=event.startmonitor.done status=received
docker-slim[build]: info=container name=dockerslimk_29216_20190805070200 id=adb10d2f8e69cda92a5f35b92d4f21f70c6268c4c053f99170aadce284add001 target.port.list=[32799] target.port.info=[80/tcp => 0.0.0.0:32799] message='YOU CAN USE THESE PORTS TO INTERACT WITH THE CONTAINER'
docker-slim[profile]: state=http.probe.starting message='WAIT FOR HTTP PROBE TO FINISH'
docker-slim[profile]: info=prompt message='USER INPUT REQUIRED, PRESS <ENTER> WHEN YOU ARE DONE USING THE CONTAINER'
docker-slim[profile]: state=http.probe.running
docker-slim[profile]: info=http.probe.call status=200 method=GET target=http://127.0.0.1:32799/ attempt=1  time=2019-08-05T07:02:12Z
docker-slim[profile]: info=http.probe.summary total=1 failures=0 successful=1
docker-slim[profile]: state=http.probe.done

docker-slim[profile]: state=container.inspection.finishing
docker-slim[profile]: state=container.inspection.artifact.processing
docker-slim[profile]: state=container.inspection.done
docker-slim[profile]: state=completed
docker-slim[profile]: state=done

$ docker run --rm -p 80:80 --security-opt seccomp=./.docker-slim-state/images/55ceb2abad47854d182034514057b038a6e50e9d029ba152bd713d535ed39603/artifacts/nginx-seccomp.json nginx:alpine
docker: Error response from daemon: cannot start a stopped process: unknown.

• docker-slim version: 1.25.3
• Docker version: moby/moby@11e48badcb
• OS: Ubuntu 19.04

[AkihiroSuda]

The generated profile:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": [
    "SCMP_ARCH_X86_64"
  ],
  "syscalls": [
    {
      "name": "lseek",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "capset",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getuid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getgid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "mprotect",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "ioctl",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "uname",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "bind",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "fstat",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "fchown",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "pwrite",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "sched_getaffinity",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "epoll_create1",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "prlimit64",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "socketpair",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "fcntl",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getdents64",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "rt_sigprocmask",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "setsockopt",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "execve",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "open",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getdents64",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "close",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getpid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "listen",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getgid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "arch_prctl",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "pread",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "setuid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "mmap",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "capget",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "chdir",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "stat",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "socket",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "chown",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "setgid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "read",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "geteuid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "set_tid_address",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "brk",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "mkdir",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getuid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getegid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getppid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "openat",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "setgroups",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "prctl",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getppid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "getpid",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "rt_sigaction",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "dup2",
      "action": "SCMP_ACT_ALLOW"
    },
    {
      "name": "fork",
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

[kcq]

Thank you for reporting the issue @AkihiroSuda! Can you share more about the container? What kind of application is it? How is the docker image structured/configured? Also wonder if you have a version with an older version of Ubuntu (18.x or below). Ubuntu 19.04 and 19.10 have problems with minification (still investigating the root cause for that).

[AkihiroSuda]

The image is nginx:alpine without any modification.

I'll try Ubuntu 18.04 later.

[kcq]

Thank you @AkihiroSuda! I'll try to repro the condition and share my finding tomorrow.

[AkihiroSuda]

Tried Ubuntu 18.04.2 (kernel 4.15.0-1037-gcp #39-Ubuntu) with Docker 18.09.7 and docker-slim 1.25.3, but same error

[kcq]

Thanks for confirming @AkihiroSuda! Investigating...

[AkihiroSuda]

@kcq

It looks runc per se requires some syscalls to be enabled explicitly, and probably the set of the required syscalls has changed since the last time you tested docker-slim seccomp generator.

The list is being discussed in https://github.com/opencontainers/runc/issues/2097

[kcq]

Yep :-) The extra syscall list is also different depending on when the seccomp profile is installed. If you use "no-new-privileges" (with an extra --security-opt param) then the list of extra syscalls is smaller. It's taking a bit of extra time chasing another bug... For some reason, the socket syscall in the profile is not recognized.

[kcq]

@AkihiroSuda do you mind trying a test build to see if it's enough (it includes a couple of extra calls Docker itself needs): https://github.com/docker-slim/docker-slim/releases/download/1.25.3/dist_linux_1.25.3.2.tar.gz
https://github.com/docker-slim/docker-slim/releases/download/1.25.3/dist_mac_1.25.3.2.zip

Still working on a bigger enhancement (to have a full and lite versions of the seccomp profile and to have better support for multi-threaded applications and to use the new seccomp profile format)...

[AkihiroSuda]

thanks, but nginx:alpine still doesn't work

$ docker run -it --rm -p 80:80 --security-opt seccomp=.docker-slim-state/images/55ceb2abad47854d182034514057b038a6e50e9d029ba152bd713d535ed39603/artifacts/nginx-seccomp.json nginx:alpine
docker: Error response from daemon: cannot start a stopped process: unknown.
ERRO[0000] error waiting for container: context canceled

[AkihiroSuda]

List of allowed syscalls:

arch_prctl
bind
brk
capget
capset
chdir
chown
close
dup2
epoll_create1
epoll_pwait
execve
exit
fchown
fcntl
fork
fstat
getdents64
getdents64
getegid
geteuid
getgid
getgid
getpid
getpid
getppid
getppid
getuid
getuid
ioctl
listen
lseek
mkdir
mmap
mprotect
newfstatat
open
openat
prctl
pread
prlimit64
pwrite
read
rt_sigaction
rt_sigprocmask
sched_getaffinity
setgid
setgroups
setsockopt
set_tid_address
setuid
socket
socketpair
stat
uname

After adding futex and write to the profile manually, the runc container starts up, but nginx fails

$ docker run -it --rm -p 80:80 --security-opt seccomp=nginx-seccomp.json nginx:alpine
2019/08/14 15:57:32 [crit] 1#-1: pread() "/etc/nginx/nginx.conf" failed (1: Operation not permitted)
nginx: [crit] pread() "/etc/nginx/nginx.conf" failed (1: Operation not permitted)

[kcq]

@AkihiroSuda thank you for double checking! Really appreciate your help with this!

[kcq]

The pread and pwrite calls are actually pread64 and pwrite64. The latest version fixes these naming problems along with a number of other related enhancements. Still need to add a few more enhancements though...

[AkihiroSuda]

do you have plan for the next binary release?

[kcq]

Got the main enhancements in place... Works pretty well with the standard and alpine nginx images :-) Need to cleanup a few side effects and it's ready for the release...

[AkihiroSuda]

Thanks, the latest revision (bc29b8cd2f77b43111cdb918a1be73291c6d041d) now produces a correct seccomp profile for nginx:1.17.3-alpine, but AppArmor profile seems broken

$ docker run -it --rm -p 80:80 --security-opt seccomp=.docker-slim-state/images/d87c83ec7a667bef183c7c501dd724783222da1a636be9b6d749f878c284281a/artifacts/nginx-seccomp.json --security-opt apparmor=nginx-apparmor-profile nginx:1.17.3-alpine
Error loading shared library libpcre.so.1: Operation not permitted (needed by /usr/sbin/nginx)
Error loading shared library libssl.so.1.1: Operation not permitted (needed by /usr/sbin/nginx)
Error loading shared library libcrypto.so.1.1: Operation not permitted (needed by /usr/sbin/nginx)
Error loading shared library libz.so.1: Operation not permitted (needed by /usr/sbin/nginx)
Error relocating /usr/sbin/nginx: TLS_method: symbol not found
Error relocating /usr/sbin/nginx: OCSP_response_get1_basic: symbol not found
...
Error relocating /usr/sbin/nginx: SSL_CTX_set_verify: symbol not found
Error relocating /usr/sbin/nginx: pcre_free_study: symbol not found
Error relocating /usr/sbin/nginx: OCSP_RESPONSE_new: symbol not found
Error relocating /usr/sbin/nginx: d2i_OCSP_RESPONSE: symbol not found
Error relocating /usr/sbin/nginx: X509_free: symbol not found
Error relocating /usr/sbin/nginx: pcre_free: symbol not found
Error relocating /usr/sbin/nginx: pcre_malloc: symbol not found

generated profile:

profile nginx-apparmor-profile flags=(attach_disconnected,mediate_deleted) {

  network,

  /usr/sbin/nginx rix,

  /run/nginx.pid w,

  /etc/group r,
  /etc/nginx/conf.d/default.conf r,
  /etc/nginx/mime.types r,
  /etc/nginx/nginx.conf r,
  /etc/passwd r,
  /etc/ssl/openssl.cnf r,
  /lib/ld-musl-x86_64.so.1 r,
  /lib/libc.musl-x86_64.so.1 r,
  /lib/libcrypto.so.1.1 r,
  /lib/libssl.so.1.1 r,
  /lib/libz.so.1 r,
  /lib/libz.so.1.2.11 r,
  /usr/lib/libcrypto.so.1.1 r,
  /usr/lib/libpcre.so.1 r,
  /usr/lib/libpcre.so.1.2.11 r,
  /usr/lib/libssl.so.1.1 r,
  /usr/share/nginx/html/index.html r,
  /var/run r,

}

[szuecs]

With version 1.25.3, Go http proxy application and docker image:

% docker run -it --security-opt seccomp:skipper-seccomp.json 
registry.opensource.zalan.do/pathfinder/skipper:v0.10.293 skipper
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "close exec fds: lstat /proc/self/fd/0: operation not permitted": unknown.

[kcq]

@AkihiroSuda and everybody who stumbles on this issue trying to figure out how to have usable seccomp profiles :-) The latest version ( 1.32.0 ) finally got the necessary (multi-process/multi-threading) enhancements to make the generated seccomp profiles usable for more applications. I also updated the examples repo to have more examples with seccomp profiles (e.g., https://github.com/docker-slim/examples/tree/master/3rdparty/nginx_alpine , https://github.com/docker-slim/examples/tree/master/3rdparty/carbon-now-sh , https://github.com/docker-slim/examples/tree/master/3rdparty/codimd , https://github.com/docker-slim/examples/tree/master/node12_fastify_swagger and anywhere you see run_fat_seccomp.sh and run_slim_seccomp.sh)

The AppArmor profiles need a few more cycles to make them usable... We'll definitely get there sooner especially when I get more help working on the core engine :-)