Docker run not working on RHEL 8

Nick2253 commented 3 years ago

[x] This is a bug report
[ ] This is a feature request
[x] I searched existing issues before opening this one

Expected behavior

When I start a docker container with docker run <container>, I expect the container to start.

Actual behavior

However, the container does not start. I'm greeted with some variation on the error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: read init-p: connection reset by peer: unknown.

I've tested this with centos:7, hello-world, and mediawiki containers.

Using the docker daemon in debug mode, I see the following errors:

ERRO[2021-03-06T17:15:21.442690132-08:00] stream copy error: reading from a closed fifo

Steps to reproduce the behavior

Install Docker on RHEL 8. Start docker. Try to start a container. Observe that it presents an error and does not start.

Importantly, this system is configured in compliance with the NIST and STIG OpenSCAP policies. I'm guessing that something in these configurations is causing a problem for Docker, but I'm unable to find any better logging/debugging in order to try to troubleshoot this issue. I'm also not extremely experienced with docker, so I know that doesn't help.

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:17:04 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:27 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 18
  Running: 0
  Paused: 0
  Stopped: 18
 Images: 3
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-240.15.1.el8_3.x86_64
 Operating System: Red Hat Enterprise Linux 8.3 (Ootpa)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.873GiB
 Name: <redacted>
 ID: <redacted>
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 25
  Goroutines: 36
  System Time: 2021-03-06T17:57:16.115748342-08:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

RHEL 8.3, running as a virtual machine inside of Hyper-V.

gabrieleara commented 3 years ago

Had a similar problem on Fedora 34. @Nick2253 have you tried using the following kernel argument: systemd.unified_cgroup_hierarchy=0 ? For me, adding it to my kernel arguments solved the issue.

You can set it using

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

subcan commented 2 years ago

@Nick2253 Were you ever able to figure out what STIG was impacting docker run? I am experiencing the same problem... Thanks

suyuyi commented 2 years ago

Had a similar problem on RHEL 8.5, and more weird. Cause it successes a few time to print "Hello World!", but the lot of times, it failed, and get the same error as above. @Nick2253 Have you solved this problem? Here is more detail about my problem

Nick2253 commented 2 years ago

@subcan Yes, sorry that I had not reported back earlier.

It was a combination of SELinux and fapolicyd. I'm familiar with SELinux, and knew enough before to setenforce 0, but fapolicyd is a STIG requirement, and prior to using a STIG'd version of RHEL 8, I had never even heard of fapolicyd.

fapolicyd is actually pretty easy to configure, but for our use case and risk factors, it was just easier to note an exception to fapolicyd than to configure it correctly.

suyuyi commented 2 years ago

@Nick2253 Thanks! I thought it's fapolicyd caused this error. After sudo systemctl stop fapolicyd, I can run hello-world image normally. But it's still confuse me, why a few time when fapolicyd was started, I can executed docker run hello-world successfully.

docker / for-linux