The official Nginx container no longer works on Podman 2.0.0-dev

[jdoss]

/kind bug

Description

The official Nginx container no longer works on Podman 2.0.0-dev

Here is a test that shows it is working with Podman 1.9.3:

$ podman version
Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.14.2
OS/Arch:            linux/amd64

$ podman run --rm --name nginx -v tmp:/usr/share/nginx/html:ro nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

Here is a test that shows it is not working with 2.0.0-dev:

$ podman version
Version:      2.0.0-dev
API Version:  1
Go Version:   go1.14.3
Git Commit:   3f026eb6a682a68e69f9376b72157a8f084e575c
Built:        Fri Jun 12 11:04:21 2020
OS/Arch:      linux/amd64

$ podman run --rm --name nginx -v tmp:/usr/share/nginx/html:ro nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2020/06/12 16:05:12 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (13: Permission denied)

The Nginx container symlinks access.log and error.log to stdout and stderr respectively which is most likely the reason why it is not working with Podman 2.0.0-dev

RUN ln -sf /dev/stdout /var/log/nginx/access.log \
    && ln -sf /dev/stderr /var/log/nginx/error.log

[goochjj]

Works for me on photon. OS? Selinux? Any errors in dmesg or other logs?

podman run --rm -d --name nginx -v tmp:/usr/share/nginx/html:ro -p 8080:80 nginx curl -kv localhost:8080

podman logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
127.0.0.1 - - [12/Jun/2020:16:48:16 +0000] "GET / HTTP/1.1" 403 153 "-" "curl/7.61.1" "-"
2020/06/12 16:48:16 [error] 28#28: *1 "/usr/share/nginx/html/index.html" is forbidden (13: Permission denied), client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost:8080"

[jdoss]

Looks like it might be an issue with cgroups v2 maybe? This works just fine on Fedora CoreOS 32.20200601.1.1 which is running cgroups v1:

$ ssh core@192.168.122.179
Warning: Permanently added '192.168.122.179' (ECDSA) to the list of known hosts.
Fedora CoreOS 32.20200601.1.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

$ podman version
Version:      2.0.0-dev
API Version:  1
Go Version:   go1.14.3
Git Commit:   3f026eb6a682a68e69f9376b72157a8f084e575c
Built:        Fri Jun 12 16:04:21 2020
OS/Arch:      linux/amd64

$ podman run --rm -d --name nginx -v tmp:/usr/share/nginx/html:ro -p 8080:80 nginx
e082e020f62475d976487df7c69959143445fb6f61cfca2ab6758f9164d6bad0

$ curl -kv localhost:8080
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.69.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.19.0
< Date: Fri, 12 Jun 2020 17:09:15 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 26 May 2020 15:00:20 GMT
< Connection: keep-alive
< ETag: "5ecd2f04-264"
< Accept-Ranges: bytes
snip

On my Fedora 32 Workstation that has cgroups v2 is where it is erroring out:

$ podman run --rm --name nginx -v tmp:/usr/share/nginx/html:ro -p 8080:80 nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2020/06/12 17:12:07 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (13: Permission denied)

$ cat /etc/redhat-release 
Fedora release 32 (Thirty Two)

$ uname -r
5.6.16-300.fc32.x86_64

Journalctl entries:

Jun 12 12:13:53 sts7 systemd[6314]: Started libcrun container.
Jun 12 12:13:54 sts7 systemd[6314]: libpod-492b5256ec5cbfdcd320c3d3e67df12fa37ac3cce404dacefcf7bda0771d9217.scope: Succeeded.
Jun 12 12:13:54 sts7 kernel: Lockdown: pmdakvm: debugfs access is restricted; see man kernel_lockdown.7

There isn't anything in dmesg besides:

[Fri Jun 12 12:13:53 2020] Lockdown: pmdakvm: debugfs access is restricted; see man kernel_lockdown.7

[rhatdan]

@giuseppe PTAL

[giuseppe]

it reminds me of https://github.com/containers/conmon/pull/112

What version of conmon are you using?

[jdoss]

@giuseppe here ya go:

$ rpm -qa conmon
conmon-2.0.17-1.fc32.x86_64

[giuseppe]

can you show me the output for:

$ which podman
$ ls -lZ $(which podman)

and could you try to temporarily disable selinux setenforce 0? Does it make any difference?

[jdoss]

@giuseppe Here ya go. Looks like SELinux. The weird part is on Fedora CoreOS I am putting it in the same location and ignition seems to give the newer podman binary the correct context.

$ podman version
Version:      2.0.0-dev
API Version:  1
Go Version:   go1.14.3
Git Commit:   3f026eb6a682a68e69f9376b72157a8f084e575c
Built:        Fri Jun 12 11:04:21 2020
OS/Arch:      linux/amd64

$ ls -lZ $(which podman)
-rwxr-xr-x. 1 root root unconfined_u:object_r:bin_t:s0 53936800 Jun 12 11:04 /usr/local/bin/podman
$ podman run --rm --name nginx -v tmp:/usr/share/nginx/html:ro nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2020/06/16 14:01:33 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (13: Permission denied)

$ sudo setenforce 0
$ podman run --rm --name nginx -v tmp:/usr/share/nginx/html:ro nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

I just installed v2.0.0-rc6 from https://koji.fedoraproject.org/koji/buildinfo?buildID=1523972 and it works fine.

$ sudo setenforce 1
$ podman version
Version:      2.0.0-rc6
API Version:  1
Go Version:   go1.14.3
Built:        Wed Dec 31 18:00:00 1969
OS/Arch:      linux/amd64
$ ls -lZ $(which podman)
-rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 52625408 Jun 15 13:02 /usr/bin/podman
$ podman run --rm --name nginx -v tmp:/usr/share/nginx/html:ro nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

[giuseppe]

thanks for checking it.

I am closing the issue

[rhatdan]

@jdoss What AVCs are you seeing?

[jdoss]

@rhatdan Here ya go:

----
time->Tue Jun 16 09:02:03 2020
type=AVC msg=audit(1592316123.154:15011): avc:  denied  { associate } for  pid=959643 comm="nginx" name="2" scontext=system_u:object_r:container_t:s0:c582,c914 tcontext=system_u:object_r:proc_t:s0 tclass=filesystem permissive=1

[rhatdan]

Thanks that is a weird AVC. Anyways since this now works in enforcing mode, I am happy.

[Meister1593]

I'm on Fedora Kinoite 35.20220411.0 and i have exactly the same issue on podman 3.4.4 with podman-compose 1.0.3

Setting sudo setenforce 0 fixes the issue

I tried to mount that folders like so:

version: '3'
services:
    nginx:
        image: nginx:latest
        ports:
            - "8080:80"
            - "4433:443"
        volumes:
            - ./hosts:/etc/nginx/conf.d
            - ./html:/usr/share/nginx/html
            - ./logs/nginx:/var/log/nginx

With just podman : podman run --rm --name nginx -v ./logs/nginx:/var/log/nginx:rw nginx it doesnt work too

EDIT: Nevermind, it worked with podman run --rm --name nginx -v ./logs/nginx:/var/log/nginx:Z,U nginx, didn't know about SELinux issues, but didn't work with podman-compose so i will move issue into compose

[rhatdan]

You need a Z on an SELinux system, this looks like it is unrelated to orginal issue.

podman run --rm --name nginx -v ./logs/nginx:/var/log/nginx:rw,Z nginx

[courtarro]

podman run --rm --name nginx -v ./logs/nginx:/var/log/nginx:rw:Z nginx

@rhatdan For anyone blindly following this info, it's actually :rw,Z if you want to combine options. The extra colon will break the syntax.

[rhatdan]

Thanks I fixed my original comment.