Permission denied for overwriting files in /tmp

[siretart]

/kind bug

Description

siretart@x1:/tmp$ podman run --rm -it docker.io/alpine
Trying to pull docker.io/alpine...
Copying config a24bb40132 done  
Writing manifest to image destination
Storing signatures
/ # touch /tmp/bar
/ # chown nobody:nobody /tmp/bar
/ # echo foo > /tmp/bar
/bin/sh: can't create /tmp/bar: Permission denied
/ # 

This issue is reproducible with Debian and alpine images, but not with ubi7 or ubi8 images. What's different about them?

Output of podman version:

siretart@x1:/tmp$ podman version
Version:      2.0.4
API Version:  1
Go Version:   go1.14.7
Built:        Wed Dec 31 19:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

siretart@x1:/tmp$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: unknown
  eventLogger: file
  hostname: x1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.0-2-amd64
  linkmode: dynamic
  memFree: 2669056000
  memTotal: 16615301120
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.1
  swapFree: 15970406400
  swapTotal: 15997071360
  uptime: 558h 50m 31.51s (Approximately 23.25 days)
registries:
  search:
  - quay.io
  - docker.io
store:
  configFile: /home/siretart/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.2
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.2
        using FUSE kernel interface version 7.31
  graphRoot: /home/siretart/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 6
  runRoot: /run/user/1000/containers
  volumePath: /home/siretart/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.0.4

Package info (e.g. output of rpm -q podman or apt list podman):

siretart@x1:/tmp$ apt list podman
Listing... Done
podman/testing,now 2.0.4+dfsg2-2 amd64 [installed]

[edsantiago]

It's the sticky bit in /tmp. Removing it makes everything work fine. strace is not especially helpful:

/ # strace tee -a /tmp/foo
execve("/usr/bin/tee", ["tee", "-a", "/tmp/foo"], 0x7ffc942827e0 /* 7 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7fac360e8d48) = 0
set_tid_address(0x7fac360e931c)         = 11
mprotect(0x7fac360e5000, 4096, PROT_READ) = 0
mprotect(0x55656f6db000, 16384, PROT_READ) = 0
getuid()                                = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fac3609928e}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
open("/tmp/foo", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 EACCES (Permission denied)
write(2, "tee: /tmp/foo: Permission denied"..., 33tee: /tmp/foo: Permission denied
) = 33

Nothing obvious wrt capabilities:

/ # grep Cap /proc/1/status
CapInh: 0000003fffffffff
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000003fffffffff

[asottile]

this also happens outside of podman so I don't think this is a bug:

root@babibox:/tmp# touch wat
root@babibox:/tmp# chown nobody wat
root@babibox:/tmp# echo hi > wat
-bash: wat: Permission denied

I believe it's due to the sticky bit on /tmp which changes how permissions work slightly (only the owning user can write to the file)

[siretart]

I've investigated this a bit further, and with the help of #debian-devel, we believe the error doesn't happen on distros that compile bash with the option --with-afs.

On Debian, the kernel sets /proc/sys/fs/protected_regular to 2 as a hardening feature (cf. https://www.kernel.org/doc/Documentation/sysctl/fs.txt). However, it seems that Fedora/RHEL based distributions defeat this protection. I've posted some strace outputs here: https://gist.github.com/siretart/130920831016cbba9df89e871c0948de -- the relevant code part in bash can be seen here: https://sources.debian.org/src/bash/5.0-7/redir.c/?hl=694#L691

I don't think there is any code change necessary in podman.

[edsantiago]

Interesting: I did confirm earlier that the problem doesn't manifest on Fedora (32):

# echo hi >/tmp/foo
#

This is indeed bash. However, something else that tries to open() the file fails:

# tee -a /tmp/foo
tee: /tmp/foo: Permission denied

So indeed, there's probably not much podman can do about it. @siretart thank you for the report and the followup. I'm closing, please reopen if new information arises.