3.0.0rc2 regression: can't create containers - socket paths too long

[yrro]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I can't start new containers with 3.0.0rc2. Previously (<= 2.1.1) this was not a problem.

Steps to reproduce the issue:

$ podman run --rm -it docker.io/library/debian:10
Error: failed to connect to container's attach socket: /run/user/876099160/libpod/tmp/socket/8b370207625cd9ab38eaa5bb142ff3077ef65f307b4ae8e6b8ac3668d8f3d4ae/attach: no such file or directory

Describe the results you received:

The path /run/user/876099160/libpod/tmp/socket/8b370207625cd9ab38eaa5bb142ff3077ef65f307b4ae8e6b8ac3668d8f3d4ae/attach is 109 bytes long. This is longer than the maximum length of a unix socket path of 108 bytes.

Output of podman version:

$ podman version
Version:      3.0.0-rc2
API Version:  3.0.0
Go Version:   go1.15.8
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.3
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: unknown
  eventLogger: journald
  hostname: fragarach.ipa.example.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 876099160
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65537
    uidmap:
    - container_id: 0
      host_id: 876099160
      size: 1
    - container_id: 1
      host_id: 231073
      size: 65536
  kernel: 5.9.0-4-amd64
  linkmode: dynamic
  memFree: 3909976064
  memTotal: 33608015872
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/876099160/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 33397403648
  swapTotal: 34359734272
  uptime: 841h 19m 38.68s (Approximately 35.04 days)
registries:
  search:
  - quay.io
  - docker.io
store:
  configFile: /home/ipa.example.com/sam.morris/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.10.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.10.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/ipa.example.com/sam.morris/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 280
  runRoot: /run/user/876099160/containers
  volumePath: /home/ipa.example.com/sam.morris/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.8
  OsArch: linux/amd64
  Version: 3.0.0-rc2

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list -a podman
Listing... Done
podman/unstable,now 3.0.0~rc2+dfsg1-2+b1 amd64 [installed]
podman/testing 2.1.1+dfsg1-6 amd64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[yrro]

https://github.com/containers/podman/pull/1704 sounds similar but is quite old - hence this seems like a regression

[yrro]

With podman 2.2.1 the sockets are called:

/run/user/876099160/libpod/tmp/socket/237f67c9648f6c96ea1f945171266f0dc15add77dbb26f650ff833c706bb340a/atta

i.e. the socket path is truncated to 108 bytes (including the NUL terminating byte).

[yrro]

Probably actually a dupe of https://github.com/containers/podman/issues/8798

[yrro]

... maybe not. As far as I can tell, https://github.com/containers/podman/pull/8933/files is already present in 3.0.0-rc2.

Delving in with strace, I can see...

633674 openat(AT_FDCWD, "/run/user/876099160/libpod/tmp/socket/9d27a9c3b532d9bfc9687c1c98e4438e7e12f35a36a8e8cd1bbe62008ba4fe95/attach", O_RDONLY|O_PATH) = -1 ENOENT (No such file or directory)

... which probably corresponds to https://github.com/containers/podman/blob/master/libpod/oci_attach_linux.go#L32 so maybe whatever is supposed to create that socket is no longer able to do so.

[yrro]

If I create the container detached...

$ podman run -d --name=foo --rm -it docker.io/library/debian:10
bb0e59cd1cbeca78858afbe07c0ba6b09970d4a0da1a55b0f43fe4d08a27aa43

$ ls -F /run/user/876099160/libpod/tmp/socket/bb0e59cd1cbeca78858afbe07c0ba6b09970d4a0da1a55b0f43fe4d08a27aa43/
artifacts/  atta=  config.json  ctl|  ctr.log  shm/  winsz|

Note the socket is present but it's called atta like it was under Podman 2.2.1!

And this explains why I can't attach to it:

$ podman attach foo
Error: error attaching to container bb0e59cd1cbeca78858afbe07c0ba6b09970d4a0da1a55b0f43fe4d08a27aa43: failed to connect to container's attach socket: /run/user/876099160/libpod/tmp/socket/bb0e59cd1cbeca78858afbe07c0ba6b09970d4a0da1a55b0f43fe4d08a27aa43/attach: no such file or directory

i.e., the socket is created with a truncated path, and the attach operation fails because it uses the untruncated path.

[yrro]

Aha, if I upgrade conmon (from 2.20.20 to 2.20.25) then the attach socket is created with its full path and podman can now attach to it. Sorry for the noise!