3.0.0, 2.1.1: rootless Podman cant run "expect" (The system has no more ptys.) [regression]

[onlyjob]

Since few releases ago (going several weeks back) I can't (re-)build a particular container image any more due to expect binary failing (in spawn command) as follows in rootless mode:

The system has no more ptys.  Ask your system administrator to create more.

(Formerly this container image was building fine by the older release of Podman).

expect(1) man page mentions the following:

Internally, spawn uses a pty, initialized the same way as the user's tty.

# ls -l /dev/pts
total 0
crw--w---- 1 root tty  136, 0 Feb  7 14:18 0
crw-rw-rw- 1 root root   5, 2 Feb  7 14:18 ptmx

I'm not sure whether this have something to do with kernel, runc, crun or other system components. I've tried with runc and crun but reproduced the problem with both of them. Podman 2.0.6 on Debian "testing"/"unstable" had no such problem.

Here is the current output of podman info:

host:                                                                                                                                                                           
  arch: amd64                                                                                                                                                                   
  buildahVersion: 1.19.3                                                                                                                                                        
  cgroupManager: systemd                                                                                                                                                        
  cgroupVersion: v2                                                                                                                                                             
  conmon:                                                                                                                                                                       
    package: 'conmon: /usr/bin/conmon'                                                                                                                                          
    path: /usr/bin/conmon                                                                                                                                                       
    version: 'conmon version 2.0.25, commit: unknown'                                                                                                                           
  cpus: 4                                                                                                                                                                       
  distribution:                                                                                                                                                                 
    distribution: debian                                                                                                                                                        
    version: unknown                                                                                                                                                            
  eventLogger: journald                                                                                                                                                         
  hostname: deblab                                                                                                                                                              
  idMappings:                                                                                                                                                                   
    gidmap:                                                                                                                                                                     
    - container_id: 0                                                                                                                                                           
      host_id: 1000                                                                                                                                                             
      size: 1                                                                                                                                                                   
    - container_id: 1                                                                                                                                                           
      host_id: 427680                                                                                                                                                           
      size: 65536                                                                                                                                                               
    uidmap:                                                                                                                                                                     
    - container_id: 0                                                                                                                                                           
      host_id: 1000                                                                                                                                                             
      size: 1                                                                                                                                                                   
    - container_id: 1                                                                                                                                                           
      host_id: 427680                                                                                                                                                           
      size: 65536                                                                                                                                                               
  kernel: 5.10.0-2-amd64                                                                                                                                                        
  linkmode: dynamic                                                                                                                                                             
  memFree: 1301323776                                                                                                                                                           
  memTotal: 67363360768                                                                                                                                                         
  ociRuntime:                                                                                                                                                                   
    name: crun                                                                                                                                                                  
    package: 'crun: /usr/bin/crun'                                                                                                                                              
    path: /usr/bin/crun                                                                                                                                                         
    version: |-                                                                                                                                                                 
      crun version 0.17                                                                                                                                                         
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a                                                                                                                          
      spec: 1.0.0                                                                                                                                                               
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL                                                                                                                     
  os: linux                                                                                                                                                                     
  remoteSocket:                                                                                                                                                                 
    exists: true                                                                                                                                                                
    path: /run/user/1000/podman/podman.sock                                                                                                                                     
  security:                                                                                                                                                                     
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 15508434944
  swapTotal: 17495486464
  uptime: 14h 13m 26.36s (Approximately 0.58 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
store:
  configFile: ~/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.10.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.10.1
        using FUSE kernel interface version 7.31
  graphRoot: ~/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 15
  runRoot: /run/user/1000
  volumePath: ~/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.0.0-rc2

CC: @siretart.

[mheon]

Can you provide additional details on how you're launching Podman - an exact command line to reproduce would be greatly helpful.

[onlyjob]

Something like podman build --no-cache --rm --force-rm -t centos7-app .

Build script invokes expect script that fails... It is a straightforward script:

#!/usr/bin/expect

set timeout 99
stty columns 80 rows 25

spawn bash /var/tmp/appsetup-linux.sh 

sleep .5
expect    "Please enter the fully qualified name, including domain information, of this host machine*" {
    send "localhost\n"
}

interact

It is invoked as runuser -u appuser /var/tmp/appsetup-linux.expect during container build.

[mheon]

@TomSweeneyRedHat PTAL

[rhatdan]

@onlyjob Could you try this against buildah in both rootful and rootless mode. Also have you tried this against the podman 3.0 rc?

[onlyjob]

In podman 3.0.0~rc2 (mentioned in title) it appears to work under root. I'll check if buildah is affected and report...

[onlyjob]

Reproduced the problem in Podman_3.0.0 and Buildah_1.19.4, both rootless.

[rhatdan]

If you run that command in your user session, non root, do you run out of the ptys as well?

[onlyjob]

If you run that command in your user session, non root, do you run out of the ptys as well?

Apparently not... expect don't complain under podman run -it (rootless)...

[rhatdan]

Are you saying it only runs out if you don't use -i?

[onlyjob]

It runs fine with run -it but fails during build -t stage.

[rhatdan]

On root it runs fine in either mode. @giuseppe Thoughts?

[rhatdan]

Basically -i keeps stdin open. But it might do something with the tty.

You could so try this with crun and runc, to see if they react any differently.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@onlyjob I never heard back on the previous question. Please respond.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[onlyjob]

You could so try this with crun and runc, to see if they react any differently.

I did not have a chance to try that. The problem manifested with crun. Did you have a chance to reproduce?

[rhatdan]

Have not been able to reproduce, please try it against current code, and /or generate a reproducer we can test against.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[onlyjob]

Just tried again on 3.0.1, still the same problem. :(

[rhatdan]

Can you contribute a Containerfile that shows the failure?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since we have had no feedback in a month, I am going to close. Reopen if you have the feedback.

[onlyjob]

I can only comment but not reopen... I'm sorry that I could not provide a perfect reproducer (no time!) but this bug report is complete with everything that is needed for replicating the issue.

[bauen1]

Minimal reproducer:

FROM debian:unstable

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        expect \
    && rm -rf /var/lib/apt/lists/*

RUN expect -c 'spawn echo 1'

Build with:

podman build --network host -f test9_dockerfile someemptydir

Expected result:

Does not print:

STEP 3/3: RUN expect -c 'spawn echo 1'
spawn echo 1
The system has no more ptys.  Ask your system administrator to create more.
    while executing
"spawn echo 1"

relevant strace excerpt from a similiar but different setup:

write(1, "spawn", 5spawn)                    = 5
write(1, " ", 1 )                        = 1
write(1, "1", 11)                        = 1
write(1, "\r\r\n", 3
)                   = 3
openat(AT_FDCWD, "/dev/ptmx", O_RDWR)   = 4
ioctl(4, TIOCGPTN, 0x7ffcfee9a1f4)      = -1 EACCES (Permission denied)
close(4)                                = 0
close(-1)                               = -1 EBADF (Bad file descriptor)
close(-1)                               = -1 EBADF (Bad file descriptor)
openat(AT_FDCWD, "/", O_RDONLY)         = 4
close(4)                                = 0
write(2, "The system has no more ptys.  As"..., 105The system has no more ptys.  Ask your system administrator to create more.
    while executing
"spawn 1") = 105

This might very well be a bug in expect, https://sources.debian.org/src/expect/5.45.4-2/exp_command.c/?hl=873#L873 is where the error message is printed.

[bauen1]

It seems that the code responsible is here https://sources.debian.org/src/expect/5.45.4-2/pty_termios.c/#L390

And a (horrible) C code reproducer:

#include <assert.h>
#include <stdio.h>
#include <fcntl.h>
#define __USE_XOPEN_EXTENDED 1
#include <stdlib.h>

int main(int argc, char *argv[]) {
    int master = open("/dev/ptmx", O_RDWR);
    if (master == -1) {
        return EXIT_FAILURE;
    }
    char * v = ptsname(master);
    printf("ptsname => '%s'\n", v);

    return EXIT_SUCCESS;
}

in podman:

openat(AT_FDCWD, "/dev/ptmx", O_RDWR)   = 3
ioctl(3, TIOCGPTN, 0x7ffc88b3bdfc)      = -1 EACCES (Permission denied)

outside:

openat(AT_FDCWD, "/dev/ptmx", O_RDWR)   = 3
ioctl(3, TIOCGPTN, [44])                = 0

[bauen1]

Please disregard the above, I managed to block the TIOCGPTN ioctl and didn't double check.

For some reason I was expecting SELinux to block with EPERM not EACCES, and I didn't double check first.

Sorry for the noise.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Sadly in a couple of years no one has picked this up.

[UNiXMIT]

That's a shame. I've hit this issue trying to build an Oracle database container using Podman.

[rhatdan]

Does it work with rootful podman?

[UNiXMIT]

Does it work with rootful podman?

For me it does yes.

[rhatdan]

My guess would be rootless users have a limited number of ptys or open files that you are hitting. Perhaps something in ulimit.

$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 256064
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 256064
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

[rhatdan]

These limits are the same

$ cat /proc/sys/kernel/pty/max 4096 $ sudo cat /proc/sys/kernel/pty/max 4096

But maybe root is able to ignore these since it has cap_sys_resource

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since I have not heard back, I am going to close. Continue the conversation here.