criu dump: /dev/tty<N> devices created by bind-mounting /dev/pts/<M> devices

[brauner]

I'm trying to fix a bunch of long-standing CRIU issues in LXC and I'm starting with dumping tty devices.

When launching LXC containers user can specify how many ttys they want via lxc.tty.max. For example lxc.tty.max = 4 would mean that LXC allocates 4 tty devices:

/dev/tty1
/dev/tty2
/dev/tty3
/dev/tty4/

These tty devices are created from the container's devpts instance. Specifically, when the container is started it will mount devpts at /dev/pts and also create a bind-mount of /dev/pts/ptmx to /dev/ptmx to please AppArmor's symlink handling. The LXC will allocate 4 pty devices via opentty(). The container's devpts instance will thus show:

/dev/pts/0
/dev/pts/1
/dev/pts/2
/dev/pts/3

The file descriptors for these devices are sent by the container to the LXC monitor process, i.e. the parent of the container. The monitor process stashes these file descriptors for later usage.

The container now proceeds to bind-mount the /dev/pts/* devices onto /dev/tty* which LXC has created as dummy bind-mount targets with permission 0000. The mount table thus looks like this:

root@f8:~# findmnt | grep tty.
│ ├─/dev/tty1                         devpts[/0]                            devpts      rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024
│ ├─/dev/tty2                         devpts[/1]                            devpts      rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024
│ ├─/dev/tty3                         devpts[/2]                            devpts      rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024
│ ├─/dev/tty4                         devpts[/3]                            devpts      rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024

When I now dump a container criu will fails with:

(00.248384) Error (criu/tty.c:2430): tty: Unable to find a master for /0

The command line to dump the container is:

lxc-checkpoint b2 20201125133018.850 INFO     criu - criu.c:exec_criu:644 - execing: /usr/sbin/criu dump --tcp-established --file-locks --link-remap --manage-cgroups=full --ext-mount-map auto --enable-external-sharing --enable-external-masters --enable-fs hugetlbfs --enable-fs tracefs -D /mnt -o /mnt/dump.log --cgroup-root :/lxc.payload.b2 --cgroup-root name=systemd:/lxc.payload.b2 --cgroup-root freezer:/lxc.payload.b2 --cgroup-root rdma:/lxc.payload.b2 --cgroup-root devices:/lxc.payload.b2 --cgroup-root net_cls,net_prio:/lxc.payload.b2 --cgroup-root cpuset:/lxc.payload.b2 --cgroup-root hugetlb:/lxc.payload.b2 --cgroup-root memory:/lxc.payload.b2 --cgroup-root perf_event:/lxc.payload.b2 --cgroup-root cpu,cpuacct:/lxc.payload.b2 --cgroup-root blkio:/lxc.payload.b2 --cgroup-root pids:/lxc.payload.b2 -v4 --ext-mount-map /sys/fs/fuse/connections:sys/fs/fuse/connections --ext-mount-map /sys/kernel/debug:sys/kernel/debug --ext-mount-map /sys/kernel/security:sys/kernel/security --ext-mount-map /sys/fs/pstore:sys/fs/pstore --ext-mount-map /sys/firmware/efi/efivars:sys/firmware/efi/efivars --ext-mount-map /proc/sys/fs/binfmt_misc:proc/sys/fs/binfmt_misc -t 2287563 --skip-in-flight --freeze-cgroup /sys/fs/cgroup/freezer///lxc.payload.b2 --ext-mount-map /dev/console:console --external tty[881f:18]  --force-irmap --leave-running 

Note that the --ext-mount-map /dev/console:console --external tty[881f:18] line is for /dev/console which is a pty device bind-mounted from the host, i.e. from the host's devpts instance not the containers!

My question is, how do I handle these /dev/tty* devices I just described above correctly during dump (and also in restore for that matter). I'm happy to patch criu if that's required.

[adrianreber]

@avagin The error message is from code you wrote (a long time ago). Do you have any ideas how to make this work?

[rst0git]

Hi @brauner, a similar problem was resolved in runc by using the orphan-pts-master option (https://github.com/opencontainers/runc/commit/1c43d091a18a2f2dd73b46d49c678c265cf1ace3). However, this option is available only via RPC, but it can be used with libcriu (https://github.com/checkpoint-restore/criu/commit/83be11f1f4a24d05814a66061e04b9b4d0825763, https://github.com/containers/crun/commit/a15a72428571debaf6cb87c1fb318f4442d96602).

[brauner]

On Mon, Nov 30, 2020 at 01:24:45PM -0800, Radostin Stoyanov wrote:

Hi @brauner, a similar problem was resolved in runc by using the orphan-pts-master option (https://github.com/opencontainers/runc/commit/1c43d091a18a2f2dd73b46d49c678c265cf1ace3). However, this option is available only via RPC, but it can be used

Ok, if that fixes it it would be great if we could also export this as a command line option.

Thanks! Christian

[rppt]

On Tue, Dec 01, 2020 at 02:11:34AM -0800, Christian Brauner wrote:

On Mon, Nov 30, 2020 at 01:24:45PM -0800, Radostin Stoyanov wrote:

Hi @brauner, a similar problem was resolved in runc by using the orphan-pts-master option (https://github.com/opencontainers/runc/commit/ 1c43d091a18a2f2dd73b46d49c678c265cf1ace3). However, this option is available only via RPC, but it can be used

Ok, if that fixes it it would be great if we could also export this as a command line option.

You know how this works in open source, right? ;-) Patches are welcome :-)

Thanks! Christian

-- Sincerely yours, Mike.

[brauner]

On Tue, Dec 01, 2020 at 03:32:23AM -0800, rppt wrote:

On Tue, Dec 01, 2020 at 02:11:34AM -0800, Christian Brauner wrote:

On Mon, Nov 30, 2020 at 01:24:45PM -0800, Radostin Stoyanov wrote:

Hi @brauner, a similar problem was resolved in runc by using the orphan-pts-master option (https://github.com/opencontainers/runc/commit/ 1c43d091a18a2f2dd73b46d49c678c265cf1ace3). However, this option is available only via RPC, but it can be used

Ok, if that fixes it it would be great if we could also export this as a command line option.

You know how this works in open source, right? ;-) Patches are welcome :-)

You should read the last sentences in my original issue message above. ;)

[rppt]

On Tue, Dec 01, 2020 at 05:00:38AM -0800, Christian Brauner wrote:

On Tue, Dec 01, 2020 at 03:32:23AM -0800, rppt wrote:

On Tue, Dec 01, 2020 at 02:11:34AM -0800, Christian Brauner wrote:

On Mon, Nov 30, 2020 at 01:24:45PM -0800, Radostin Stoyanov wrote:

Hi @brauner, a similar problem was resolved in runc by using the orphan-pts-master option (https://github.com/opencontainers/runc/commit/ 1c43d091a18a2f2dd73b46d49c678c265cf1ace3). However, this option is available only via RPC, but it can be used

Ok, if that fixes it it would be great if we could also export this as a command line option.

You know how this works in open source, right? ;-) Patches are welcome :-)

You should read the last sentences in my original issue message above. ;)

tl'dr, sorry :)

Highly appreciated!

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[stgraber]

Any update on this one? As seen above, it's breaking basic operation for us in LXC/LXD.

[adrianreber]

Any update on this one? As seen above, it's breaking basic operation for us in LXC/LXD.

Unfortunately, I am not aware of any changes to solve this.