search

checkpoint-restore / criu

Checkpoint/Restore tool
criu.org
Other
2.94k stars 590 forks source link

"Ghost files" and read-only filesystems? "Can't create ghost regfile" #429

Closed erulabs closed 2 years ago

erulabs commented 6 years ago

Hello!

I'm having an issue restoring checkpoints that require ghost files on a read-only filesystem to be "remapped". A ghost file is created (from restore.log):

Opening ghost file 0x1 for app/bcrypt/lib/binding/bcrypt_lib.node

And then fails when unlink fails:

(00.396152) Running post-setup-namespaces scripts
(00.396161)     RPC
(00.396305)      1: Preparing info about shared resources
(00.396323)      1: Configuring remap 0xc -> 0x1
(00.396758)      1: Error (criu/files-reg.c:358): Can't create ghost regfile: Read-only file system
(00.397233) Error (criu/cr-restore.c:1301): 31708 killed by signal 127: Unknown signal 127
(00.461438) mnt: Switching to new ns to clean ghosts
(00.461550) Unlink remap app/bcrypt/lib/binding/bcrypt_lib.node.cr.1.ghost
(00.462950) Error (criu/files-reg.c:680): Couldn't unlink remap 10 app/bcrypt/lib/binding/bcrypt_lib.node.cr.1.ghost: Read-only file system
(00.481359) Error (criu/cr-restore.c:2171): Restoring FAILED.

I'm diving into CRIU and would appreciate any hints, but Im wondering why this file in particular has to be "remapped", and what I can do to prevent that. Is CRIU snapshotting a process which has a filehandle open to a read-only filesystem supported? I'm trying to understand what about this file causes criu to consider it a ghost file. Would love some help in understanding this behavior.

I believe the problem is that the "Ghost isnt found": https://github.com/checkpoint-restore/criu/blob/0509bc9f177bc38a574c274452dc106b95741ee9/criu/files-reg.c#L385-L393 for this particular file. I suspect this is my real issue?

I've tested this on CRIU v3.6, kernel 4.4 and 4.10, docker version 17.03.1-ce and 17.03.2-ce.

The error returned by Docker server:

oci runtime error: criu failed: type NOTIFY errno 0

The CRIU error:

*** Error in `criu': free(): invalid pointer: 0x00007f2dfd5c0bf0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f2dfc3c97e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fe0a)[0x7f2dfc3d1e0a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f2dfc3d598c]
criu(prepare_remaps+0x2c2)[0x447752]
criu[0x43f9a1]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f2dfc45882d]
======= Memory map: ========
00400000-00512000 r-xp 00000000 fc:00 801539                             /usr/local/sbin/criu
00711000-00712000 r--p 00111000 fc:00 801539                             /usr/local/sbin/criu
00712000-00718000 rw-p 00112000 fc:00 801539                             /usr/local/sbin/criu
00718000-00728000 rw-p 00000000 00:00 0
012f2000-01313000 rw-p 00000000 00:00 0                                  [heap]
7f2df4000000-7f2df4021000 rw-p 00000000 00:00 0
7f2df4021000-7f2df8000000 ---p 00000000 00:00 0
7f2dfbbc3000-7f2dfbbd9000 r-xp 00000000 fc:00 1570327                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f2dfbbd9000-7f2dfbdd8000 ---p 00016000 fc:00 1570327                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f2dfbdd8000-7f2dfbdd9000 rw-p 00015000 fc:00 1570327                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f2dfbdd9000-7f2dfbee1000 r-xp 00000000 fc:00 1576143                    /lib/x86_64-linux-gnu/libm-2.23.so
7f2dfbee1000-7f2dfc0e0000 ---p 00108000 fc:00 1576143                    /lib/x86_64-linux-gnu/libm-2.23.so
7f2dfc0e0000-7f2dfc0e1000 r--p 00107000 fc:00 1576143                    /lib/x86_64-linux-gnu/libm-2.23.so
7f2dfc0e1000-7f2dfc0e2000 rw-p 00108000 fc:00 1576143                    /lib/x86_64-linux-gnu/libm-2.23.so
7f2dfc0e2000-7f2dfc150000 r-xp 00000000 fc:00 1570381                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f2dfc150000-7f2dfc350000 ---p 0006e000 fc:00 1570381                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f2dfc350000-7f2dfc351000 r--p 0006e000 fc:00 1570381                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f2dfc351000-7f2dfc352000 rw-p 0006f000 fc:00 1570381                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f2dfc352000-7f2dfc511000 r-xp 00000000 fc:00 1576138                    /lib/x86_64-linux-gnu/libc-2.23.so
7f2dfc511000-7f2dfc711000 ---p 001bf000 fc:00 1576138                    /lib/x86_64-linux-gnu/libc-2.23.so
7f2dfc711000-7f2dfc715000 r--p 001bf000 fc:00 1576138                    /lib/x86_64-linux-gnu/libc-2.23.so
7f2dfc715000-7f2dfc717000 rw-p 001c3000 fc:00 1576138                    /lib/x86_64-linux-gnu/libc-2.23.so
7f2dfc717000-7f2dfc71b000 rw-p 00000000 00:00 0
7f2dfc71b000-7f2dfc731000 r-xp 00000000 fc:00 799059                     /usr/lib/x86_64-linux-gnu/libnet.so.1.7.0
7f2dfc731000-7f2dfc930000 ---p 00016000 fc:00 799059                     /usr/lib/x86_64-linux-gnu/libnet.so.1.7.0
7f2dfc930000-7f2dfc931000 r--p 00015000 fc:00 799059                     /usr/lib/x86_64-linux-gnu/libnet.so.1.7.0
7f2dfc931000-7f2dfc932000 rw-p 00016000 fc:00 799059                     /usr/lib/x86_64-linux-gnu/libnet.so.1.7.0
7f2dfc932000-7f2dfc934000 rw-p 00000000 00:00 0
7f2dfc934000-7f2dfc951000 r-xp 00000000 fc:00 1570611                    /lib/x86_64-linux-gnu/libnl-3.so.200.22.0
7f2dfc951000-7f2dfcb50000 ---p 0001d000 fc:00 1570611                    /lib/x86_64-linux-gnu/libnl-3.so.200.22.0
7f2dfcb50000-7f2dfcb52000 r--p 0001c000 fc:00 1570611                    /lib/x86_64-linux-gnu/libnl-3.so.200.22.0
7f2dfcb52000-7f2dfcb53000 rw-p 0001e000 fc:00 1570611                    /lib/x86_64-linux-gnu/libnl-3.so.200.22.0
7f2dfcb53000-7f2dfcb56000 r-xp 00000000 fc:00 1576137                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f2dfcb56000-7f2dfcd55000 ---p 00003000 fc:00 1576137                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f2dfcd55000-7f2dfcd56000 r--p 00002000 fc:00 1576137                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f2dfcd56000-7f2dfcd57000 rw-p 00003000 fc:00 1576137                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f2dfcd57000-7f2dfcd5f000 r-xp 00000000 fc:00 799205                     /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.1.0.0
7f2dfcd5f000-7f2dfcf5e000 ---p 00008000 fc:00 799205                     /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.1.0.0
7f2dfcf5e000-7f2dfcf5f000 r--p 00007000 fc:00 799205                     /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.1.0.0
7f2dfcf5f000-7f2dfcf60000 rw-p 00008000 fc:00 799205                     /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.1.0.0
7f2dfcf60000-7f2dfcf78000 r-xp 00000000 fc:00 1576144                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f2dfcf78000-7f2dfd177000 ---p 00018000 fc:00 1576144                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f2dfd177000-7f2dfd178000 r--p 00017000 fc:00 1576144                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f2dfd178000-7f2dfd179000 rw-p 00018000 fc:00 1576144                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f2dfd179000-7f2dfd17d000 rw-p 00000000 00:00 0
7f2dfd17d000-7f2dfd19c000 r-xp 00000000 fc:00 1570398                    /lib/x86_64-linux-gnu/libselinux.so.1
7f2dfd19c000-7f2dfd39b000 ---p 0001f000 fc:00 1570398                    /lib/x86_64-linux-gnu/libselinux.so.1
7f2dfd39b000-7f2dfd39c000 r--p 0001e000 fc:00 1570398                    /lib/x86_64-linux-gnu/libselinux.so.1
7f2dfd39c000-7f2dfd39d000 rw-p 0001f000 fc:00 1570398                    /lib/x86_64-linux-gnu/libselinux.so.1
7f2dfd39d000-7f2dfd39f000 rw-p 00000000 00:00 0
7f2dfd39f000-7f2dfd3c5000 r-xp 00000000 fc:00 1576127                    /lib/x86_64-linux-gnu/ld-2.23.so
7f2dfd5a5000-7f2dfd5bb000 rw-p 00000000 00:00 0
7f2dfd5bd000-7f2dfd5be000 rw-p 00000000 00:00 0
7f2dfd5be000-7f2dfd5c0000 rw-s 00000000 00:05 1221136                    /dev/zero (deleted)
7f2dfd5c0000-7f2dfd5c2000 rw-s 00000000 00:05 1221134                    /dev/zero (deleted)
7f2dfd5c2000-7f2dfd5c4000 rw-p 00000000 00:00 0
7f2dfd5c4000-7f2dfd5c5000 r--p 00025000 fc:00 1576127                    /lib/x86_64-linux-gnu/ld-2.23.so
7f2dfd5c5000-7f2dfd5c6000 rw-p 00026000 fc:00 1576127                    /lib/x86_64-linux-gnu/ld-2.23.so
7f2dfd5c6000-7f2dfd5c7000 rw-p 00000000 00:00 0
7fff521dc000-7fff521fd000 rw-p 00000000 00:00 0                          [stack]
7fff5233a000-7fff5233c000 r--p 00000000 00:00 0                          [vvar]
7fff5233c000-7fff5233e000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
criu failed: type NOTIFY errno 0
log file: /checkpoints/5690588e5b28a1aa5904aae4f3362520/707cb0d0abb571d2d46ae32410988bde78acf724.0/criu.work/restore-2017-12-17T01:06:32Z/restore.log

The full restore.log

(00.000346) cpu: fpu:1 fxsr:1 xsave:1
(00.000419) kernel pid_max=32768
(00.000423) Reading image tree
(00.000518) Add mnt ns 12 pid 1
(00.000528) pstree pid_max=1
(00.000536) Will restore in 6c020000 namespaces
(00.000543) NS mask to use 6c020000
(00.000545) Collecting 37/54 (flags 2)
(00.000600) Collected [usr/local/bin/node] ID 0x1
(00.000610) Collected [app/system/node_modules/evaluator-connection/build/Release/connection.node] ID 0x2
(00.000613) Collected [app/system/node_modules/object-description/build/Release/internal.node] ID 0x3
(00.000616) Collected [lib/x86_64-linux-gnu/libc-2.19.so] ID 0x4
(00.000622) Collected [lib/x86_64-linux-gnu/libpthread-2.19.so] ID 0x5
(00.000625) Collected [lib/x86_64-linux-gnu/libgcc_s.so.1] ID 0x6
(00.000627) Collected [lib/x86_64-linux-gnu/libm-2.19.so] ID 0x7
(00.000629) Collected [usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20] ID 0x8
(00.000632) Collected [lib/x86_64-linux-gnu/librt-2.19.so] ID 0x9
(00.000634) Collected [lib/x86_64-linux-gnu/libdl-2.19.so] ID 0xa
(00.000636) Collected [lib/x86_64-linux-gnu/ld-2.19.so] ID 0xb
(00.000638) Collected [app/available_modules/1513475394000/bcrypt/lib/binding/bcrypt_lib.node] ID 0xc
(00.000642) Collected pipe entry ID 0xd PIPE ID 0x267e1
(00.000646) Found id pipe:[157665] (fd 0) in inherit fd list
(00.000649) Collected pipe entry ID 0xe PIPE ID 0x267e2
(00.000652) Found id pipe:[157666] (fd 1) in inherit fd list
(00.000722) Collected pipe entry ID 0xf PIPE ID 0x267e3
(00.000729) Found id pipe:[157667] (fd 2) in inherit fd list
(00.000733) Collected pipe entry ID 0x10 PIPE ID 0x26ed8
(00.000737) Collected pipe entry ID 0x11 PIPE ID 0x26ed8
(00.000745) epoll: Collected eventpoll: id 0x000012 flags 0x80002
(00.000752) Collected pipe entry ID 0x13 PIPE ID 0x26ed9
(00.000756) Collected pipe entry ID 0x14 PIPE ID 0x26ed9
(00.000784) eventfd: Collected : id 0x000015 flags 0x80802 counter 0x00000000000028
(00.000791) Collected [dev/null] ID 0x16
(00.000806) Collected [app] ID 0x19
(00.000810) Collected [.] ID 0x1a
(00.000815)  `- ... done
(00.000818) Collecting 43/59 (flags 0)
(00.000826) Opening ghost file 0x1 for app/available_modules/1513475394000/bcrypt/lib/binding/bcrypt_lib.node
(00.000830)  `- ... done
(00.000940) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000955) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000959) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000963) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000966) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000969) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000972) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000976) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000980) cg: rewriting docker/1b57aed5fa5482559f4d38dc95fa2e0c548315fe9b8abf4adba9b63ad82d8f2c to /docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.000985) cg: Preparing cgroups yard (cgroups restore mode 0x4)
(00.005644) cg: Opening .criu.cgyard.bo6hNQ as cg yard
(00.005688) cg:     Making controller dir .criu.cgyard.bo6hNQ/blkio (blkio)
(00.005745) cg: Determined cgroup dir blkio//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.005762) cg: Skip restoring properties on cgroup dir blkio//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.005773) cg:     Making controller dir .criu.cgyard.bo6hNQ/hugetlb (hugetlb)
(00.006261) cg: Determined cgroup dir hugetlb//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.006267) cg: Skip restoring properties on cgroup dir hugetlb//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.006278) cg:     Making controller dir .criu.cgyard.bo6hNQ/memory (memory)
(00.007270) cg: Determined cgroup dir memory//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.007276) cg: Skip restoring properties on cgroup dir memory//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.007283) cg:     Making controller dir .criu.cgyard.bo6hNQ/devices (devices)
(00.007501) cg: Determined cgroup dir devices//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.007507) cg: Skip restoring properties on cgroup dir devices//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.007516) cg:     Making controller dir .criu.cgyard.bo6hNQ/pids (pids)
(00.012183) cg: Determined cgroup dir pids//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.012193) cg: Skip restoring properties on cgroup dir pids//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.012216) cg:     Making controller dir .criu.cgyard.bo6hNQ/perf_event (perf_event)
(00.012246) cg: Determined cgroup dir perf_event//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.012248) cg: Skip restoring properties on cgroup dir perf_event//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.012253) cg:     Making controller dir .criu.cgyard.bo6hNQ/cpuset (cpuset)
(00.012268) cg: Determined cgroup dir cpuset//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.012269) cg: Skip restoring properties on cgroup dir cpuset//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.012275) cg:     Making controller dir .criu.cgyard.bo6hNQ/net_cls,net_prio (net_cls,net_prio)
(00.012291) cg: Determined cgroup dir net_cls,net_prio/docker/96218d1e332afc18ae46618d2ec4c2229bf884c483bb840f8ee997c36c4ded61 already exist
(00.012292) cg: Skip restoring properties on cgroup dir net_cls,net_prio/docker/96218d1e332afc18ae46618d2ec4c2229bf884c483bb840f8ee997c36c4ded61
(00.012297) cg:     Making controller dir .criu.cgyard.bo6hNQ/freezer (freezer)
(00.012310) cg: Determined cgroup dir freezer//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.012312) cg: Skip restoring properties on cgroup dir freezer//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.012316) cg:     Making controller dir .criu.cgyard.bo6hNQ/cpu,cpuacct (cpu,cpuacct)
(00.012331) cg: Determined cgroup dir cpu,cpuacct/docker/96218d1e332afc18ae46618d2ec4c2229bf884c483bb840f8ee997c36c4ded61 already exist
(00.012332) cg: Skip restoring properties on cgroup dir cpu,cpuacct/docker/96218d1e332afc18ae46618d2ec4c2229bf884c483bb840f8ee997c36c4ded61
(00.012356) cg:     Making controller dir .criu.cgyard.bo6hNQ/systemd (none,name=systemd)
(00.012375) cg: Determined cgroup dir systemd//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959 already exist
(00.012377) cg: Skip restoring properties on cgroup dir systemd//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959
(00.012391) Running pre-restore scripts
(00.012393)     RPC
(00.012814) Saved netns fd for links restore
(00.013091) mnt: Reading mountpoint images (id 12 pid 1)
(00.013100) mnt:        Will mount 2033 from /null
(00.013103) mnt:        Will mount 2033 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug
(00.013105) mnt:    Read 2033 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug
(00.013107) mnt:        Will mount 2032 from /null
(00.013114) mnt:        Will mount 2032 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats
(00.013116) mnt:    Read 2032 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats
(00.013117) mnt:        Will mount 2031 from /null
(00.013119) mnt:        Will mount 2031 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list
(00.013120) mnt:    Read 2031 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list
(00.013121) mnt:        Will mount 2030 from /null
(00.013123) mnt:        Will mount 2030 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore
(00.013124) mnt:    Read 2030 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore
(00.013128) mnt:        Will mount 2029 from /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/hosts (E)
(00.013129) mnt:        Will mount 2029 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts
(00.013130) mnt:    Read 2029 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts
(00.013133) mnt:        Will mount 2028 from /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/hostname (E)
(00.013134) mnt:        Will mount 2028 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname
(00.013135) mnt:    Read 2028 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname
(00.013140) mnt:        Will mount 2027 from /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/resolv.conf (E)
(00.013142) mnt:        Will mount 2027 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf
(00.013143) mnt:    Read 2027 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf
(00.013145) mnt:        Will mount 2026 from /mnt/shared/module-fs-dev/8.x.x (E)
(00.013146) mnt:        Will mount 2026 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules
(00.013147) mnt:    Read 2026 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules
(00.013149) mnt:        Will mount 2025 from /sys/fs/cgroup/blkio/system.slice/evaldocker.service (E)
(00.013150) mnt:        Will mount 2025 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/blkio
(00.013151) mnt:    Read 2025 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/blkio
(00.013153) mnt:        Will mount 2024 from /sys/fs/cgroup/hugetlb (E)
(00.013155) mnt:        Will mount 2024 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/hugetlb
(00.013156) mnt:    Read 2024 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/hugetlb
(00.013158) mnt:        Will mount 2023 from /sys/fs/cgroup/memory/system.slice/evaldocker.service (E)
(00.013159) mnt:        Will mount 2023 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/memory
(00.013160) mnt:    Read 2023 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/memory
(00.013163) mnt:        Will mount 2022 from /sys/fs/cgroup/devices/system.slice/evaldocker.service (E)
(00.013164) mnt:        Will mount 2022 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/devices
(00.013165) mnt:    Read 2022 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/devices
(00.013167) mnt:        Will mount 2021 from /sys/fs/cgroup/pids/system.slice/evaldocker.service (E)
(00.013169) mnt:        Will mount 2021 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/pids
(00.013170) mnt:    Read 2021 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/pids
(00.013172) mnt:        Will mount 2020 from /sys/fs/cgroup/perf_event (E)
(00.013173) mnt:        Will mount 2020 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/perf_event
(00.013174) mnt:    Read 2020 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/perf_event
(00.013176) mnt:        Will mount 2019 from /sys/fs/cgroup/cpuset (E)
(00.013177) mnt:        Will mount 2019 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpuset
(00.013178) mnt:    Read 2019 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpuset
(00.013180) mnt:        Will mount 2018 from /sys/fs/cgroup/net_cls,net_prio (E)
(00.013181) mnt:        Will mount 2018 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/net_cls,net_prio
(00.013182) mnt:    Read 2018 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/net_cls,net_prio
(00.013184) mnt:        Will mount 2016 from /sys/fs/cgroup/freezer (E)
(00.013187) mnt:        Will mount 2016 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/freezer
(00.013188) mnt:    Read 2016 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/freezer
(00.013194) mnt:        Will mount 2014 from /sys/fs/cgroup/cpu,cpuacct/system.slice/evaldocker.service (E)
(00.013196) mnt:        Will mount 2014 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpu,cpuacct
(00.013197) mnt:    Read 2014 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpu,cpuacct
(00.013199) mnt:        Will mount 2013 from /sys/fs/cgroup/systemd/system.slice/evaldocker.service (E)
(00.013200) mnt:        Will mount 2013 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/systemd
(00.013201) mnt:    Read 2013 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/systemd
(00.013203) mnt:        Will mount 1283 from /
(00.013204) mnt:        Will mount 1283 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup
(00.013205) mnt:    Read 1283 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup
(00.013207) mnt:        Will mount 1282 from /
(00.013208) mnt:        Will mount 1282 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/firmware
(00.013209) mnt:    Read 1282 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/firmware
(00.013210) mnt:        Will mount 1281 from /
(00.013212) mnt:        Will mount 1281 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys
(00.013213) mnt:    Read 1281 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/sys
(00.013216) mnt:        Will mount 1280 from /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/shm (E)
(00.013217) mnt:        Will mount 1280 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/shm
(00.013218) mnt:    Read 1280 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/shm
(00.013220) mnt:        Will mount 1279 from /
(00.013221) mnt:        Will mount 1279 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/mqueue
(00.013222) mnt:    Read 1279 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/mqueue
(00.013224) mnt:        Will mount 1278 from /
(00.013225) mnt:        Will mount 1278 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/pts
(00.013226) mnt:    Read 1278 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/pts
(00.013227) mnt:        Will mount 1277 from /
(00.013228) mnt:        Will mount 1277 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev
(00.013230) mnt:    Read 1277 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/dev
(00.013231) mnt:        Will mount 1276 from /sysrq-trigger
(00.013232) mnt:        Will mount 1276 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger
(00.013233) mnt:    Read 1276 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger
(00.013236) mnt:        Will mount 1275 from /sys
(00.013237) mnt:        Will mount 1275 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys
(00.013239) mnt:    Read 1275 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys
(00.013240) mnt:        Will mount 1274 from /irq
(00.013241) mnt:        Will mount 1274 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq
(00.013242) mnt:    Read 1274 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq
(00.013244) mnt:        Will mount 1273 from /fs
(00.013245) mnt:        Will mount 1273 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs
(00.013246) mnt:    Read 1273 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs
(00.013248) mnt:        Will mount 1272 from /bus
(00.013249) mnt:        Will mount 1272 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus
(00.013250) mnt:    Read 1272 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus
(00.013251) mnt:        Will mount 1271 from /asound
(00.013252) mnt:        Will mount 1271 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound
(00.013254) mnt:    Read 1271 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound
(00.013255) mnt:        Will mount 1270 from /
(00.013256) mnt:        Will mount 1270 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc
(00.013257) mnt:    Read 1270 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/proc
(00.013264) mnt:        Will mount 1269 from /
(00.013265) mnt:        Will mount 1269 @ /tmp/.criu.mntns.s8pOlp/12-0000000000/
(00.013266) mnt:    Read 1269 mp @ /tmp/.criu.mntns.s8pOlp/12-0000000000/
(00.013301) Warn  (criu/cr-restore.c:1161): Set CLONE_PARENT | CLONE_NEWPID but it might cause restore problem,because not all kernels support such clone flags combinations!
(00.013305) Forking task with 1 pid (flags 0x6c028000)
(00.025788) PID: real 31708 virt 1
(00.025893) Wait until namespaces are created
(00.162050)      1: Found fd 0 (id pipe:[157665]) in inherit fd list (caller close_old_fds)
(00.162078)      1: Found fd 1 (id pipe:[157666]) in inherit fd list (caller close_old_fds)
(00.162082)      1: Found fd 2 (id pipe:[157667]) in inherit fd list (caller close_old_fds)
(00.162153) Running setup-namespaces scripts
(00.162166)     RPC
(00.237324)      1: cg: Move into 2
(00.237359)      1: cg:   `-> blkio//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.237408)      1: cg:   `-> cpu,cpuacct//docker/96218d1e332afc18ae46618d2ec4c2229bf884c483bb840f8ee997c36c4ded61/tasks
(00.237436)      1: cg:   `-> cpuset//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.237455)      1: cg:   `-> devices//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.237470)      1: cg:   `-> freezer//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.237486)      1: cg:   `-> hugetlb//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.237501)      1: cg:   `-> memory//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.237514)      1: cg:   `-> systemd//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.237528)      1: cg:   `-> net_cls,net_prio//docker/96218d1e332afc18ae46618d2ec4c2229bf884c483bb840f8ee997c36c4ded61/tasks
(00.243840)      1: cg:   `-> perf_event//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.243979)      1: cg:   `-> pids//docker/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/tasks
(00.244088)      1: Calling restore_sid() for init
(00.244096)      1: Restoring 1 to 1 sid
(00.244273)      1: Mount procfs in crtools-proc.Cb6OGY
(00.271653)      1: Collecting 41/37 (flags 2)
(00.271678)      1: No tty-info.img image
(00.271703)      1:  `- ... done
(00.271706)      1: Collecting 42/51 (flags 0)
(00.271712)      1: No tty-data.img image
(00.271716)      1:  `- ... done
(00.271719)      1: Restoring namespaces 1 flags 0x6c028000
(00.280689)      1: kernel/hostname nr 12
(00.280845)      1: kernel/domainname nr 6
(00.282030)      1: Restoring IPC namespace
(00.282049)      1: Restoring IPC variables
(00.290134)      1: Restoring IPC shared memory
(00.290157)      1: No ipcns-shm-10.img image
(00.290160)      1: Restoring IPC message queues
(00.290163)      1: No ipcns-msg-10.img image
(00.290165)      1: Restoring IPC semaphores sets
(00.290168)      1: No ipcns-sem-10.img image
(00.290171)      1: mnt: Restoring mount namespace
(00.290179)      1: mnt: Building mountpoints tree
(00.290183)      1: mnt:    Building plain mount tree
(00.290185)      1: mnt:        Working on 1269->1598
(00.290191)      1: mnt: Mountpoint 1269 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/) w/o parent 1598
(00.290194)      1: mnt: Mountpoint 1269 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/) get parent 0 (@/tmp/.criu.mntns.s8pOlp)
(00.290199)      1: mnt:        Working on 1270->1269
(00.290203)      1: mnt:        Working on 1271->1270
(00.290204)      1: mnt:        Working on 1272->1270
(00.290205)      1: mnt:        Working on 1273->1270
(00.290207)      1: mnt:        Working on 1274->1270
(00.290208)      1: mnt:        Working on 1275->1270
(00.290212)      1: mnt:        Working on 1276->1270
(00.290215)      1: mnt:        Working on 1277->1269
(00.290216)      1: mnt:        Working on 1278->1277
(00.290218)      1: mnt:        Working on 1279->1277
(00.290219)      1: mnt:        Working on 1280->1277
(00.290220)      1: mnt:        Working on 1281->1269
(00.290225)      1: mnt:        Working on 1282->1281
(00.290228)      1: mnt:        Working on 1283->1281
(00.290229)      1: mnt:        Working on 2013->1283
(00.290230)      1: mnt:        Working on 2014->1283
(00.290232)      1: mnt:        Working on 2016->1283
(00.290236)      1: mnt:        Working on 2018->1283
(00.290247)      1: mnt:        Working on 2019->1283
(00.290248)      1: mnt:        Working on 2020->1283
(00.290250)      1: mnt:        Working on 2021->1283
(00.290251)      1: mnt:        Working on 2022->1283
(00.290252)      1: mnt:        Working on 2023->1283
(00.290257)      1: mnt:        Working on 2024->1283
(00.290260)      1: mnt:        Working on 2025->1283
(00.290262)      1: mnt:        Working on 2026->1269
(00.290263)      1: mnt:        Working on 2027->1269
(00.290264)      1: mnt:        Working on 2028->1269
(00.290268)      1: mnt:        Working on 2029->1269
(00.290271)      1: mnt:        Working on 2030->1270
(00.290273)      1: mnt:        Working on 2031->1270
(00.290274)      1: mnt:        Working on 2032->1270
(00.290280)      1: mnt:        Working on 2033->1270
(00.290283)      1: mnt:    Resorting siblings on 0
(00.290285)      1: mnt:    Resorting siblings on 1269
(00.290286)      1: mnt:    Resorting siblings on 1270
(00.290287)      1: mnt:    Resorting siblings on 1271
(00.290289)      1: mnt:    Resorting siblings on 1272
(00.290290)      1: mnt:    Resorting siblings on 1273
(00.290292)      1: mnt:    Resorting siblings on 1274
(00.290293)      1: mnt:    Resorting siblings on 1275
(00.290294)      1: mnt:    Resorting siblings on 1276
(00.290296)      1: mnt:    Resorting siblings on 2030
(00.290297)      1: mnt:    Resorting siblings on 2031
(00.290299)      1: mnt:    Resorting siblings on 2032
(00.290300)      1: mnt:    Resorting siblings on 2033
(00.290301)      1: mnt:    Resorting siblings on 1277
(00.290303)      1: mnt:    Resorting siblings on 1278
(00.290304)      1: mnt:    Resorting siblings on 1279
(00.290305)      1: mnt:    Resorting siblings on 1280
(00.290307)      1: mnt:    Resorting siblings on 1281
(00.290308)      1: mnt:    Resorting siblings on 1282
(00.290310)      1: mnt:    Resorting siblings on 1283
(00.290311)      1: mnt:    Resorting siblings on 2013
(00.290312)      1: mnt:    Resorting siblings on 2014
(00.290314)      1: mnt:    Resorting siblings on 2016
(00.290315)      1: mnt:    Resorting siblings on 2018
(00.290317)      1: mnt:    Resorting siblings on 2019
(00.290318)      1: mnt:    Resorting siblings on 2020
(00.290320)      1: mnt:    Resorting siblings on 2021
(00.290321)      1: mnt:    Resorting siblings on 2022
(00.290323)      1: mnt:    Resorting siblings on 2023
(00.290324)      1: mnt:    Resorting siblings on 2024
(00.290326)      1: mnt:    Resorting siblings on 2025
(00.290327)      1: mnt:    Resorting siblings on 2026
(00.290329)      1: mnt:    Resorting siblings on 2027
(00.290330)      1: mnt:    Resorting siblings on 2028
(00.290332)      1: mnt:    Resorting siblings on 2029
(00.290333)      1: mnt: Done:
(00.290334)      1: mnt: [/tmp/.criu.mntns.s8pOlp](0->0)
(00.290336)      1: mnt:  [/tmp/.criu.mntns.s8pOlp/12-0000000000/](1269->1598)
(00.290338)      1: mnt:   [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc](1270->1269)
(00.290339)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound](1271->1270)
(00.290341)      1: mnt:    <--
(00.290342)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug](2033->1270)
(00.290344)      1: mnt:    <--
(00.290345)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats](2032->1270)
(00.290347)      1: mnt:    <--
(00.290348)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list](2031->1270)
(00.290349)      1: mnt:    <--
(00.290351)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore](2030->1270)
(00.290352)      1: mnt:    <--
(00.290354)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger](1276->1270)
(00.290355)      1: mnt:    <--
(00.290356)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys](1275->1270)
(00.290358)      1: mnt:    <--
(00.290359)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq](1274->1270)
(00.290361)      1: mnt:    <--
(00.290362)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs](1273->1270)
(00.290363)      1: mnt:    <--
(00.290365)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus](1272->1270)
(00.290366)      1: mnt:    <--
(00.290370)      1: mnt:   <--
(00.290372)      1: mnt:   [/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts](2029->1269)
(00.290373)      1: mnt:   <--
(00.290375)      1: mnt:   [/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname](2028->1269)
(00.290376)      1: mnt:   <--
(00.290377)      1: mnt:   [/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf](2027->1269)
(00.290379)      1: mnt:   <--
(00.290380)      1: mnt:   [/tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules](2026->1269)
(00.290382)      1: mnt:   <--
(00.290383)      1: mnt:   [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys](1281->1269)
(00.290385)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/firmware](1282->1281)
(00.290386)      1: mnt:    <--
(00.290387)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup](1283->1281)
(00.290389)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/systemd](2013->1283)
(00.290390)      1: mnt:     <--
(00.290392)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/blkio](2025->1283)
(00.290393)      1: mnt:     <--
(00.290395)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/hugetlb](2024->1283)
(00.290396)      1: mnt:     <--
(00.290397)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/memory](2023->1283)
(00.290399)      1: mnt:     <--
(00.290400)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/devices](2022->1283)
(00.290402)      1: mnt:     <--
(00.290403)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/pids](2021->1283)
(00.290404)      1: mnt:     <--
(00.290406)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/perf_event](2020->1283)
(00.290407)      1: mnt:     <--
(00.290408)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpuset](2019->1283)
(00.290410)      1: mnt:     <--
(00.290411)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/net_cls,net_prio](2018->1283)
(00.290413)      1: mnt:     <--
(00.290414)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/freezer](2016->1283)
(00.290415)      1: mnt:     <--
(00.290417)      1: mnt:     [/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpu,cpuacct](2014->1283)
(00.290418)      1: mnt:     <--
(00.290420)      1: mnt:    <--
(00.290421)      1: mnt:   <--
(00.290422)      1: mnt:   [/tmp/.criu.mntns.s8pOlp/12-0000000000/dev](1277->1269)
(00.290424)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/pts](1278->1277)
(00.290425)      1: mnt:    <--
(00.290426)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/shm](1280->1277)
(00.290428)      1: mnt:    <--
(00.290429)      1: mnt:    [/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/mqueue](1279->1277)
(00.290431)      1: mnt:    <--
(00.290432)      1: mnt:   <--
(00.290433)      1: mnt:  <--
(00.290435)      1: mnt: <--
(00.290436)      1: mnt: Inspecting sharing on 1269 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/)
(00.290438)      1: mnt: Inspecting sharing on 1270 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc)
(00.290440)      1: mnt:    The mount 1271 is bind for 1270 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc)
(00.290442)      1: mnt:    The mount 1272 is bind for 1270 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc)
(00.290444)      1: mnt:    The mount 1273 is bind for 1270 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc)
(00.290445)      1: mnt:    The mount 1274 is bind for 1270 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc)
(00.290447)      1: mnt:    The mount 1275 is bind for 1270 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc)
(00.290448)      1: mnt:    The mount 1276 is bind for 1270 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc)
(00.290453)      1: mnt: Inspecting sharing on 1271 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound)
(00.290454)      1: mnt: Inspecting sharing on 1272 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus)
(00.290456)      1: mnt: Inspecting sharing on 1273 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs)
(00.290457)      1: mnt: Inspecting sharing on 1274 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq)
(00.290459)      1: mnt: Inspecting sharing on 1275 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys)
(00.290460)      1: mnt: Inspecting sharing on 1276 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger)
(00.290462)      1: mnt: Inspecting sharing on 1277 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/dev)
(00.290464)      1: mnt:    The mount 2030 is bind for 1277 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev)
(00.290465)      1: mnt:    The mount 2031 is bind for 1277 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev)
(00.290467)      1: mnt:    The mount 2032 is bind for 1277 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev)
(00.290468)      1: mnt:    The mount 2033 is bind for 1277 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev)
(00.290470)      1: mnt: Inspecting sharing on 1278 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/pts)
(00.290472)      1: mnt: Inspecting sharing on 1279 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/mqueue)
(00.290473)      1: mnt: Inspecting sharing on 1280 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/shm)
(00.290475)      1: mnt: Inspecting sharing on 1281 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys)
(00.290477)      1: mnt: Inspecting sharing on 1282 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/firmware)
(00.290478)      1: mnt: Inspecting sharing on 1283 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup)
(00.290480)      1: mnt: Inspecting sharing on 2013 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/systemd)
(00.290482)      1: mnt: Inspecting sharing on 2014 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpu,cpuacct)
(00.290483)      1: mnt: Inspecting sharing on 2016 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/freezer)
(00.290485)      1: mnt: Inspecting sharing on 2018 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/net_cls,net_prio)
(00.290487)      1: mnt: Inspecting sharing on 2019 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpuset)
(00.290488)      1: mnt: Inspecting sharing on 2020 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/perf_event)
(00.290490)      1: mnt: Inspecting sharing on 2021 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/pids)
(00.290491)      1: mnt: Inspecting sharing on 2022 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/devices)
(00.290493)      1: mnt: Inspecting sharing on 2023 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/memory)
(00.290495)      1: mnt: Inspecting sharing on 2024 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/hugetlb)
(00.290496)      1: mnt: Inspecting sharing on 2025 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/blkio)
(00.290498)      1: mnt: Inspecting sharing on 2026 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules)
(00.290499)      1: mnt: Inspecting sharing on 2027 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf)
(00.290503)      1: mnt:    The mount 2028 is bind for 2027 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf)
(00.290505)      1: mnt:    The mount 2029 is bind for 2027 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts -> @/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf)
(00.290507)      1: mnt: Inspecting sharing on 2028 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname)
(00.290508)      1: mnt: Inspecting sharing on 2029 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts)
(00.290510)      1: mnt: Inspecting sharing on 2030 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore)
(00.290511)      1: mnt: Inspecting sharing on 2031 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list)
(00.290513)      1: mnt: Inspecting sharing on 2032 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats)
(00.290514)      1: mnt: Inspecting sharing on 2033 shared_id 0 master_id 0 (@/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug)
(00.290518)      1: mnt: Start with 0:/tmp/.criu.mntns.s8pOlp
(00.297286)      1: mnt: Start with 0:/tmp/.criu.mntns.s8pOlp
(00.297302)      1: mnt:    Mounting overlay @/tmp/.criu.mntns.s8pOlp/12-0000000000/ (0)
(00.297319)      1: mnt: 1269:/tmp/.criu.mntns.s8pOlp/12-0000000000/ private 1 shared 0 slave 0
(00.297345)      1: mnt:    Mounting proc @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc (0)
(00.297415)      1: mnt: 1270:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc private 1 shared 0 slave 0
(00.297428)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger
(00.297442)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys
(00.297445)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq
(00.297447)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs
(00.297449)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus
(00.297451)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound
(00.297454)      1: mnt:    Mounting proc @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound (0)
(00.297459)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound
(00.297481)      1: mnt: 1271:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/asound private 1 shared 0 slave 0
(00.300329)      1: mnt: Postpone slave /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug
(00.300340)      1: mnt: Postpone slave /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats
(00.300343)      1: mnt: Postpone slave /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list
(00.300344)      1: mnt: Postpone slave /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore
(00.300346)      1: mnt:    Mounting proc @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger (0)
(00.300349)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger
(00.300367)      1: mnt: 1276:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sysrq-trigger private 1 shared 0 slave 0
(00.300372)      1: mnt:    Mounting proc @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys (0)
(00.300374)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys
(00.300383)      1: mnt: 1275:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sys private 1 shared 0 slave 0
(00.300387)      1: mnt:    Mounting proc @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq (0)
(00.300389)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq
(00.300397)      1: mnt: 1274:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/irq private 1 shared 0 slave 0
(00.300400)      1: mnt:    Mounting proc @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs (0)
(00.300402)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs
(00.300446)      1: mnt: 1273:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/fs private 1 shared 0 slave 0
(00.300454)      1: mnt:    Mounting proc @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus (0)
(00.300457)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus
(00.300470)      1: mnt: 1272:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/bus private 1 shared 0 slave 0
(00.300477)      1: mnt:    Mounting unsupported @/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts (0)
(00.300479)      1: mnt:    Bind /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/hosts to /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts
(00.300506)      1: mnt: 2029:/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hosts private 1 shared 0 slave 0
(00.300526)      1: mnt:    Mounting unsupported @/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname (0)
(00.300530)      1: mnt:    Bind /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/hostname to /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname
(00.300547)      1: mnt: 2028:/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/hostname private 1 shared 0 slave 0
(00.300556)      1: mnt:    Mounting unsupported @/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf (0)
(00.300560)      1: mnt:    Bind /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/resolv.conf to /tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf
(00.300574)      1: mnt: 2027:/tmp/.criu.mntns.s8pOlp/12-0000000000/etc/resolv.conf private 1 shared 0 slave 0
(00.300582)      1: mnt:    Mounting fuse @/tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules (0)
(00.300585)      1: mnt:    Bind /mnt/shared/module-fs-dev/8.x.x to /tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules
(00.300618)      1: mnt: 2026:/tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules private 1 shared 0 slave 0
(00.300625)      1: mnt:    Mounting sysfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys (0)
(00.300713)      1: mnt: 1281:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys private 1 shared 0 slave 0
(00.300723)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/firmware (0)
(00.307458)      1: Found fd 0 (id pipe:[157665]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.307517)      1: Inherit fd 0 moved to 7 to resolve clash
(00.307526)      1: Found fd 1 (id pipe:[157666]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.307529)      1: Inherit fd 1 moved to 3 to resolve clash
(00.307531)      1: Found fd 2 (id pipe:[157667]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.307533)      1: Inherit fd 2 moved to 5 to resolve clash
(00.312411)      1: mnt: 1282:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/firmware private 1 shared 0 slave 0
(00.312437)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup (0)
(00.318450)      1: Found fd 0 (id pipe:[157665]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.318500)      1: Inherit fd 0 moved to 7 to resolve clash
(00.318508)      1: Found fd 1 (id pipe:[157666]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.318510)      1: Inherit fd 1 moved to 3 to resolve clash
(00.318512)      1: Found fd 2 (id pipe:[157667]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.318513)      1: Inherit fd 2 moved to 5 to resolve clash
(00.321435)      1: mnt: 1283:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup private 1 shared 0 slave 0
(00.321456)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/systemd (0)
(00.321459)      1: mnt:    Bind /sys/fs/cgroup/systemd/system.slice/evaldocker.service to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/systemd
(00.321480)      1: mnt: 2013:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/systemd private 1 shared 0 slave 0
(00.321487)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/blkio (0)
(00.321488)      1: mnt:    Bind /sys/fs/cgroup/blkio/system.slice/evaldocker.service to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/blkio
(00.321506)      1: mnt: 2025:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/blkio private 1 shared 0 slave 0
(00.321510)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/hugetlb (0)
(00.321512)      1: mnt:    Bind /sys/fs/cgroup/hugetlb to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/hugetlb
(00.321519)      1: mnt: 2024:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/hugetlb private 1 shared 0 slave 0
(00.321523)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/memory (0)
(00.321525)      1: mnt:    Bind /sys/fs/cgroup/memory/system.slice/evaldocker.service to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/memory
(00.321533)      1: mnt: 2023:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/memory private 1 shared 0 slave 0
(00.321538)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/devices (0)
(00.321540)      1: mnt:    Bind /sys/fs/cgroup/devices/system.slice/evaldocker.service to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/devices
(00.321548)      1: mnt: 2022:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/devices private 1 shared 0 slave 0
(00.321552)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/pids (0)
(00.321554)      1: mnt:    Bind /sys/fs/cgroup/pids/system.slice/evaldocker.service to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/pids
(00.321561)      1: mnt: 2021:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/pids private 1 shared 0 slave 0
(00.321565)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/perf_event (0)
(00.321567)      1: mnt:    Bind /sys/fs/cgroup/perf_event to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/perf_event
(00.321573)      1: mnt: 2020:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/perf_event private 1 shared 0 slave 0
(00.321577)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpuset (0)
(00.321578)      1: mnt:    Bind /sys/fs/cgroup/cpuset to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpuset
(00.321584)      1: mnt: 2019:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpuset private 1 shared 0 slave 0
(00.321588)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/net_cls,net_prio (0)
(00.321590)      1: mnt:    Bind /sys/fs/cgroup/net_cls,net_prio to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/net_cls,net_prio
(00.321596)      1: mnt: 2018:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/net_cls,net_prio private 1 shared 0 slave 0
(00.321600)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/freezer (0)
(00.321602)      1: mnt:    Bind /sys/fs/cgroup/freezer to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/freezer
(00.321608)      1: mnt: 2016:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/freezer private 1 shared 0 slave 0
(00.321612)      1: mnt:    Mounting cgroup @/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpu,cpuacct (0)
(00.321613)      1: mnt:    Bind /sys/fs/cgroup/cpu,cpuacct/system.slice/evaldocker.service to /tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpu,cpuacct
(00.321620)      1: mnt: 2014:/tmp/.criu.mntns.s8pOlp/12-0000000000/sys/fs/cgroup/cpu,cpuacct private 1 shared 0 slave 0
(00.321626)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev (0)
(00.327122)      1: Found fd 0 (id pipe:[157665]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.327166)      1: Inherit fd 0 moved to 7 to resolve clash
(00.327173)      1: Found fd 1 (id pipe:[157666]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.327175)      1: Inherit fd 1 moved to 3 to resolve clash
(00.327177)      1: Found fd 2 (id pipe:[157667]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.327179)      1: Inherit fd 2 moved to 5 to resolve clash
(00.330535)      1: mnt: 1277:/tmp/.criu.mntns.s8pOlp/12-0000000000/dev private 1 shared 0 slave 0
(00.330562)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug
(00.330566)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats
(00.330567)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list
(00.330569)      1: mnt:        Bind private /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore
(00.330571)      1: mnt:    Mounting devpts @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/pts (0)
(00.330624)      1: mnt: 1278:/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/pts private 1 shared 0 slave 0
(00.330629)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/shm (0)
(00.330630)      1: mnt:    Bind /var/lib/evaldocker/containers/8606da38a707f960b5b27a634478fd19550c041f688a7d437ee6605e4f921959/shm to /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/shm
(00.330640)      1: mnt: 1280:/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/shm private 1 shared 0 slave 0
(00.330644)      1: mnt:    Mounting mqueue @/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/mqueue (0)
(00.330654)      1: mnt: 1279:/tmp/.criu.mntns.s8pOlp/12-0000000000/dev/mqueue private 1 shared 0 slave 0
(00.330658)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug (0)
(00.330661)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/null to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug
(00.330668)      1: mnt: 2033:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/sched_debug private 1 shared 0 slave 0
(00.330672)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats (0)
(00.330673)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/null to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats
(00.330678)      1: mnt: 2032:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_stats private 1 shared 0 slave 0
(00.330682)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list (0)
(00.330683)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/null to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list
(00.330688)      1: mnt: 2031:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/timer_list private 1 shared 0 slave 0
(00.330691)      1: mnt:    Mounting tmpfs @/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore (0)
(00.330693)      1: mnt:    Bind /tmp/.criu.mntns.s8pOlp/12-0000000000/dev/null to /tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore
(00.330697)      1: mnt: 2030:/tmp/.criu.mntns.s8pOlp/12-0000000000/proc/kcore private 1 shared 0 slave 0
(00.330701)      1: mnt: Start with 0:/tmp/.criu.mntns.s8pOlp
(00.349266)      1: mnt: Move the root to //tmp/.criu.mntns.s8pOlp/12-0000000000
(00.388003) Found fd 0 (id pipe:[157665]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.388048) Inherit fd 0 moved to 12 to resolve clash
(00.388054) Found fd 1 (id pipe:[157666]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.388056) Inherit fd 1 moved to 8 to resolve clash
(00.388058) Found fd 2 (id pipe:[157667]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.388060) Inherit fd 2 moved to 10 to resolve clash
iptables-restore: invalid option -- 'w'
(00.394970) Found fd 0 (id pipe:[157665]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.395007) Inherit fd 0 moved to 12 to resolve clash
(00.395013) Found fd 1 (id pipe:[157666]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.395015) Inherit fd 1 moved to 8 to resolve clash
(00.395016) Found fd 2 (id pipe:[157667]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.395018) Inherit fd 2 moved to 10 to resolve clash
ip6tables-restore: invalid option -- 'w'
(00.396152) Running post-setup-namespaces scripts
(00.396161)     RPC
(00.396305)      1: Preparing info about shared resources
(00.396323)      1: Configuring remap 0xc -> 0x1
(00.396758)      1: Error (criu/files-reg.c:358): Can't create ghost regfile: Read-only file system
(00.397233) Error (criu/cr-restore.c:1301): 31708 killed by signal 127: Unknown signal 127
(00.461438) mnt: Switching to new ns to clean ghosts
(00.461550) Unlink remap app/available_modules/1513475394000/bcrypt/lib/binding/bcrypt_lib.node.cr.1.ghost
(00.462950) Error (criu/files-reg.c:680): Couldn't unlink remap 10 app/available_modules/1513475394000/bcrypt/lib/binding/bcrypt_lib.node.cr.1.ghost: Read-only file system
(00.481359) Error (criu/cr-restore.c:2171): Restoring FAILED.

Any help is appreciated! Thanks!

avagin commented 6 years ago

Error in `criu': free(): invalid pointer: 0x00007f2dfd5c0bf0

this issue was fixed in 4d3ae51725552a38a2b5a9f0ba2c83e0d1fab89d

avagin commented 6 years ago

Cc: @Snorch

avagin commented 6 years ago

I'm diving into CRIU and would appreciate any hints, but Im wondering why this file in particular has to be "remapped", and what I can do to prevent that.

It means that one of processes has a file descriptor to this file, but this file was unlinked from a file system. You can use "lsof | grep '(deleted)'" to find such files. https://criu.org/How_hard_is_it_to_open_a_file

avagin commented 6 years ago

(00.300582) 1: mnt: Mounting fuse @/tmp/.criu.mntns.s8pOlp/12-0000000000/app/available_modules (0)

Is it a fuse mount? CRIU doesn't support fuse mounts. Why do we not return an error on dump?

erulabs commented 6 years ago

Hrm, the filesystem is read-only, so that file certainly shouldn't have been deleted. It still exists on the host, at least. Will keep investigating, thanks!

github-actions[bot] commented 3 years ago

A friendly reminder that this issue had no activity for 30 days.