Closed diabolusss closed 2 years ago
adrianreber
commented
4 years ago This is confusing. This are different containers, right? With different failures.
It would make sense to start with a simple container. We have examples for Docker and Podman (https://criu.org/Podman) which are just running a simple loop in the container. Try to set up something like this with LXD and then we can try to see if that works.
diabolusss
commented
4 years ago Yes, the first - xpraserver - is the old one container that i have tuned for my needs, but the second one was freshly created before posting to check that problem is not in my container.
Another one test, created specially for this answer:
$ lxc launch images:archlinux/current test2 --storage=secondpool
Creating test2
$ lxc info test2
Name: test2
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/04/06 18:17 UTC
Status: Running
Type: container
Profiles: default
Pid: 22476
Ips:
eth0: inet 192.168.100.207 vetha63ae4cb
eth0: inet6 fe80::216:3eff:fecb:d20d vetha63ae4cb
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 9
CPU usage:
CPU usage (in seconds): 2
Memory usage:
Memory (current): 171.70MB
Network usage:
eth0:
Bytes received: 3.71kB
Bytes sent: 3.16kB
Packets received: 31
Packets sent: 27
lo:
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0Trying to stop instantly after starting.
$ lxc stop test2 --stateful
Error: snapshot dump failed
(00.000062) Warn (criu/log.c:203): The early log isn't empty
(00.101377) Error (criu/namespaces.c:415): Can't dump nested uts namespace for 22578
(00.101380) Error (criu/namespaces.c:670): Can't make utsns id
(00.103470) Error (criu/util.c:610): exited, status=1
(00.105301) Error (criu/util.c:610): exited, status=1
(00.105442) Error (criu/cr-dump.c:1743): Dumping FAILED.
Try `lxc info --show-log test2` for more info
$lxc info --show-log test2
...
Log:
lxc test2 20200406181800.709 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.test2"
lxc test2 20200406181800.710 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.test2"
lxc test2 20200406181800.711 ERROR utils - utils.c:lxc_can_use_pidfd:1855 - Invalid argument - Kernel does not support waiting on processes through pidfds
lxc test2 20200406181800.712 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1455 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc 20200406181952.521 ERROR criu - criu.c:do_dump:1304 - dump failed with 1
lxc 20200406181952.521 ERROR criu - criu.c:do_dump:1318 - criu output:Logs taken from snap/ldx folder:
$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log|grep test2
t=2020-04-06T21:17:53+0300 lvl=info msg="Creating container" ephemeral=false name=test2 project=default
t=2020-04-06T21:17:53+0300 lvl=info msg="Created container" ephemeral=false name=test2 project=default
t=2020-04-06T21:18:00+0300 lvl=info msg="Starting container" action=start created=2020-04-06T21:17:53+0300 ephemeral=false name=test2 project=default stateful=false used=1970-01-01T03:00:00+0300
t=2020-04-06T21:18:00+0300 lvl=info msg="Started container" action=start created=2020-04-06T21:17:53+0300 ephemeral=false name=test2 project=default stateful=false used=1970-01-01T03:00:00+0300
t=2020-04-06T21:19:52+0300 lvl=info msg="Stopping container" action=stop created=2020-04-06T21:17:53+0300 ephemeral=false name=test2 project=default stateful=true used=2020-04-06T21:18:00+0300
t=2020-04-06T21:19:52+0300 lvl=info msg="Migrating container" actionscript=false created=2020-04-06T21:17:53+0300 ephemeral=false features=0 name=test2 predumpdir= project=default statedir=/var/snap/lxd/common/lxd/containers/test2/state stop=true used=2020-04-06T21:18:00+0300
t=2020-04-06T21:19:52+0300 lvl=info msg="Failed migrating container" actionscript=false created=2020-04-06T21:17:53+0300 ephemeral=false features=0 name=test2 predumpdir= project=default statedir=/var/snap/lxd/common/lxd/containers/test2/state stop=true used=2020-04-06T21:18:00+0300
t=2020-04-06T21:19:52+0300 lvl=eror msg="Failed stopping container" action=stop created=2020-04-06T21:17:53+0300 ephemeral=false name=test2 project=default stateful=true used=2020-04-06T21:18:00+0300$ sudo cat /var/snap/lxd/common/lxd/logs/test2/snapshot_dump_2020-04-06T21:19:52+03:00.log
(00.000000) Will dump/restore TCP connections
(00.000000) Will skip in-flight TCP connections
(00.000000) Will allow link remaps on FS
(00.000062) Warn (criu/log.c:203): The early log isn't empty
(00.000066) Version: 3.13 (gitid v3.13)
(00.000071) Running on fallback-os Linux 5.2.11-arch1-1-ARCH #1 SMP PREEMPT Thu Aug 29 08:09:36 UTC 2019 x86_
64
(00.000082) Loaded kdat cache from /run/criu.kdat
(00.000085) ========================================
(00.000088) Dumping processes (pid: 22476)
(00.000091) ========================================
(00.000097) rlimit: RLIMIT_NOFILE unlimited for self
(00.000104) Running pre-dump scripts
(00.000122) irmap: Searching irmap cache in work dir
(00.000133) No irmap-cache image
(00.000137) irmap: Searching irmap cache in parent
(00.000143) irmap: No irmap cache
(00.000148) cpu: x86_family 23 x86_vendor_id AuthenticAMD x86_model_id AMD Ryzen 5 2600 Six-Core Processor
(00.000152) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 832
(00.000156) cpu: fpu: x87 floating point registers xstate_offsets 0 / 0 xstate_sizes 160 / 1
60
(00.000160) cpu: fpu: AVX registers xstate_offsets 576 / 576 xstate_sizes 256 / 2
56
(00.000164) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.000257) cg-prop: Parsing controller "cpu"
(00.000263) cg-prop: Strategy "replace"
(00.000267) cg-prop: Property "cpu.shares"
(00.000270) cg-prop: Property "cpu.cfs_period_us"
(00.000273) cg-prop: Property "cpu.cfs_quota_us"
(00.000276) cg-prop: Property "cpu.rt_period_us"
(00.000279) cg-prop: Property "cpu.rt_runtime_us"
(00.000282) cg-prop: Parsing controller "memory"
(00.000285) cg-prop: Strategy "replace"
(00.000288) cg-prop: Property "memory.limit_in_bytes"
(00.000291) cg-prop: Property "memory.memsw.limit_in_bytes"
(00.000294) cg-prop: Property "memory.swappiness"
(00.000297) cg-prop: Property "memory.soft_limit_in_bytes"
(00.000300) cg-prop: Property "memory.move_charge_at_immigrate"
(00.000303) cg-prop: Property "memory.oom_control"
(00.000306) cg-prop: Property "memory.use_hierarchy"
(00.000308) cg-prop: Property "memory.kmem.limit_in_bytes"
(00.000311) cg-prop: Property "memory.kmem.tcp.limit_in_bytes"
(00.000314) cg-prop: Parsing controller "cpuset"
(00.000317) cg-prop: Strategy "replace"
(00.000320) cg-prop: Property "cpuset.cpus"
(00.000323) cg-prop: Property "cpuset.mems"
(00.000326) cg-prop: Property "cpuset.memory_migrate"
(00.000329) cg-prop: Property "cpuset.cpu_exclusive"
(00.000332) cg-prop: Property "cpuset.mem_exclusive"
(00.000335) cg-prop: Property "cpuset.mem_hardwall"
(00.000338) cg-prop: Property "cpuset.memory_spread_page"
(00.000341) cg-prop: Property "cpuset.memory_spread_slab"
(00.000343) cg-prop: Property "cpuset.sched_load_balance"
(00.000346) cg-prop: Property "cpuset.sched_relax_domain_level"
(00.000349) cg-prop: Parsing controller "blkio"
(00.000352) cg-prop: Strategy "replace"
(00.000355) cg-prop: Property "blkio.weight"
(00.000358) cg-prop: Parsing controller "freezer"
(00.000361) cg-prop: Strategy "replace"
(00.000364) cg-prop: Parsing controller "perf_event"
(00.000367) cg-prop: Strategy "replace"
(00.000370) cg-prop: Parsing controller "net_cls"
(00.000373) cg-prop: Strategy "replace"
(00.000376) cg-prop: Property "net_cls.classid"
(00.000379) cg-prop: Parsing controller "net_prio"
(00.000382) cg-prop: Strategy "replace"
(00.000385) cg-prop: Property "net_prio.ifpriomap"
(00.000388) cg-prop: Parsing controller "pids"
(00.000391) cg-prop: Strategy "replace"
(00.000394) cg-prop: Property "pids.max"
(00.000397) cg-prop: Parsing controller "devices"
(00.000400) cg-prop: Strategy "replace"
(00.000403) cg-prop: Property "devices.list"
(00.000454) Perparing image inventory (version 1)
(00.000480) Add pid ns 1 pid 22862
(00.000488) Add net ns 2 pid 22862
(00.000496) Add ipc ns 3 pid 22862
(00.000503) Add uts ns 4 pid 22862
(00.000509) Add mnt ns 5 pid 22862
(00.000516) Add user ns 6 pid 22862
(00.000523) Add cgroup ns 7 pid 22862
(00.000526) cg: Dumping cgroups for 22862
(00.000556) cg: `- New css ID 1
(00.000560) cg: `- [blkio] -> [/] [0]
(00.000564) cg: `- [cpu,cpuacct] -> [/] [0]
(00.000567) cg: `- [cpuset] -> [/] [0]
(00.000570) cg: `- [devices] -> [/] [0]
(00.000573) cg: `- [freezer] -> [/] [0]
(00.000576) cg: `- [hugetlb] -> [/] [0]
(00.000578) cg: `- [memory] -> [/] [0]
(00.000581) cg: `- [name=systemd] -> [/] [0]
(00.000584) cg: `- [net_cls,net_prio] -> [/] [0]
(00.000587) cg: `- [perf_event] -> [/] [0]
(00.000590) cg: `- [pids] -> [/] [0]
(00.000593) cg: `- [rdma] -> [/] [0]
(00.000596) cg: Set 1 is criu one
(00.000610) freezing processes: 100000 attempts with 100 ms steps
(00.000629) freezer.state=THAWED
(00.000668) freezer.state=FREEZING
(00.100728) freezer.state=FROZEN
(00.100733) freezing processes: 1 attempts done
(00.100760) SEIZE 22476: success
(00.100764) SEIZE 22567: success
(00.100768) SEIZE 22578: success
(00.100771) SEIZE 22583: success
(00.100774) SEIZE 22587: success
(00.100777) SEIZE 22588: success
(00.100781) SEIZE 22589: success
(00.100784) SEIZE 22590: success
(00.100787) SEIZE 22592: success
(00.100871) Seized task 22476, state 1
(00.100876) seccomp: Collected tid_real 22476 mode 0x2
(00.100887) Collected (0 attempts, 0 in_progress)
(00.100901) Seized task 22567, state 0
(00.100924) seccomp: Collected tid_real 22567 mode 0x2
(00.100933) Collected (0 attempts, 0 in_progress)
(00.100942) Collected (0 attempts, 0 in_progress)
(00.100952) Collected 22567 in 1 state
(00.100954) Seized task 22578, state 0
(00.100978) seccomp: Collected tid_real 22578 mode 0x2
(00.100988) Collected (0 attempts, 0 in_progress)
(00.100997) Collected (0 attempts, 0 in_progress)
(00.101000) Collected 22578 in 1 state
(00.101002) Seized task 22583, state 0
(00.101027) seccomp: Collected tid_real 22583 mode 0x2
(00.101036) Collected (0 attempts, 0 in_progress)
(00.101045) Collected (0 attempts, 0 in_progress)
(00.101048) Collected 22583 in 1 state
(00.101050) Seized task 22587, state 0
(00.101073) seccomp: Collected tid_real 22587 mode 0x2
(00.101081) Collected (0 attempts, 0 in_progress)
(00.101090) Collected (0 attempts, 0 in_progress)
(00.101094) Collected 22587 in 1 state
(00.101096) Seized task 22588, state 0
(00.101118) seccomp: Collected tid_real 22588 mode 0x2
(00.101128) Collected (0 attempts, 0 in_progress)
(00.101137) Collected (0 attempts, 0 in_progress)
(00.101141) Collected 22588 in 1 state
(00.101143) Seized task 22589, state 0
(00.101164) seccomp: Collected tid_real 22589 mode 0x2
(00.101174) Collected (0 attempts, 0 in_progress)
(00.101183) Collected (0 attempts, 0 in_progress)
(00.101187) Collected 22589 in 1 state
(00.101189) Seized task 22590, state 0
(00.101212) seccomp: Collected tid_real 22590 mode 0x2
(00.101221) Collected (0 attempts, 0 in_progress)
(00.101230) Collected (0 attempts, 0 in_progress)
(00.101234) Collected 22590 in 1 state
(00.101236) Seized task 22592, state 0
(00.101258) seccomp: Collected tid_real 22592 mode 0x2
(00.101267) Collected (0 attempts, 0 in_progress)
(00.101276) Collected (0 attempts, 0 in_progress)
(00.101280) Collected 22592 in 1 state
(00.101292) Collected (-1 attempts, 0 in_progress)
(00.101295) Collected 22476 in 1 state
(00.101305) Will take pid namespace in the image
(00.101307) Add pid ns 8 pid 22476
(00.101312) Will take net namespace in the image
(00.101314) Add net ns 9 pid 22476
(00.101318) Will take ipc namespace in the image
(00.101320) Add ipc ns 10 pid 22476
(00.101323) Will take uts namespace in the image
(00.101325) Add uts ns 11 pid 22476
(00.101329) Will take mnt namespace in the image
(00.101331) Add mnt ns 12 pid 22476
(00.101335) Will take user namespace in the image
(00.101337) Add user ns 13 pid 22476
(00.101341) Will take cgroup namespace in the image
(00.101343) Add cgroup ns 14 pid 22476
(00.101377) Error (criu/namespaces.c:415): Can't dump nested uts namespace for 22578
(00.101380) Error (criu/namespaces.c:670): Can't make utsns id
(00.101384) Unlock network
(00.101386) Running network-unlock scripts
iptables-restore: invalid option -- 'w'
iptables-restore: line 3 failed
(00.103470) Error (criu/util.c:610): exited, status=1
ip6tables-restore: invalid option -- 'w'
ip6tables-restore: line 3 failed
(00.105301) Error (criu/util.c:610): exited, status=1
(00.105328) Unfreezing tasks into 1
(00.105335) Unseizing 22476 into 1
(00.105344) Unseizing 22567 into 1
(00.105354) Unseizing 22578 into 1
(00.101212) seccomp: Collected tid_real 22590 mode 0x2
(00.101221) Collected (0 attempts, 0 in_progress)
(00.101230) Collected (0 attempts, 0 in_progress)
(00.101234) Collected 22590 in 1 state
(00.101236) Seized task 22592, state 0
(00.101258) seccomp: Collected tid_real 22592 mode 0x2
(00.101267) Collected (0 attempts, 0 in_progress)
(00.101276) Collected (0 attempts, 0 in_progress)
(00.101280) Collected 22592 in 1 state
(00.101292) Collected (-1 attempts, 0 in_progress)
(00.101295) Collected 22476 in 1 state
(00.101305) Will take pid namespace in the image
(00.101307) Add pid ns 8 pid 22476
(00.101312) Will take net namespace in the image
(00.101314) Add net ns 9 pid 22476
(00.101318) Will take ipc namespace in the image
(00.101320) Add ipc ns 10 pid 22476
(00.101323) Will take uts namespace in the image
(00.101325) Add uts ns 11 pid 22476
(00.101329) Will take mnt namespace in the image
(00.101331) Add mnt ns 12 pid 22476
(00.101335) Will take user namespace in the image
(00.101337) Add user ns 13 pid 22476
(00.101341) Will take cgroup namespace in the image
(00.101343) Add cgroup ns 14 pid 22476
(00.101377) Error (criu/namespaces.c:415): Can't dump nested uts namespace for 22578
(00.101380) Error (criu/namespaces.c:670): Can't make utsns id
(00.101384) Unlock network
(00.101386) Running network-unlock scripts
iptables-restore: invalid option -- 'w'
iptables-restore: line 3 failed
(00.103470) Error (criu/util.c:610): exited, status=1
ip6tables-restore: invalid option -- 'w'
ip6tables-restore: line 3 failed
(00.105301) Error (criu/util.c:610): exited, status=1
(00.105328) Unfreezing tasks into 1
(00.105335) Unseizing 22476 into 1
(00.105344) Unseizing 22567 into 1
(00.105354) Unseizing 22578 into 1
(00.105364) Unseizing 22583 into 1
(00.105373) Unseizing 22587 into 1
(00.105382) Unseizing 22588 into 1
(00.105391) Unseizing 22589 into 1
(00.105403) Unseizing 22590 into 1
(00.105412) Unseizing 22592 into 1
(00.105442) Error (criu/cr-dump.c:1743): Dumping FAILED.
$ iptables --version
iptables v1.8.3 (legacy)$ iptables-restore --version
iptables-restore v1.8.3 (legacy)
$ iptables-restore -h
Usage: iptables-restore [-c] [-v] [-V] [-t] [-h] [-n] [-w secs] [-W usecs] [-T table] [-M command]
[ --counters ]
[ --verbose ]
[ --version]
[ --test ]
[ --help ]
[ --noflush ]
[ --wait=<seconds>
[ --wait-interval=<usecs>
[ --table=<TABLE> ]
[ --modprobe=<command> ]What is running in your container? Is there an init system running in your container? systemd?
If that is the case, can you try a container without sytemd?
diabolusss
commented
4 years ago Is it important to run smth in container before making stateful stop or snapshot? Previous systems all have systemd. Based on google Alpine linux is one that have no systemd and is listed in lxd images: so i used it for a test:
$ lxc launch images:alpine/edge alpineedge --storage=secondpool
Creating alpineedge
Starting alpineedge
$ lxc stop alpineedge --stateful
Error: snapshot dump failed
(00.000063) Warn (criu/log.c:203): The early log isn't empty
(00.114481) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/13979/status
(00.114498) Error (criu/cr-dump.c:1345): Dump files (pid: 14004) failed with -1
(00.118748) Error (criu/cr-dump.c:1743): Dumping FAILED.
Try `lxc info --show-log alpineedge` for more info
$ lxc info --show-log alpineedge
...
Log:
lxc alpineedge 20200406193428.540 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.alpineedge"
lxc alpineedge 20200406193428.541 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.alpineedge"
lxc alpineedge 20200406193428.542 ERROR utils - utils.c:lxc_can_use_pidfd:1855 - Invalid argument - Kernel does not support waiting on processes through pidfds
lxc alpineedge 20200406193428.543 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1455 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc 20200406193835.915 ERROR criu - criu.c:do_dump:1304 - dump failed with 1
lxc 20200406193835.915 ERROR criu - criu.c:do_dump:1318 - criu output: $ sudo cat /var/snap/lxd/common/lxd/logs/alpineedge/snapshot_dump_2020-04-06T22:38:35+03:00.log
(00.000000) Will dump/restore TCP connections
(00.000000) Will skip in-flight TCP connections
(00.000000) Will allow link remaps on FS
(00.000063) Warn (criu/log.c:203): The early log isn't empty
(00.000067) Version: 3.13 (gitid v3.13)
(00.000073) Running on fallback-os Linux 5.2.11-arch1-1-ARCH #1 SMP PREEMPT Thu Aug 29 08:09:36 UTC 2019 x86_64
(00.000083) Loaded kdat cache from /run/criu.kdat
(00.000087) ========================================
(00.000090) Dumping processes (pid: 14004)
(00.000093) ========================================
(00.000099) rlimit: RLIMIT_NOFILE unlimited for self
(00.000107) Running pre-dump scripts
(00.000125) irmap: Searching irmap cache in work dir
(00.000138) No irmap-cache image
(00.000141) irmap: Searching irmap cache in parent
(00.000147) irmap: No irmap cache
(00.000153) cpu: x86_family 23 x86_vendor_id AuthenticAMD x86_model_id AMD Ryzen 5 2600 Six-Core Processor
(00.000156) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 832
(00.000160) cpu: fpu: x87 floating point registers xstate_offsets 0 / 0 xstate_sizes 160 / 160
(00.000165) cpu: fpu: AVX registers xstate_offsets 576 / 576 xstate_sizes 256 / 256
(00.000169) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.000262) cg-prop: Parsing controller "cpu"
(00.000269) cg-prop: Strategy "replace"
(00.000273) cg-prop: Property "cpu.shares"
(00.000276) cg-prop: Property "cpu.cfs_period_us"
(00.000279) cg-prop: Property "cpu.cfs_quota_us"
(00.000282) cg-prop: Property "cpu.rt_period_us"
(00.000285) cg-prop: Property "cpu.rt_runtime_us"
(00.000288) cg-prop: Parsing controller "memory"
(00.000291) cg-prop: Strategy "replace"
(00.000294) cg-prop: Property "memory.limit_in_bytes"
(00.000297) cg-prop: Property "memory.memsw.limit_in_bytes"
(00.000300) cg-prop: Property "memory.swappiness"
(00.000303) cg-prop: Property "memory.soft_limit_in_bytes"
(00.000306) cg-prop: Property "memory.move_charge_at_immigrate"
(00.000309) cg-prop: Property "memory.oom_control"
(00.000311) cg-prop: Property "memory.use_hierarchy"
(00.000314) cg-prop: Property "memory.kmem.limit_in_bytes"
(00.000317) cg-prop: Property "memory.kmem.tcp.limit_in_bytes"
(00.000320) cg-prop: Parsing controller "cpuset"
(00.000323) cg-prop: Strategy "replace"
(00.000326) cg-prop: Property "cpuset.cpus"
(00.000329) cg-prop: Property "cpuset.mems"
(00.000332) cg-prop: Property "cpuset.memory_migrate"
(00.000335) cg-prop: Property "cpuset.cpu_exclusive"
(00.000338) cg-prop: Property "cpuset.mem_exclusive"
(00.000341) cg-prop: Property "cpuset.mem_hardwall"
(00.000344) cg-prop: Property "cpuset.memory_spread_page"
(00.000346) cg-prop: Property "cpuset.memory_spread_slab"
(00.000349) cg-prop: Property "cpuset.sched_load_balance"
(00.000352) cg-prop: Property "cpuset.sched_relax_domain_level"
(00.000355) cg-prop: Parsing controller "blkio"
(00.000358) cg-prop: Strategy "replace"
(00.000361) cg-prop: Property "blkio.weight"
(00.000364) cg-prop: Parsing controller "freezer"
(00.000367) cg-prop: Strategy "replace"
(00.000370) cg-prop: Parsing controller "perf_event"
(00.000373) cg-prop: Strategy "replace"
(00.000376) cg-prop: Parsing controller "net_cls"
(00.000379) cg-prop: Strategy "replace"
(00.000382) cg-prop: Property "net_cls.classid"
(00.000385) cg-prop: Parsing controller "net_prio"
(00.000388) cg-prop: Strategy "replace"
(00.000391) cg-prop: Property "net_prio.ifpriomap"
(00.000394) cg-prop: Parsing controller "pids"
(00.000397) cg-prop: Strategy "replace"
(00.000400) cg-prop: Property "pids.max"
(00.000403) cg-prop: Parsing controller "devices"
(00.000406) cg-prop: Strategy "replace"
(00.000408) cg-prop: Property "devices.list"
(00.000438) Perparing image inventory (version 1)
(00.000460) Add pid ns 1 pid 14924
(00.000468) Add net ns 2 pid 14924
(00.000476) Add ipc ns 3 pid 14924
(00.000482) Add uts ns 4 pid 14924
(00.000489) Add mnt ns 5 pid 14924
(00.000496) Add user ns 6 pid 14924
(00.000502) Add cgroup ns 7 pid 14924
(00.000506) cg: Dumping cgroups for 14924
(00.000533) cg: `- New css ID 1
(00.000538) cg: `- [blkio] -> [/] [0]
(00.000541) cg: `- [cpu,cpuacct] -> [/] [0]
(00.000544) cg: `- [cpuset] -> [/] [0]
(00.000547) cg: `- [devices] -> [/] [0]
(00.000550) cg: `- [freezer] -> [/] [0]
(00.000553) cg: `- [hugetlb] -> [/] [0]
(00.000556) cg: `- [memory] -> [/] [0]
(00.000559) cg: `- [name=systemd] -> [/] [0]
(00.000562) cg: `- [net_cls,net_prio] -> [/] [0]
(00.000565) cg: `- [perf_event] -> [/] [0]
(00.000567) cg: `- [pids] -> [/] [0]
(00.000570) cg: `- [rdma] -> [/] [0]
(00.000573) cg: Set 1 is criu one
(00.000587) freezing processes: 100000 attempts with 100 ms steps
(00.000607) freezer.state=THAWED
(00.000629) freezer.state=FREEZING
(00.100700) freezer.state=FROZEN
(00.100720) freezing processes: 1 attempts done
(00.100781) SEIZE 14004: success
(00.100794) SEIZE 14248: success
(00.100805) SEIZE 14275: success
(00.100814) SEIZE 14373: success
(00.100823) SEIZE 14384: success
(00.101018) Seized task 14004, state 1
(00.101031) seccomp: Collected tid_real 14004 mode 0x2
(00.101061) Collected (0 attempts, 0 in_progress)
(00.101096) Seized task 14248, state 0
(00.101165) seccomp: Collected tid_real 14248 mode 0x2
(00.101189) Collected (0 attempts, 0 in_progress)
(00.101216) Collected (0 attempts, 0 in_progress)
(00.101240) Collected 14248 in 1 state
(00.101247) Seized task 14275, state 0
(00.101313) seccomp: Collected tid_real 14275 mode 0x2
(00.101342) Collected (0 attempts, 0 in_progress)
(00.101369) Collected (0 attempts, 0 in_progress)
(00.101378) Collected 14275 in 1 state
(00.101384) Seized task 14373, state 0
(00.101451) seccomp: Collected tid_real 14373 mode 0x2
(00.101479) Collected (0 attempts, 0 in_progress)
(00.101506) Collected (0 attempts, 0 in_progress)
(00.101514) Collected 14373 in 1 state
(00.101520) Seized task 14384, state 0
(00.101584) seccomp: Collected tid_real 14384 mode 0x2
(00.101608) Collected (0 attempts, 0 in_progress)
(00.101634) Collected (0 attempts, 0 in_progress)
(00.101646) Collected 14384 in 1 state
(00.101681) Collected (-1 attempts, 0 in_progress)
(00.101692) Collected 14004 in 1 state
(00.101718) Will take pid namespace in the image
(00.101725) Add pid ns 8 pid 14004
(00.101737) Will take net namespace in the image
(00.101743) Add net ns 9 pid 14004
(00.101770) Will take ipc namespace in the image
(00.101777) Add ipc ns 10 pid 14004
(00.101788) Will take uts namespace in the image
(00.101794) Add uts ns 11 pid 14004
(00.101805) Will take mnt namespace in the image
(00.101811) Add mnt ns 12 pid 14004
(00.101822) Will take user namespace in the image
(00.101828) Add user ns 13 pid 14004
(00.101839) Will take cgroup namespace in the image
(00.101845) Add cgroup ns 14 pid 14004
(00.102066) Lock network
(00.102075) Running network-lock scripts
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.106239) id_map: 0 1000000 1000000000
(00.106262) id_map: 0 1000000 1000000000
(00.107183) type btrfs source /dev/loop12 mnt_id 3238 s_dev 0x42 /containers/alpineedge/rootfs @ ./ flags 0x280000 options ssd,space_cache,user_subvol_rm_allowed,subvolid=349,subvol=/containers/alpineedge/rootfs
(00.107199) type tmpfs source none mnt_id 3239 s_dev 0xa2 / @ ./dev flags 0x200000 options size=492k,mode=755,uid=0,gid=0
(00.107208) type proc source proc mnt_id 3240 s_dev 0xa1 / @ ./proc flags 0x20000e options
(00.107215) type sysfs source sysfs mnt_id 3241 s_dev 0xa3 / @ ./sys flags 0x200000 options
(00.107224) type devtmpfs source dev mnt_id 3242 s_dev 0x6 /fuse @ ./dev/fuse flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107232) type devtmpfs source dev mnt_id 3243 s_dev 0x6 /net/tun @ ./dev/net/tun flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107240) type binfmt_misc source binfmt_misc mnt_id 3244 s_dev 0x2b / @ ./proc/sys/fs/binfmt_misc flags 0x28000e options
(00.107248) type fusectl source fusectl mnt_id 3245 s_dev 0x3a / @ ./sys/fs/fuse/connections flags 0x28000e options
(00.107255) type pstore source pstore mnt_id 3246 s_dev 0x1c / @ ./sys/fs/pstore flags 0x28000e options
(00.107294) type configfs source configfs mnt_id 3247 s_dev 0x14 / @ ./sys/kernel/config flags 0x28000e options
(00.107302) type debugfs source debugfs mnt_id 3248 s_dev 0x7 / @ ./sys/kernel/debug flags 0x28000e options
(00.107309) type securityfs source securityfs mnt_id 3249 s_dev 0x8 / @ ./sys/kernel/security flags 0x28000e options
(00.107316) type sysfs source sys mnt_id 3250 s_dev 0x15 /kernel/tracing @ ./sys/kernel/tracing flags 0x28000e options
(00.107322) type mqueue source mqueue mnt_id 3251 s_dev 0x13 / @ ./dev/mqueue flags 0x28000e options
(00.107329) type tmpfs source tmpfs mnt_id 3252 s_dev 0x3e / @ ./dev/lxd flags 0x200000 options size=100k,mode=755
(00.107336) type tmpfs source tmpfs mnt_id 3253 s_dev 0x3d /alpineedge @ ./dev/.lxd-mounts flags 0x280000 options size=100k,mode=711
(00.107343) type fuse source lxcfs mnt_id 3254 s_dev 0x39 /proc/cpuinfo @ ./proc/cpuinfo flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107354) type fuse source lxcfs mnt_id 3255 s_dev 0x39 /proc/diskstats @ ./proc/diskstats flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107363) type fuse source lxcfs mnt_id 3256 s_dev 0x39 /proc/loadavg @ ./proc/loadavg flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107390) type fuse source lxcfs mnt_id 3257 s_dev 0x39 /proc/meminfo @ ./proc/meminfo flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107399) type fuse source lxcfs mnt_id 3258 s_dev 0x39 /proc/stat @ ./proc/stat flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107408) type fuse source lxcfs mnt_id 3259 s_dev 0x39 /proc/swaps @ ./proc/swaps flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107416) type fuse source lxcfs mnt_id 3260 s_dev 0x39 /proc/uptime @ ./proc/uptime flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107425) type fuse source lxcfs mnt_id 3261 s_dev 0x39 /sys/devices/system/cpu/online @ ./sys/devices/system/cpu/online flags 0x280006 options user_id=0,group_id=0,allow_other
(00.107435) type devtmpfs source dev mnt_id 3262 s_dev 0x6 /full @ ./dev/full flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107444) type devtmpfs source dev mnt_id 3263 s_dev 0x6 /null @ ./dev/null flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107468) type devtmpfs source dev mnt_id 3264 s_dev 0x6 /random @ ./dev/random flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107476) type devtmpfs source dev mnt_id 3265 s_dev 0x6 /tty @ ./dev/tty flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107483) type devtmpfs source dev mnt_id 3266 s_dev 0x6 /urandom @ ./dev/urandom flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107490) type devtmpfs source dev mnt_id 3267 s_dev 0x6 /zero @ ./dev/zero flags 0x280002 options size=8172568k,nr_inodes=2043142,mode=755
(00.107498) type devpts source devpts mnt_id 3268 s_dev 0x31 /2 @ ./dev/console flags 0x200000 options gid=-1,mode=620,ptmxmode=666
(00.107514) type tmpfs source none mnt_id 2977 s_dev 0xa2 /.lxc-boot-id @ ./proc/sys/kernel/random/boot_id flags 0x20000f options size=492k,mode=755,uid=0,gid=0
(00.107523) type devpts source devpts mnt_id 2978 s_dev 0xa4 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666,max=1024
(00.107533) type devpts source devpts mnt_id 2979 s_dev 0xa4 /ptmx @ ./dev/ptmx flags 0x20000a options gid=5,mode=620,ptmxmode=666,max=1024
(00.107542) type tmpfs source tmpfs mnt_id 1111 s_dev 0xa5 / @ ./run flags 0x200004 options size=1643464k,mode=755,uid=0,gid=0
(00.107548) mnt: Building mountpoints tree
(00.107551) mnt: Building plain mount tree
(00.107554) mnt: Working on 1111->3238
(00.107557) mnt: Working on 2979->3239
(00.107560) mnt: Working on 2978->3239
(00.107562) mnt: Working on 2977->3240
(00.107565) mnt: Working on 3268->3239
(00.107568) mnt: Working on 3267->3239
(00.107571) mnt: Working on 3266->3239
(00.107573) mnt: Working on 3265->3239
(00.107576) mnt: Working on 3264->3239
(00.107588) mnt: Working on 3263->3239
(00.107591) mnt: Working on 3262->3239
(00.107594) mnt: Working on 3261->3241
(00.107596) mnt: Working on 3260->3240
(00.107599) mnt: Working on 3259->3240
(00.107602) mnt: Working on 3258->3240
(00.107605) mnt: Working on 3257->3240
(00.107607) mnt: Working on 3256->3240
(00.107610) mnt: Working on 3255->3240
(00.107613) mnt: Working on 3254->3240
(00.107615) mnt: Working on 3253->3239
(00.107618) mnt: Working on 3252->3239
(00.107621) mnt: Working on 3251->3239
(00.107623) mnt: Working on 3250->3241
(00.107626) mnt: Working on 3249->3241
(00.107629) mnt: Working on 3248->3241
(00.107631) mnt: Working on 3247->3241
(00.107634) mnt: Working on 3246->3241
(00.107637) mnt: Working on 3245->3241
(00.107639) mnt: Working on 3244->3240
(00.107642) mnt: Working on 3243->3239
(00.107645) mnt: Working on 3242->3239
(00.107647) mnt: Working on 3241->3238
(00.107650) mnt: Working on 3240->3238
(00.107653) mnt: Working on 3239->3238
(00.107656) mnt: Working on 3238->2976
(00.107658) mnt: Resorting siblings on 3238
(00.107661) mnt: Resorting siblings on 1111
(00.107664) mnt: Resorting siblings on 3241
(00.107668) mnt: Resorting siblings on 3261
(00.107670) mnt: Resorting siblings on 3250
(00.107673) mnt: Resorting siblings on 3249
(00.107676) mnt: Resorting siblings on 3248
(00.107678) mnt: Resorting siblings on 3247
(00.107681) mnt: Resorting siblings on 3246
(00.107684) mnt: Resorting siblings on 3245
(00.107687) mnt: Resorting siblings on 3240
(00.107689) mnt: Resorting siblings on 2977
(00.107692) mnt: Resorting siblings on 3260
(00.107694) mnt: Resorting siblings on 3259
(00.107697) mnt: Resorting siblings on 3258
(00.107700) mnt: Resorting siblings on 3257
(00.107703) mnt: Resorting siblings on 3256
(00.107705) mnt: Resorting siblings on 3255
(00.107708) mnt: Resorting siblings on 3254
(00.107711) mnt: Resorting siblings on 3244
(00.107713) mnt: Resorting siblings on 3239
(00.107716) mnt: Resorting siblings on 2979
(00.107719) mnt: Resorting siblings on 2978
(00.107721) mnt: Resorting siblings on 3268
(00.107724) mnt: Resorting siblings on 3267
(00.107727) mnt: Resorting siblings on 3266
(00.107729) mnt: Resorting siblings on 3265
(00.107732) mnt: Resorting siblings on 3264
(00.107735) mnt: Resorting siblings on 3263
(00.107737) mnt: Resorting siblings on 3262
(00.107740) mnt: Resorting siblings on 3253
(00.107743) mnt: Resorting siblings on 3252
(00.107745) mnt: Resorting siblings on 3251
(00.107748) mnt: Resorting siblings on 3243
(00.107751) mnt: Resorting siblings on 3242
(00.107753) mnt: Done:
(00.107756) mnt: [./](3238->2976)
(00.107759) mnt: [./run](1111->3238)
(00.107762) mnt: <--
(00.107765) mnt: [./sys](3241->3238)
(00.107767) mnt: [./sys/devices/system/cpu/online](3261->3241)
(00.107770) mnt: <--
(00.107773) mnt: [./sys/fs/fuse/connections](3245->3241)
(00.107776) mnt: <--
(00.107778) mnt: [./sys/kernel/tracing](3250->3241)
(00.107781) mnt: <--
(00.107784) mnt: [./sys/kernel/security](3249->3241)
(00.107786) mnt: <--
(00.107789) mnt: [./sys/kernel/debug](3248->3241)
(00.107792) mnt: <--
(00.107794) mnt: [./sys/kernel/config](3247->3241)
(00.107797) mnt: <--
(00.107800) mnt: [./sys/fs/pstore](3246->3241)
(00.107802) mnt: <--
(00.107805) mnt: <--
(00.107808) mnt: [./proc](3240->3238)
(00.107810) mnt: [./proc/sys/kernel/random/boot_id](2977->3240)
(00.107813) mnt: <--
(00.107816) mnt: [./proc/sys/fs/binfmt_misc](3244->3240)
(00.107818) mnt: <--
(00.107821) mnt: [./proc/uptime](3260->3240)
(00.107824) mnt: <--
(00.107826) mnt: [./proc/swaps](3259->3240)
(00.107829) mnt: <--
(00.107832) mnt: [./proc/stat](3258->3240)
(00.107834) mnt: <--
(00.107837) mnt: [./proc/meminfo](3257->3240)
(00.107840) mnt: <--
(00.107842) mnt: [./proc/loadavg](3256->3240)
(00.107845) mnt: <--
(00.107848) mnt: [./proc/diskstats](3255->3240)
(00.107850) mnt: <--
(00.107853) mnt: [./proc/cpuinfo](3254->3240)
(00.107856) mnt: <--
(00.107858) mnt: <--
(00.107866) mnt: [./dev](3239->3238)
(00.107869) mnt: [./dev/net/tun](3243->3239)
(00.107871) mnt: <--
(00.107874) mnt: [./dev/ptmx](2979->3239)
(00.107877) mnt: <--
(00.107879) mnt: [./dev/pts](2978->3239)
(00.107882) mnt: <--
(00.107885) mnt: [./dev/console](3268->3239)
(00.107887) mnt: <--
(00.107890) mnt: [./dev/zero](3267->3239)
(00.107893) mnt: <--
(00.107895) mnt: [./dev/urandom](3266->3239)
(00.107898) mnt: <--
(00.107900) mnt: [./dev/tty](3265->3239)
(00.107903) mnt: <--
(00.107906) mnt: [./dev/random](3264->3239)
(00.107908) mnt: <--
(00.107911) mnt: [./dev/null](3263->3239)
(00.107914) mnt: <--
(00.107916) mnt: [./dev/full](3262->3239)
(00.107919) mnt: <--
(00.107922) mnt: [./dev/.lxd-mounts](3253->3239)
(00.107924) mnt: <--
(00.107927) mnt: [./dev/lxd](3252->3239)
(00.107930) mnt: <--
(00.107932) mnt: [./dev/mqueue](3251->3239)
(00.107935) mnt: <--
(00.107938) mnt: [./dev/fuse](3242->3239)
(00.107940) mnt: <--
(00.107943) mnt: <--
(00.107945) mnt: <--
(00.107997) type ext4 source /dev/mapper/fallback--os-root mnt_id 1091 s_dev 0xfe00000 / @ ./var/lib/snapd/hostfs flags 0x80400 options
(00.108005) type ext4 source /dev/mapper/fallback--os-usr mnt_id 1092 s_dev 0xfe00001 / @ ./var/lib/snapd/hostfs/usr flags 0x80400 options
(00.108013) type ext4 source /dev/sda2 mnt_id 1093 s_dev 0x800002 /arch_home_links/usr/share/keepass/Plugins @ ./var/lib/snapd/hostfs/usr/share/keepass/Plugins flags 0x80400 options data=ordered
.... a lot of similar records ...
(00.111877) mnt: Found console mapping for ./dev/console mountpoint
(00.111883) mnt: autodetected external mount /dev/zero for ./dev/zero
(00.111889) mnt: autodetected external mount /dev/urandom for ./dev/urandom
(00.111893) mnt: autodetected external mount /dev/tty for ./dev/tty
(00.111898) mnt: autodetected external mount /dev/random for ./dev/random
(00.111903) mnt: autodetected external mount /dev/null for ./dev/null
(00.111907) mnt: autodetected external mount /dev/full for ./dev/full
(00.111912) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/sys/devices/system/cpu/onli
ne for ./sys/devices/system/cpu/online
(00.111917) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/proc/uptime for ./proc/upti
me
(00.111921) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/proc/swaps for ./proc/swaps
(00.111925) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/proc/stat for ./proc/stat
(00.111929) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/proc/meminfo for ./proc/mem
info
(00.111934) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/proc/loadavg for ./proc/loadavg
(00.111938) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/proc/diskstats for ./proc/diskstats
(00.111942) mnt: autodetected external mount /var/snap/lxd/common/shmounts/lxcfs/proc/cpuinfo for ./proc/cpuinfo
(00.111946) mnt: autodetected external mount /var/snap/lxd/common/shmounts/containers/alpineedge for ./dev/.lxd-mounts
(00.111949) mnt: Found dev/lxd mapping for ./dev/lxd mountpoint
(00.111952) mnt: Found dev/mqueue mapping for ./dev/mqueue mountpoint
(00.111955) mnt: Found sys/kernel/tracing mapping for ./sys/kernel/tracing mountpoint
(00.111958) mnt: Found sys/kernel/security mapping for ./sys/kernel/security mountpoint
(00.111961) mnt: Found sys/kernel/debug mapping for ./sys/kernel/debug mountpoint
(00.111964) mnt: Found sys/kernel/config mapping for ./sys/kernel/config mountpoint
(00.111967) mnt: Found sys/fs/pstore mapping for ./sys/fs/pstore mountpoint
(00.111970) mnt: Found sys/fs/fuse/connections mapping for ./sys/fs/fuse/connections mountpoint
(00.111973) mnt: Found proc/sys/fs/binfmt_misc mapping for ./proc/sys/fs/binfmt_misc mountpoint
(00.111976) mnt: Found dev/net/tun mapping for ./dev/net/tun mountpoint
(00.111979) mnt: Found dev/fuse mapping for ./dev/fuse mountpoint
(00.111986) mnt: Inspecting sharing on 1111 shared_id 0 master_id 0 (@./run)
(00.111989) mnt: Inspecting sharing on 2979 shared_id 0 master_id 0 (@./dev/ptmx)
(00.111993) mnt: The mount 2978 is bind for 2979 (@./dev/pts -> @./dev/ptmx)
(00.111996) mnt: Inspecting sharing on 2978 shared_id 0 master_id 0 (@./dev/pts)
(00.111999) mnt: Inspecting sharing on 2977 shared_id 0 master_id 0 (@./proc/sys/kernel/random/boot_id)
(00.112002) mnt: The mount 3239 is bind for 2977 (@./dev -> @./proc/sys/kernel/random/boot_id)
(00.112005) mnt: Inspecting sharing on 3268 shared_id 0 master_id 0 (@./dev/console)
(00.112008) mnt: Inspecting sharing on 3267 shared_id 0 master_id 3 (@./dev/zero)
(00.112011) mnt: Inspecting sharing on 3266 shared_id 0 master_id 3 (@./dev/urandom)
(00.112014) mnt: Inspecting sharing on 3265 shared_id 0 master_id 3 (@./dev/tty)
(00.112017) mnt: Inspecting sharing on 3264 shared_id 0 master_id 3 (@./dev/random)
(00.112021) mnt: Inspecting sharing on 3263 shared_id 0 master_id 3 (@./dev/null)
(00.112024) mnt: Inspecting sharing on 3262 shared_id 0 master_id 3 (@./dev/full)
(00.112027) mnt: Inspecting sharing on 3261 shared_id 0 master_id 602 (@./sys/devices/system/cpu/online)
(00.112030) mnt: Inspecting sharing on 3260 shared_id 0 master_id 602 (@./proc/uptime)
(00.112033) mnt: Inspecting sharing on 3259 shared_id 0 master_id 602 (@./proc/swaps)
(00.112036) mnt: Inspecting sharing on 3258 shared_id 0 master_id 602 (@./proc/stat)
(00.112045) mnt: Inspecting sharing on 3257 shared_id 0 master_id 602 (@./proc/meminfo)
(00.112048) mnt: Inspecting sharing on 3256 shared_id 0 master_id 602 (@./proc/loadavg)
(00.112051) mnt: Inspecting sharing on 3255 shared_id 0 master_id 602 (@./proc/diskstats)
(00.112054) mnt: Inspecting sharing on 3254 shared_id 0 master_id 602 (@./proc/cpuinfo)
(00.112057) mnt: Inspecting sharing on 3253 shared_id 0 master_id 636 (@./dev/.lxd-mounts)
(00.112060) mnt: Inspecting sharing on 3252 shared_id 0 master_id 0 (@./dev/lxd)
(00.112063) mnt: Inspecting sharing on 3251 shared_id 0 master_id 29 (@./dev/mqueue)
(00.112066) mnt: Inspecting sharing on 3250 shared_id 0 master_id 7 (@./sys/kernel/tracing)
(00.112069) mnt: Inspecting sharing on 3249 shared_id 0 master_id 8 (@./sys/kernel/security)
(00.112072) mnt: Inspecting sharing on 3248 shared_id 0 master_id 27 (@./sys/kernel/debug)
(00.112075) mnt: Inspecting sharing on 3247 shared_id 0 master_id 65 (@./sys/kernel/config)
(00.112078) mnt: Inspecting sharing on 3246 shared_id 0 master_id 12 (@./sys/fs/pstore)
(00.112081) mnt: Inspecting sharing on 3245 shared_id 0 master_id 603 (@./sys/fs/fuse/connections)
(00.112084) mnt: Inspecting sharing on 3244 shared_id 0 master_id 30 (@./proc/sys/fs/binfmt_misc)
(00.112087) mnt: Inspecting sharing on 3243 shared_id 0 master_id 3 (@./dev/net/tun)
(00.112090) mnt: The mount 3242 is bind for 3243 (@./dev/fuse -> @./dev/net/tun)
(00.112093) mnt: Inspecting sharing on 3242 shared_id 0 master_id 3 (@./dev/fuse)
(00.112096) mnt: Inspecting sharing on 3241 shared_id 0 master_id 0 (@./sys)
(00.112099) mnt: Inspecting sharing on 3240 shared_id 0 master_id 0 (@./proc)
(00.112102) mnt: Inspecting sharing on 3239 shared_id 0 master_id 0 (@./dev)
(00.112104) mnt: Inspecting sharing on 3238 shared_id 0 master_id -1 (@./)
(00.112109) Collecting netns 9/14004
(00.112113) Switching to 14004's net for collecting sockets
(00.112221) unix: Collected: ino 245695 peer_ino 0 family 1 type 2 state 7 name /dev/log
(00.112228) unix: Collected: ino 247377 peer_ino 245695 family 1 type 2 state 7 name (null)
(00.112232) unix: Collected: ino 257362 peer_ino 0 family 1 type 5 state 7 name (null)
(00.113211) netlink: Collect netlink sock 0x3d969
(00.113217) netlink: Collect netlink sock 0x3d98f
(00.113221) netlink: Collect netlink sock 0x3ed51
(00.113225) netlink: Collect netlink sock 0x3d96a
(00.113228) netlink: Collect netlink sock 0x3d96d
(00.113231) netlink: Collect netlink sock 0x3d9a4
(00.113233) netlink: Collect netlink sock 0x3d96b
(00.113236) netlink: Collect netlink sock 0x3d96c
(00.113254) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 14004
(00.113259) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 14248
(00.113265) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 14275
(00.113270) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 14373
(00.113276) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 14384
(00.113307) ========================================
(00.113311) Dumping task (pid: 14004)
(00.113314) ========================================
(00.113316) Obtaining task stat ...
(00.113342)
(00.113345) Collecting mappings (pid: 14004)
(00.113348) ----------------------------------------
(00.113437) Dumping path for -3 fd via self 13 [/bin/busybox]
(00.113482) vma 555cd1d7c000 borrows vfi from previous 555cd1d70000
(00.113493) vma 555cd1e18000 borrows vfi from previous 555cd1d7c000
(00.113502) vma 555cd1e3a000 borrows vfi from previous 555cd1e18000
(00.113510) vma 555cd1e3e000 borrows vfi from previous 555cd1e3a000
(00.113539) Dumping path for -3 fd via self 13 [/lib/ld-musl-x86_64.so.1]
(00.113573) vma 7f46a421a000 borrows vfi from previous 7f46a4205000
(00.113582) vma 7f46a4261000 borrows vfi from previous 7f46a421a000
(00.113591) vma 7f46a4296000 borrows vfi from previous 7f46a4261000
(00.113599) vma 7f46a4297000 borrows vfi from previous 7f46a4296000
(00.113636) Collected, longest area occupies 156 pages
(00.113640) 0x555cd1d70000-0x555cd1d7c000 (48K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp shmid: 0x1
(00.113650) 0x555cd1d7c000-0x555cd1e18000 (624K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0xc000 reg fp shmid: 0x1
(00.113654) 0x555cd1e18000-0x555cd1e39000 (132K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0xa8000 reg fp shmid: 0x1
(00.113658) 0x555cd1e3a000-0x555cd1e3e000 (16K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0xc9000 reg fp shmid: 0x1
(00.113661) 0x555cd1e3e000-0x555cd1e3f000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0xcd000 reg fp shmid: 0x1
(00.113665) 0x7f46a4205000-0x7f46a421a000 (84K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp shmid: 0x2
(00.113668) 0x7f46a421a000-0x7f46a4261000 (284K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0x15000 reg fp shmid: 0x2
(00.113671) 0x7f46a4261000-0x7f46a4295000 (208K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x5c000 reg fp shmid: 0x2
(00.113674) 0x7f46a4296000-0x7f46a4297000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x90000 reg fp shmid: 0x2
(00.113678) 0x7f46a4297000-0x7f46a4298000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x91000 reg fp shmid: 0x2
(00.113681) 0x7f46a4298000-0x7f46a429b000 (12K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap shmid: 0
(00.113684) 0x7fff6710b000-0x7fff6712c000 (132K) prot 0x3 flags 0x122 fdflags 0 st 0x201 off 0 reg ap shmid: 0
(00.113688) 0x7fff6717c000-0x7fff6717f000 (12K) prot 0x1 flags 0x22 fdflags 0 st 0x1201 off 0 reg vvar ap shmid: 0
(00.113691) 0x7fff6717f000-0x7fff67180000 (4K) prot 0x5 flags 0x22 fdflags 0 st 0x209 off 0 reg vdso ap shmid: 0
(00.113694) ----------------------------------------
(00.113697)
(00.113700) Collecting fds (pid: 14004)
(00.113702) ----------------------------------------
(00.113721) Found 4 file descriptors
(00.113724) ----------------------------------------
(00.113733) Dump private signals of 14004
(00.113738) Dump shared signals of 14004
(00.113748) Parasite syscall_ip at 0x555cd1d7c000
(00.113890) Set up parasite blob using memfd
(00.113896) Putting parasite blob into 0x7f93b4a58000->0x7f46a41f8000
(00.113955) Dumping general registers for 14004 in native mode
(00.113959) Dumping GP/FPU registers for 14004
(00.113968) x86: xsave runtime structure
(00.113971) x86: -----------------------
(00.113974) x86: cwd:0 swd:0 twd:0 fop:0 mxcsr:0 mxcsr_mask:0
(00.113977) x86: magic1:0x46505853 extended_size:836 xstate_bv:0x7 xstate_size:832
(00.113980) x86: xstate_bv: 0x7
(00.113983) x86: -----------------------
(00.113986) Putting tsock into pid 14004
(00.114047) Wait for parasite being daemonized...
(00.114052) Wait for ack 2 on daemon socket
pie: 1: Running daemon thread leader
pie: 1: __sent ack msg: 2 2 0
(00.114085) Fetched ack: 2 2 0
pie: 1: Daemon waits for command
(00.114089) Parasite 14004 has been switched to daemon mode
(00.114100) Sent msg to daemon 72 0 0
pie: 1: __fetched msg: 72 0 0
(00.114133) Wait for ack 72 on daemon socket
pie: 1: __sent ack msg: 72 72 0
(00.114137) Fetched ack: 72 72 0
pie: 1: Daemon waits for command
(00.114144) vdso: vDSO hint is reliable - omit checking
(00.114154) Sent msg to daemon 74 0 0
(00.114157) Wait for ack 74 on daemon socket
pie: 1: __fetched msg: 74 0 0
pie: 1: __sent ack msg: 74 74 0
(00.114171) Fetched ack: 74 74 0
pie: 1: Daemon waits for command
(00.114179) Sent msg to daemon 70 0 0
pie: 1: __fetched msg: 70 0 0
(00.114184) Wait for ack 70 on daemon socket
pie: 1: __sent ack msg: 70 70 0
(00.114200) Fetched ack: 70 70 0
pie: 1: Daemon waits for command
(00.114207) sid=1 pgid=1 pid=1
(00.114233)
(00.114237) Dumping opened files (pid: 14004)
(00.114239) ----------------------------------------
(00.114246) Sent msg to daemon 71 0 0
pie: 1: __fetched msg: 71 0 0
(00.114277) Wait for ack 71 on daemon socket
pie: 1: __sent ack msg: 71 71 0
(00.114281) Fetched ack: 71 71 0
pie: 1: Daemon waits for command
(00.114305) 14004 fdinfo 0: pos: 0 flags: 100002/0
(00.114319) tty: Dumping tty 16 with id 0x3
(00.114326) Sent msg to daemon 73 0 0
(00.114329) Wait for ack 73 on daemon socket
pie: 1: __fetched msg: 73 0 0
pie: 1: __sent ack msg: 73 73 0
(00.114356) Fetched ack: 73 73 0
pie: 1: Daemon waits for command
(00.114429) 14004 fdinfo 1: pos: 0 flags: 100002/0
(00.114454) 14004 fdinfo 2: pos: 0 flags: 100002/0
(00.114472) 14004 fdinfo 14: pos: 0x40b flags: 100000/0
(00.114481) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/13979/status
(00.114486) ----------------------------------------
(00.114498) Error (criu/cr-dump.c:1345): Dump files (pid: 14004) failed with -1
(00.114521) Waiting for 14004 to trap
(00.114534) Daemon 14004 exited trapping
(00.114542) Sent msg to daemon 3 0 0
(00.114548) Force no-breakpoints restore
(00.114561) 14004 was trapped
(00.114566) 14004 (native) is going to execute the syscall 45, required is 15
(00.114580) 14004 was trapped
(00.114583) `- Expecting exit
(00.114593) 14004 was trapped
(00.114599) 14004 (native) is going to execute the syscall 186, required is 15
(00.114609) 14004 was trapped
(00.114612) `- Expecting exit
(00.114623) 14004 was trapped
(00.114628) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: __fetched msg: 3 0 0
(00.114652) 14004 was trapped
(00.114655) `- Expecting exit
(00.114669) 14004 was trapped
(00.114674) 14004 (native) is going to execute the syscall 186, required is 15
(00.114685) 14004 was trapped
(00.114688) `- Expecting exit
(00.114698) 14004 was trapped
(00.114703) 14004 (native) is going to execute the syscall 186, required is 15
(00.114713) 14004 was trapped
(00.114716) `- Expecting exit
(00.114727) 14004 was trapped
(00.114732) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: 1: new_sp=0x7f46a41ff008 ip 0x7f46a4258872
(00.114749) 14004 was trapped
(00.114752) `- Expecting exit
(00.114762) 14004 was trapped
(00.114767) 14004 (native) is going to execute the syscall 3, required is 15
(00.114787) 14004 was trapped
(00.114790) `- Expecting exit
(00.114800) 14004 was trapped
(00.114806) 14004 (native) is going to execute the syscall 3, required is 15
(00.114817) 14004 was trapped
(00.114820) `- Expecting exit
(00.114580) 14004 was trapped
(00.114583) `- Expecting exit
(00.114593) 14004 was trapped
(00.114599) 14004 (native) is going to execute the syscall 186, required is 15
(00.114609) 14004 was trapped
(00.114612) `- Expecting exit
(00.114623) 14004 was trapped
(00.114628) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: __fetched msg: 3 0 0
(00.114652) 14004 was trapped
(00.114655) `- Expecting exit
(00.114669) 14004 was trapped
(00.114674) 14004 (native) is going to execute the syscall 186, required is 15
(00.114685) 14004 was trapped
(00.114688) `- Expecting exit
(00.114698) 14004 was trapped
(00.114703) 14004 (native) is going to execute the syscall 186, required is 15
(00.114713) 14004 was trapped
(00.114716) `- Expecting exit
(00.114727) 14004 was trapped
(00.114732) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: 1: new_sp=0x7f46a41ff008 ip 0x7f46a4258872
(00.114749) 14004 was trapped
(00.114752) `- Expecting exit
(00.114762) 14004 was trapped
(00.114767) 14004 (native) is going to execute the syscall 3, required is 15
(00.114787) 14004 was trapped
(00.114790) `- Expecting exit
(00.114800) 14004 was trapped
(00.114806) 14004 (native) is going to execute the syscall 3, required is 15
(00.114817) 14004 was trapped
(00.114820) `- Expecting exit
(00.114830) 14004 was trapped
(00.114835) 14004 (native) is going to execute the syscall 15, required is 15
(00.114848) 14004 was stopped
(00.114956) Unlock network
(00.114961) Running network-unlock scripts
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.118705) Unfreezing tasks into 1
(00.118715) Unseizing 14004 into 1
(00.118721) Unseizing 14248 into 1
(00.118725) Unseizing 14275 into 1
(00.118729) Unseizing 14373 into 1
(00.118734) Unseizing 14384 into 1
(00.118748) Error (criu/cr-dump.c:1743): Dumping FAILED.Now i've tried to make stateful stop after running endless loop in another terminal.
$ lxc exec alpineedge -- /bin/sh -c \
> 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
$ lxc stop alpineedge --stateful
Error: snapshot dump failed
(00.000051) Warn (criu/log.c:203): The early log isn't empty
(00.101030) Warn (criu/seize.c:221): Unexpected process 15631 in the freezer cgroup (status 0x80057f)
(00.101034) Warn (criu/seize.c:221): Unexpected process 15563 in the freezer cgroup (status 0x80057f)
(00.112256) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/13979/status
(00.112289) Error (criu/cr-dump.c:1345): Dump files (pid: 14004) failed with -1
(00.115844) Error (criu/cr-dump.c:1743): Dumping FAILED.
Try `lxc info --show-log alpineedge` for more infoIs there an easy way to link a file here?
$ sudo cat /var/snap/lxd/common/lxd/logs/alpineedge/snapshot_dump_2020-04-06T22:41:08+03:00.log
... a lot of output similar to above ...
(00.110790) ========================================
(00.110792) Dumping task (pid: 14004)
(00.110794) ========================================
(00.110796) Obtaining task stat ...
(00.110815)
(00.110817) Collecting mappings (pid: 14004)
(00.110819) ----------------------------------------
(00.110889) Dumping path for -3 fd via self 13 [/bin/busybox]
(00.110921) vma 555cd1d7c000 borrows vfi from previous 555cd1d70000
(00.110928) vma 555cd1e18000 borrows vfi from previous 555cd1d7c000
(00.110934) vma 555cd1e3a000 borrows vfi from previous 555cd1e18000
(00.110939) vma 555cd1e3e000 borrows vfi from previous 555cd1e3a000
(00.110960) Dumping path for -3 fd via self 13 [/lib/ld-musl-x86_64.so.1]
(00.110985) vma 7f46a421a000 borrows vfi from previous 7f46a4205000
(00.110991) vma 7f46a4261000 borrows vfi from previous 7f46a421a000
(00.110996) vma 7f46a4296000 borrows vfi from previous 7f46a4261000
(00.111007) vma 7f46a4297000 borrows vfi from previous 7f46a4296000
(00.111035) Collected, longest area occupies 156 pages
(00.111038) 0x555cd1d70000-0x555cd1d7c000 (48K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp shmid: 0x1
(00.111041) 0x555cd1d7c000-0x555cd1e18000 (624K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0xc000 reg fp shmid: 0x1
(00.111043) 0x555cd1e18000-0x555cd1e39000 (132K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0xa8000 reg fp shmid: 0x1
(00.111046) 0x555cd1e3a000-0x555cd1e3e000 (16K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0xc9000 reg fp shmid: 0x1
(00.111048) 0x555cd1e3e000-0x555cd1e3f000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0xcd000 reg fp shmid: 0x1
(00.111051) 0x7f46a4205000-0x7f46a421a000 (84K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp shmid: 0x2
(00.111053) 0x7f46a421a000-0x7f46a4261000 (284K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0x15000 reg fp shmid: 0x2
(00.111056) 0x7f46a4261000-0x7f46a4295000 (208K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x5c000 reg fp shmid: 0x2
(00.111058) 0x7f46a4296000-0x7f46a4297000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x90000 reg fp shmid: 0x2
(00.111060) 0x7f46a4297000-0x7f46a4298000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x91000 reg fp shmid: 0x2
(00.111063) 0x7f46a4298000-0x7f46a429b000 (12K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap shmid: 0
(00.111065) 0x7fff6710b000-0x7fff6712c000 (132K) prot 0x3 flags 0x122 fdflags 0 st 0x201 off 0 reg ap shmid: 0
(00.111067) 0x7fff6717c000-0x7fff6717f000 (12K) prot 0x1 flags 0x22 fdflags 0 st 0x1201 off 0 reg vvar ap shmid: 0
(00.111070) 0x7fff6717f000-0x7fff67180000 (4K) prot 0x5 flags 0x22 fdflags 0 st 0x209 off 0 reg vdso ap shmid: 0
(00.111072) ----------------------------------------
(00.111074)
(00.111076) Collecting fds (pid: 14004)
(00.111078) ----------------------------------------
(00.111091) Found 4 file descriptors
(00.111093) ----------------------------------------
(00.111099) Dump private signals of 14004
(00.111102) Dump shared signals of 14004
(00.111111) Parasite syscall_ip at 0x555cd1d7c000
(00.111326) Set up parasite blob using memfd
(00.111346) Putting parasite blob into 0x7fdca4abf000->0x7f46a41f8000
(00.111440) Dumping general registers for 14004 in native mode
(00.111451) Dumping GP/FPU registers for 14004
(00.111471) x86: xsave runtime structure
(00.111477) x86: -----------------------
(00.111482) x86: cwd:0x40 swd:0 twd:0 fop:0 mxcsr:0 mxcsr_mask:0x2ffff
(00.111489) x86: magic1:0x46505853 extended_size:836 xstate_bv:0x7 xstate_size:832
(00.111496) x86: xstate_bv: 0x7
(00.111501) x86: -----------------------
(00.111507) Putting tsock into pid 14004
(00.111597) Wait for parasite being daemonized...
(00.111606) Wait for ack 2 on daemon socket
pie: 1: Running daemon thread leader
pie: 1: __sent ack msg: 2 2 0
pie: 1: Daemon waits for command
(00.111643) Fetched ack: 2 2 0
(00.111653) Parasite 14004 has been switched to daemon mode
(00.111668) Sent msg to daemon 72 0 0
pie: 1: __fetched msg: 72 0 0
pie: 1: __sent ack msg: 72 72 0
(00.111708) Wait for ack 72 on daemon socket
pie: 1: Daemon waits for command
(00.111717) Fetched ack: 72 72 0
(00.111725) vdso: vDSO hint is reliable - omit checking
(00.111736) Sent msg to daemon 74 0 0
pie: 1: __fetched msg: 74 0 0
(00.111742) Wait for ack 74 on daemon socket
pie: 1: __sent ack msg: 74 74 0
(00.111753) Fetched ack: 74 74 0
pie: 1: Daemon waits for command
(00.111763) Sent msg to daemon 70 0 0
pie: 1: __fetched msg: 70 0 0
(00.111769) Wait for ack 70 on daemon socket
pie: 1: __sent ack msg: 70 70 0
(00.111786) Fetched ack: 70 70 0
pie: 1: Daemon waits for command
(00.111794) sid=1 pgid=1 pid=1
(00.111857)
(00.111864) Dumping opened files (pid: 14004)
(00.111871) ----------------------------------------
(00.111883) Sent msg to daemon 71 0 0
pie: 1: __fetched msg: 71 0 0
pie: 1: __sent ack msg: 71 71 0
(00.111910) Wait for ack 71 on daemon socket
pie: 1: Daemon waits for command
(00.111918) Fetched ack: 71 71 0
(00.111954) 14004 fdinfo 0: pos: 0 flags: 100002/0
(00.112015) tty: Dumping tty 16 with id 0x3
(00.112032) Sent msg to daemon 73 0 0
pie: 1: __fetched msg: 73 0 0
(00.112039) Wait for ack 73 on daemon socket
pie: 1: __sent ack msg: 73 73 0
(00.112052) Fetched ack: 73 73 0
pie: 1: Daemon waits for command
(00.112156) 14004 fdinfo 1: pos: 0 flags: 100002/0
(00.112199) 14004 fdinfo 2: pos: 0 flags: 100002/0
(00.112233) 14004 fdinfo 14: pos: 0x40b flags: 100000/0
(00.112256) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/13979/status
(00.112266) ----------------------------------------
(00.112289) Error (criu/cr-dump.c:1345): Dump files (pid: 14004) failed with -1
(00.112335) Waiting for 14004 to trap
(00.112344) Daemon 14004 exited trapping
(00.112359) Sent msg to daemon 3 0 0
(00.112372) Force no-breakpoints restore
(00.112396) 14004 was trapped
(00.112408) 14004 (native) is going to execute the syscall 45, required is 15
(00.112431) 14004 was trapped
(00.112437) `- Expecting exit
(00.112457) 14004 was trapped
(00.112467) 14004 (native) is going to execute the syscall 186, required is 15
(00.112488) 14004 was trapped
(00.112494) `- Expecting exit
(00.112513) 14004 was trapped
(00.112524) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: __fetched msg: 3 0 0
(00.112547) 14004 was trapped
(00.112554) `- Expecting exit
(00.112573) 14004 was trapped
(00.112583) 14004 (native) is going to execute the syscall 186, required is 15
(00.112600) 14004 was trapped
(00.112606) `- Expecting exit
(00.112621) 14004 was trapped
(00.112631) 14004 (native) is going to execute the syscall 186, required is 15
(00.112647) 14004 was trapped
(00.112653) `- Expecting exit
(00.112669) 14004 was trapped
(00.112680) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: 1: new_sp=0x7f46a41ff008 ip 0x7f46a4258872
(00.112702) 14004 was trapped
(00.112708) `- Expecting exit
(00.112723) 14004 was trapped
(00.112733) 14004 (native) is going to execute the syscall 3, required is 15
(00.112759) 14004 was trapped
(00.112431) 14004 was trapped
(00.112437) `- Expecting exit
(00.112457) 14004 was trapped
(00.112467) 14004 (native) is going to execute the syscall 186, required is 15
(00.112488) 14004 was trapped
(00.112494) `- Expecting exit
(00.112513) 14004 was trapped
(00.112524) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: __fetched msg: 3 0 0
(00.112547) 14004 was trapped
(00.112554) `- Expecting exit
(00.112573) 14004 was trapped
(00.112583) 14004 (native) is going to execute the syscall 186, required is 15
(00.112600) 14004 was trapped
(00.112606) `- Expecting exit
(00.112621) 14004 was trapped
(00.112631) 14004 (native) is going to execute the syscall 186, required is 15
(00.112647) 14004 was trapped
(00.112653) `- Expecting exit
(00.112669) 14004 was trapped
(00.112680) 14004 (native) is going to execute the syscall 1, required is 15
pie: 1: 1: new_sp=0x7f46a41ff008 ip 0x7f46a4258872
(00.112702) 14004 was trapped
(00.112708) `- Expecting exit
(00.112723) 14004 was trapped
(00.112733) 14004 (native) is going to execute the syscall 3, required is 15
(00.112759) 14004 was trapped
(00.112765) `- Expecting exit
(00.112781) 14004 was trapped
(00.112791) 14004 (native) is going to execute the syscall 3, required is 15
(00.112808) 14004 was trapped
(00.112814) `- Expecting exit
(00.112829) 14004 was trapped
(00.112839) 14004 (native) is going to execute the syscall 15, required is 15
(00.112859) 14004 was stopped
(00.113044) Unlock network
(00.113052) Running network-unlock scripts
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.115759) Unfreezing tasks into 1
(00.115778) Unseizing 14004 into 1
(00.115788) Unseizing 14248 into 1
(00.115797) Unseizing 14275 into 1
(00.115806) Unseizing 14373 into 1
(00.115817) Unseizing 14384 into 1
(00.115844) Error (criu/cr-dump.c:1743): Dumping FAILED.Ah, I think btrfs does not work with CRIU. I saw that earlier but did not remember it immediately.
https://criu.org/Filesystems_pecularities
Try it with another storage backend.
diabolusss
commented
4 years ago Ok, could you, please, give a clue which storage backend will work with criu reliably? Is lvm a good option?
diabolusss
commented
4 years ago I've tried with LVM as loop device, similar story. Later i'll try with real lvm device.
$ lxc storage create pool1 lvm
$ lxc stop alpineedge --stateful
Error: snapshot dump failed
(00.000038) Warn (criu/log.c:203): The early log isn't empty
(00.118577) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/25248/status
(00.118604) Error (criu/cr-dump.c:1345): Dump files (pid: 25271) failed with -1
(00.123209) Error (criu/cr-dump.c:1743): Dumping FAILED.
Try `lxc info --show-log alpineedge` for more info
$ lxc info --show-log alpineedge
Log:
lxc alpineedge 20200406204941.837 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.alpineedge"
lxc alpineedge 20200406204941.838 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.alpineedge"
lxc alpineedge 20200406204941.839 ERROR utils - utils.c:lxc_can_use_pidfd:1855 - Invalid argument - Kernel does not support waiting on processes through pidfds
lxc alpineedge 20200406204941.840 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1455 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc 20200406205023.984 ERROR criu - criu.c:do_dump:1304 - dump failed with 1
lxc 20200406205023.984 ERROR criu - criu.c:do_dump:1318 - criu output:
$ sudo cat /var/snap/lxd/common/lxd/logs/alpineedge/snapshot_dump_2020-04-06T23:50:23+03:00.log
(00.118275) Dumping opened files (pid: 25271)
(00.118279) ----------------------------------------
(00.118289) Sent msg to daemon 71 0 0
pie: 1: __fetched msg: 71 0 0
pie: 1: __sent ack msg: 71 71 0
pie: 1: Daemon waits for command
(00.118311) Wait for ack 71 on daemon socket
(00.118327) Fetched ack: 71 71 0
(00.118362) 25271 fdinfo 0: pos: 0 flags: 100002/0
(00.118387) tty: Dumping tty 16 with id 0x3
(00.118399) Sent msg to daemon 73 0 0
pie: 1: __fetched msg: 73 0 0
(00.118403) Wait for ack 73 on daemon socket
pie: 1: __sent ack msg: 73 73 0
pie: 1: Daemon waits for command
(00.118409) Fetched ack: 73 73 0
(00.118488) 25271 fdinfo 1: pos: 0 flags: 100002/0
(00.118527) 25271 fdinfo 2: pos: 0 flags: 100002/0
(00.118560) 25271 fdinfo 14: pos: 0x40a flags: 100000/0
(00.118577) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/25248/status
(00.118585) ----------------------------------------
(00.118604) Error (criu/cr-dump.c:1345): Dump files (pid: 25271) failed with -1
(00.118642) Waiting for 25271 to trap
(00.118649) Daemon 25271 exited trapping
(00.118662) Sent msg to daemon 3 0 0
(00.118672) Force no-breakpoints restore
(00.118689) 25271 was trapped
(00.118698) 25271 (native) is going to execute the syscall 45, required is 15
(00.118713) 25271 was trapped
(00.118717) `- Expecting exit
(00.118729) 25271 was trapped
(00.118738) 25271 (native) is going to execute the syscall 186, required is 15
(00.118750) 25271 was trapped
(00.118754) `- Expecting exit
(00.118765) 25271 was trapped
(00.118773) 25271 (native) is going to execute the syscall 1, required is 15
pie: 1: __fetched msg: 3 0 0
(00.118793) 25271 was trapped
(00.118797) `- Expecting exit
(00.118808) 25271 was trapped
(00.118817) 25271 (native) is going to execute the syscall 186, required is 15
(00.118832) 25271 was trapped
(00.118835) `- Expecting exit
(00.118849) 25271 was trapped
(00.118857) 25271 (native) is going to execute the syscall 186, required is 15
(00.118870) 25271 was trapped
(00.118874) `- Expecting exit
(00.118887) 25271 was trapped
(00.118896) 25271 (native) is going to execute the syscall 1, required is 15
pie: 1: 1: new_sp=0x7f0d72c6c008 ip 0x7f0d72cc5872
(00.118910) 25271 was trapped
(00.118914) `- Expecting exit
(00.118925) 25271 was trapped
(00.118934) 25271 (native) is going to execute the syscall 3, required is 15
(00.118953) 25271 was trapped
(00.118957) `- Expecting exit
(00.118968) 25271 was trapped
(00.118977) 25271 (native) is going to execute the syscall 3, required is 15
(00.118989) 25271 was trapped
(00.118992) `- Expecting exit
(00.119004) 25271 was trapped
(00.119012) 25271 (native) is going to execute the syscall 15, required is 15
(00.119028) 25271 was stopped
(00.119188) Unlock network
(00.119194) Running network-unlock scripts
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.123136) Unfreezing tasks into 1
(00.123148) Unseizing 25271 into 1
(00.123158) Unseizing 25515 into 1
(00.123167) Unseizing 25542 into 1
(00.123174) Unseizing 25699 into 1
(00.123183) Unseizing 25710 into 1
(00.123209) Error (criu/cr-dump.c:1743): Dumping FAILED.
avagin
commented
4 years ago (00.118577) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/25248/status
Do you know what the process 25248 is? Can you show ls -l /proc/pid/fd for the init process of the target container?
diabolusss
commented
4 years ago No, i don't. And according to a result it's missing.
$ lxc stop alpineedge --stateful
Error: snapshot dump failed
(00.000041) Warn (criu/log.c:203): The early log isn't empty
(00.083673) Error (criu/uffd.c:264): uffd: Lazy pages are not available: Function not implemented
(00.204983) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/2432/status
(00.205009) Error (criu/cr-dump.c:1345): Dump files (pid: 2445) failed with -1
(00.209810) Error (criu/cr-dump.c:1743): Dumping FAILED.
Try `lxc info --show-log alpineedge` for more info
$ lxc exec alpineedge -- ls -l /proc/pid/fd
ls: /proc/pid/fd: No such file or directory
$ lxc exec alpineedge -- ls -l /proc/pid
ls: /proc/pid: No such file or directory
$ lxc exec alpineedge -- ls -l /proc/2432/fd
ls: /proc/2432/fd: No such file or directory
$ lxc exec alpineedge -- ls -l /proc/2432/14
ls: /proc/2432/14: No such file or directory
$ lxc exec alpineedge -- ls -l /proc/2432
ls: /proc/2432: No such file or directory
$ lxc exec alpineedge -- ls -l /proc/
total 0
dr-xr-xr-x 9 root root 0 Apr 7 03:44 1
dr-xr-xr-x 9 root wheel 0 Apr 7 03:44 217
dr-xr-xr-x 9 root root 0 Apr 7 06:48 244
dr-xr-xr-x 9 root root 0 Apr 7 06:48 288
dr-xr-xr-x 9 root root 0 Apr 7 06:48 299
dr-xr-xr-x 9 root root 0 Apr 7 06:50 304
dr-xr-xr-x 2 nobody nobody 0 Apr 7 06:48 acpi
dr-xr-xr-x 6 nobody nobody 0 Apr 7 06:48 asound
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 buddyinfo
dr-xr-xr-x 4 nobody nobody 0 Apr 7 06:48 bus
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 cgroups
-r--r--r-- 1 nobody nobody 0 Apr 7 03:44 cmdline
-r--r--r-- 1 nobody nobody 57506 Apr 7 06:48 config.gz
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 consoles
-r--r--r-- 1 nobody nobody 0 Apr 7 06:50 cpuinfo
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 crypto
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 devices
-r--r--r-- 1 nobody nobody 0 Apr 7 06:50 diskstats
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 dma
dr-xr-xr-x 2 nobody nobody 0 Apr 7 06:48 driver
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 execdomains
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 fb
-r--r--r-- 1 nobody nobody 0 Apr 7 03:44 filesystems
dr-xr-xr-x 7 nobody nobody 0 Apr 7 06:48 fs
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 interrupts
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 iomem
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 ioports
dr-xr-xr-x 78 nobody nobody 0 Apr 7 06:48 irq
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 kallsyms
-r-------- 1 nobody nobody 140737471590400 Apr 7 06:48 kcore
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 key-users
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 keys
-r-------- 1 nobody nobody 0 Apr 7 06:48 kmsg
-r-------- 1 nobody nobody 0 Apr 7 06:48 kpagecgroup
-r-------- 1 nobody nobody 0 Apr 7 06:48 kpagecount
-r-------- 1 nobody nobody 0 Apr 7 06:48 kpageflags
-rw-r--r-- 1 nobody nobody 0 Apr 7 06:48 latency_stats
-r--r--r-- 1 nobody nobody 0 Apr 7 06:50 loadavg
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 locks
-r--r--r-- 1 nobody nobody 0 Apr 7 06:50 meminfo
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 misc
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 modules
lrwxrwxrwx 1 nobody nobody 11 Apr 7 03:44 mounts -> self/mounts
-rw-r--r-- 1 nobody nobody 0 Apr 7 06:48 mtrr
lrwxrwxrwx 1 nobody nobody 8 Apr 7 06:48 net -> self/net
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 pagetypeinfo
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 partitions
dr-xr-xr-x 2 nobody nobody 0 Apr 7 06:48 pressure
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 sched_debug
rw-r--r-- 1 nobody nobody 0 Apr 7 06:48 mtrr
lrwxrwxrwx 1 nobody nobody 8 Apr 7 06:48 net -> self/net
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 pagetypeinfo
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 partitions
dr-xr-xr-x 2 nobody nobody 0 Apr 7 06:48 pressure
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 sched_debug
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 schedstat
dr-xr-xr-x 4 nobody nobody 0 Apr 7 06:48 scsi
lrwxrwxrwx 1 nobody nobody 0 Apr 7 03:44 self -> 304
-r-------- 1 nobody nobody 0 Apr 7 06:48 slabinfo
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 softirqs
dr-xr-xr-x 4 nobody nobody 0 Apr 7 06:48 spl
-r--r--r-- 1 nobody nobody 0 Apr 7 06:50 stat
-r--r--r-- 1 nobody nobody 0 Apr 7 06:50 swaps
dr-xr-xr-x 1 nobody nobody 0 Apr 7 03:44 sys
--w------- 1 nobody nobody 0 Apr 7 06:48 sysrq-trigger
dr-xr-xr-x 2 nobody nobody 0 Apr 7 06:48 sysvipc
lrwxrwxrwx 1 nobody nobody 0 Apr 7 03:44 thread-self -> 304/task/304
-r-------- 1 nobody nobody 0 Apr 7 06:48 timer_list
dr-xr-xr-x 4 nobody nobody 0 Apr 7 06:48 tty
-r--r--r-- 1 nobody nobody 0 Apr 7 06:50 uptime
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 version
-r-------- 1 nobody nobody 0 Apr 7 06:48 vmallocinfo
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 vmstat
-r--r--r-- 1 nobody nobody 0 Apr 7 06:48 zoneinfo$ sudo cat /var/snap/lxd/common/lxd/logs/alpineedge/snapshot_dump_2020-04-07T09:48:54+03:00.log | grep 1225 -n -B 10 -A 10
405-(00.107854) type ext4 source /dev/sda2 mnt_id 1215 s_dev 0x800002 /arch_home_links/home/.config/user-dirs.conf @ ./home/colt/.config/user-dirs.conf flags 0x80400 options data=ordered
406-(00.107875) type ext4 source /dev/sda2 mnt_id 1216 s_dev 0x800002 /arch_home_links/home/.local/share/data/Mega Limited @ ./home/colt/.local/share/data/Mega Limited flags 0x80400 options data=ordered
407-(00.107890) type ext4 source /dev/sda2 mnt_id 1217 s_dev 0x800002 /arch_home_links/home/.config/user-dirs.dirs @ ./home/colt/.config/user-dirs.dirs flags 0x80400 options data=ordered
409-(00.107913) type ext4 source /dev/sda2 mnt_id 1219 s_dev 0x800002 /arch_home_links/home/.local/share/nemo @ ./home/colt/.local/share/nemo flags 0x80400 options data=ordered
410-(00.107959) type ext4 source /dev/sda2 mnt_id 1220 s_dev 0x800002 /arch_home_links/home/.config/autostart @ ./home/colt/.config/autostart flags 0x80400 options data=ordered
411-(00.107971) type ext4 source /dev/sda2 mnt_id 1221 s_dev 0x800002 /arch_home_links/home/.cache/Clementine @ ./home/colt/.cache/Clementine flags 0x80400 options data=ordered
414-(00.108005) type ext4 source /dev/mapper/fallback--os-root mnt_id 1224 s_dev 0xfe00000 /root @ ./root flags 0x80400 options
415:(00.108020) type proc source proc mnt_id 1225 s_dev 0x4 / @ ./proc flags 0x28000e options
416-(00.108036) type autofs source systemd-1 mnt_id 1226 s_dev 0x29 / @ ./proc/sys/fs/binfmt_misc flags 0x280000 options fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19782
417-(00.108052) type binfmt_misc source binfmt_misc mnt_id 1227 s_dev 0x2b / @ ./proc/sys/fs/binfmt_misc flags 0x28000e options
418-(00.108063) type sysfs source sys mnt_id 1228 s_dev 0x15 / @ ./sys flags 0x28000e options
419-(00.108116) type securityfs source securityfs mnt_id 1229 s_dev 0x8 / @ ./sys/kernel/security flags 0x28000e options
420-(00.108135) type tmpfs source tmpfs mnt_id 1230 s_dev 0x19 / @ ./sys/fs/cgroup flags 0x108000f options mode=755
421-(00.108147) type cgroup2 source cgroup2 mnt_id 1231 s_dev 0x1a / @ ./sys/fs/cgroup/unified flags 0x28000e options
422-(00.108163) type cgroup source cgroup mnt_id 1232 s_dev 0x1b / @ ./sys/fs/cgroup/systemd flags 0x28000e options xattr,name=systemd
423-(00.108175) type cgroup source cgroup mnt_id 1233 s_dev 0x1e / @ ./sys/fs/cgroup/cpu,cpuacct flags 0x28000e options cpu,cpuacct
424-(00.108187) type cgroup source cgroup mnt_id 1234 s_dev 0x1f / @ ./sys/fs/cgroup/blkio flags 0x28000e options blkio
425-(00.108198) type cgroup source cgroup mnt_id 1235 s_dev 0x20 / @ ./sys/fs/cgroup/rdma flags 0x28000e options rdma
--Possibly not related question, but why binded folders (with bindfs, line 405-413) on my host are used here as they aren't exposed to my container?
--
1157-(00.112089) mnt: [./sys/fs/cgroup/cpu,cpuacct](1233->1230)
1158-(00.112093) mnt: <--
1159-(00.112096) mnt: [./sys/fs/cgroup/systemd](1232->1230)
1160-(00.112100) mnt: <--
1161-(00.112103) mnt: [./sys/fs/cgroup/unified](1231->1230)
1162-(00.112107) mnt: <--
1163-(00.112110) mnt: <--
1164-(00.112113) mnt: [./sys/kernel/security](1229->1228)
1165-(00.112117) mnt: <--
1166-(00.112120) mnt: <--
1167:(00.112123) mnt: [./proc](1225->1190)
1168:(00.112127) mnt: [./proc/sys/fs/binfmt_misc](1226->1225)
1169-(00.112130) mnt: [./proc/sys/fs/binfmt_misc](1227->1226)
1170-(00.112134) mnt: <--
1171-(00.112137) mnt: <--
1172-(00.112140) mnt: <--
1173-(00.112144) mnt: [./root](1224->1190)
1174-(00.112147) mnt: <--
1175-(00.112150) mnt: [./home](1199->1190)
1176-(00.112154) mnt: [./home/colt/.local/share/data/Mega Limited](1216->1199)
1177-(00.112158) mnt: <--
1178-(00.112161) mnt: [./home/colt/.local/share/nemo](1219->1199)
--
--
1412-(00.116576) tty: Dumping tty 16 with id 0x3
1413-(00.116587) Sent msg to daemon 73 0 0
1414-pie: 1: __fetched msg: 73 0 0
1415-(00.116591) Wait for ack 73 on daemon socket
1416-pie: 1: __sent ack msg: 73 73 0
1417-pie: 1: Daemon waits for command
1418-(00.116597) Fetched ack: 73 73 0
1419-(00.116672) 2445 fdinfo 1: pos: 0 flags: 100002/0
1420-(00.116709) 2445 fdinfo 2: pos: 0 flags: 100002/0
1421-(00.116740) 2445 fdinfo 14: pos: 0x404 flags: 100000/0
1422:(00.116758) Error (criu/files-reg.c:1297): Can't lookup mount=1225 for fd=14 path=/proc/2432/status
1423-(00.116765) ----------------------------------------
1424-(00.116792) Error (criu/cr-dump.c:1345): Dump files (pid: 2445) failed with -1
1425-(00.116829) Waiting for 2445 to trap
1426-(00.116836) Daemon 2445 exited trapping
1427-(00.116849) Sent msg to daemon 3 0 0
1428-(00.116858) Force no-breakpoints restore
1429-(00.116876) 2445 was trapped
1430-(00.116884) 2445 (native) is going to execute the syscall 45, required is 15
1431-(00.116908) 2445 was trapped
1432-(00.116912) `- Expecting exit$ lxc exec alpineedge -- cat /proc/mounts
/dev/pool1/containers_alpineedge / ext4 rw,relatime,discard,stripe=16 0 0
none /dev tmpfs rw,relatime,size=492k,mode=755,uid=1000000,gid=1000000 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
dev /dev/fuse devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
dev /dev/net/tun devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
sys /sys/kernel/tracing sysfs rw,nosuid,nodev,noexec,relatime 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/lxd tmpfs rw,relatime,size=100k,mode=755 0 0
tmpfs /dev/.lxd-mounts tmpfs rw,relatime,size=100k,mode=711 0 0
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /sys/devices/system/cpu/online fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
dev /dev/full devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
dev /dev/null devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
dev /dev/random devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
dev /dev/tty devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
dev /dev/urandom devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
dev /dev/zero devtmpfs rw,nosuid,relatime,size=8172576k,nr_inodes=2043144,mode=755 0 0
devpts /dev/console devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0
none /proc/sys/kernel/random/boot_id tmpfs ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,uid=1000000,gid=1000000 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/ptmx devpts rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666,max=1024 0 0
tmpfs /run tmpfs rw,nodev,relatime,size=1643468k,mode=755,uid=1000000,gid=1000000 0 0If you are doing it from the inside of the container please show us:
lxc exec alpineedge -- ls -la /proc/1/fd
diabolusss
commented
4 years ago lxc exec runs command inside container, so yes.
$ lxc exec alpineedge -- ls -la /proc/1/fd
total 0
dr-x------ 2 root root 0 Apr 7 07:54 .
dr-xr-xr-x 9 root root 0 Apr 7 03:44 ..
lrwx------ 1 root root 64 Apr 7 07:54 0 -> /dev/pts/0
lrwx------ 1 root root 64 Apr 7 07:54 1 -> /dev/pts/0
lr-x------ 1 root root 64 Apr 7 07:54 14 -> /proc/2432/status
lrwx------ 1 root root 64 Apr 7 07:54 2 -> /dev/pts/0
$ lxc exec alpineedge -- ls -la /proc/2432/status
ls: /proc/2432/status: No such file or directoryThis seems really strange. The init process (1) has an open FD pointing to a proc entry (status) of a process outside of the PID namespace.
adrianreber
commented
4 years ago So @brauner (thanks a lot!) found a possible bug in LXC and opened https://github.com/lxc/lxc/pull/3360 to fix this.
diabolusss
commented
4 years ago Cool! Thank you @brauner and @adrianreber for assistance. Hope it's the actual cause. ps. I suppose i should close this issue only if lxc fix will help.
adrianreber
commented
4 years ago You can close it and re-open it later or keep it open. Whatever works for better for you.
diabolusss
commented
4 years ago Oddly enough, but i have the same error after update to the latest snap and lxd (which i suppose have related bugfix) with my old containers. With freshly created container on any storage backend i don't have mentioned before errors, but have another one and more catastrophic (kind of).
$ snap --version snap 2.44.3-1 snapd 2.44.3-1 series 16 arch - kernel 5.2.11-arch1-1-ARCH
$ snap list Name Version Rev Tracking Publisher Notes core18 20200311 1705 latest/stable canonical✓ base lxd 4.0.1 14804 latest/stable canonical✓ - snapd 2.44.3 7264 latest/stable canonical✓ snapd
$ lxc profile show default config: {} description: Default LXD profile devices: eth0: name: eth0 nictype: bridged parent: virbr01 type: nic root: path: / pool: stablepool50G type: disk name: default used_by: ...
Test with imported old container.
$ lxc snapshot openvpn --stateful Error: snapshot dump failed (00.000043) Warn (criu/log.c:203): The early log isn't empty (00.101371) Error (criu/namespaces.c:415): Can't dump nested uts namespace for 2018 (00.101373) Error (criu/namespaces.c:670): Can't make utsns id (00.102700) Error (criu/util.c:610): exited, status=1 (00.105048) Error (criu/util.c:610): exited, status=1 (00.105238) Error (criu/cr-dump.c:1743): Dumping FAILED.
$ lxc stop openvpn --stateful Error: snapshot dump failed (00.000054) Warn (criu/log.c:203): The early log isn't empty (00.102457) Error (criu/namespaces.c:415): Can't dump nested uts namespace for 2018 (00.102464) Error (criu/namespaces.c:670): Can't make utsns id (00.105009) Error (criu/util.c:610): exited, status=1 (00.106376) Error (criu/util.c:610): exited, status=1 (00.106554) Error (criu/cr-dump.c:1743): Dumping FAILED. Try
lxc info --show-log openvpnfor more info$ lxc info --show-log openvpn Name: openvpn Location: none Remote: unix:// Architecture: x86_64 Created: 2020/04/23 18:20 UTC Status: Running Type: container Profiles: default Pid: 1950 Ips: tap0: inet 10.20.30.254 tap0: inet6 fe80::dc0e:a7ff:feac:32b2 eth0: inet 192.168.100.163 veth154df357 eth0: inet6 fe80::216:3eff:fe70:f99e veth154df357 lo: inet 127.0.0.1 lo: inet6 ::1 Resources: Processes: 12 CPU usage: CPU usage (in seconds): 3 Memory usage: Memory (current): 67.19MB Network usage: eth0: Bytes received: 868.41kB Bytes sent: 269.13kB Packets received: 9902 Packets sent: 3460 lo: Bytes received: 0B Bytes sent: 0B Packets received: 0 Packets sent: 0 tap0: Bytes received: 54.87kB Bytes sent: 15.94kB Packets received: 694 Packets sent: 82
Log:
lxc openvpn 20200423182100.926 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.openvpn" lxc openvpn 20200423182100.927 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.openvpn" lxc openvpn 20200423182100.928 ERROR utils - utils.c:lxc_can_use_pidfd:1855 - Invalid argument - Kernel does not support waiting on processes through pidfds lxc openvpn 20200423182100.932 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1455 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW ) lxc 20200423182330.170 ERROR criu - criu.c:do_dump:1304 - dump failed with 1 lxc 20200423182330.170 ERROR criu - criu.c:do_dump:1318 - criu output: lxc 20200423182852.283 ERROR criu - criu.c:do_dump:1304 - dump failed with 1 lxc 20200423182852.283 ERROR criu - criu.c:do_dump:1318 - criu output: lxc 20200423184026.606 ERROR criu - criu.c:do_dump:1304 - dump failed with 1 lxc 20200423184026.606 ERROR criu - criu.c:do_dump:1318 - criu output: lxc 20200423194010.458 ERROR criu - criu.c:do_dump:1304 - dump failed with 1 lxc 20200423194010.458 ERROR criu - criu.c:do_dump:1318 - criu output: lxc 20200423195623.878 ERROR criu - criu.c:do_dump:1304 - dump failed with 1 lxc 20200423195623.878 ERROR criu - criu.c:do_dump:1318 - criu output: lxc 20200423195636.408 ERROR criu - criu.c:do_dump:1304 - dump failed with 1 lxc 20200423195636.408 ERROR criu - criu.c:do_dump:1318 - criu output:
$ lxc exec openvpn -- ls -la /proc/1/fd total 0 dr-x------ 2 root root 0 Apr 23 20:00 . dr-xr-xr-x 9 root root 0 Apr 23 18:21 .. lrwx------ 1 root root 64 Apr 23 20:00 0 -> /dev/null lrwx------ 1 root root 64 Apr 23 20:00 1 -> /dev/null lr-x------ 1 root root 64 Apr 23 20:00 10 -> /proc/1/mountinfo lr-x------ 1 root root 64 Apr 23 20:00 11 -> anon_inode:inotify lr-x------ 1 root root 64 Apr 23 20:00 13 -> anon_inode:inotify lrwx------ 1 root root 64 Apr 23 20:00 14 -> 'socket:[205278]' lrwx------ 1 root root 64 Apr 23 20:00 15 -> 'socket:[205280]' lrwx------ 1 root root 64 Apr 23 20:00 16 -> 'socket:[205282]' lrwx------ 1 root root 64 Apr 23 20:00 17 -> 'socket:[205283]' lrwx------ 1 root root 64 Apr 23 20:00 18 -> 'socket:[205284]' lrwx------ 1 root root 64 Apr 23 20:00 19 -> 'socket:[205289]' lrwx------ 1 root root 64 Apr 23 20:00 2 -> /dev/null lr-x------ 1 root root 64 Apr 23 20:00 24 -> anon_inode:inotify lr-x------ 1 root root 64 Apr 23 20:00 25 -> anon_inode:inotify lrwx------ 1 root root 64 Apr 23 20:00 26 -> 'anon_inode:[timerfd]' lrwx------ 1 root root 64 Apr 23 20:00 27 -> /run/dmeventd-server lrwx------ 1 root root 64 Apr 23 20:00 28 -> /run/dmeventd-client lrwx------ 1 root root 64 Apr 23 20:00 29 -> 'socket:[205296]' lrwx------ 1 root root 64 Apr 23 20:00 3 -> 'anon_inode:[timerfd]' lrwx------ 1 root root 64 Apr 23 20:00 30 -> /run/initctl lrwx------ 1 root root 64 Apr 23 20:00 31 -> 'socket:[205304]' lrwx------ 1 root root 64 Apr 23 20:00 32 -> 'socket:[205309]' lrwx------ 1 root root 64 Apr 23 20:00 33 -> 'socket:[205312]' lrwx------ 1 root root 64 Apr 23 20:00 34 -> 'socket:[205315]' lrwx------ 1 root root 64 Apr 23 20:00 35 -> 'socket:[205316]' lrwx------ 1 root root 64 Apr 23 20:00 36 -> 'socket:[205320]' lrwx------ 1 root root 64 Apr 23 20:00 37 -> 'socket:[205321]' lrwx------ 1 root root 64 Apr 23 20:00 38 -> 'socket:[220250]' lrwx------ 1 root root 64 Apr 23 20:00 39 -> 'socket:[183985]' lrwx------ 1 root root 64 Apr 23 20:00 4 -> 'anon_inode:[eventpoll]' lrwx------ 1 root root 64 Apr 23 20:00 40 -> 'socket:[210316]' lrwx------ 1 root root 64 Apr 23 20:00 41 -> 'socket:[219222]' lrwx------ 1 root root 64 Apr 23 20:00 42 -> 'socket:[184010]' lrwx------ 1 root root 64 Apr 23 20:00 43 -> 'socket:[184018]' lrwx------ 1 root root 64 Apr 23 20:00 44 -> 'socket:[184020]' lrwx------ 1 root root 64 Apr 23 20:00 45 -> 'socket:[184021]' lrwx------ 1 root root 64 Apr 23 20:00 46 -> 'socket:[211323]' lrwx------ 1 root root 64 Apr 23 20:00 47 -> 'socket:[211325]' lrwx------ 1 root root 64 Apr 23 20:00 48 -> 'socket:[211327]' lrwx------ 1 root root 64 Apr 23 20:00 49 -> 'socket:[221293]' lrwx------ 1 root root 64 Apr 23 20:00 5 -> 'anon_inode:[signalfd]' lrwx------ 1 root root 64 Apr 23 20:00 50 -> 'socket:[205536]' lr-x------ 1 root root 64 Apr 23 20:00 6 -> anon_inode:inotify lr-x------ 1 root root 64 Apr 23 20:00 7 -> /sys/fs/cgroup/unified lrwx------ 1 root root 64 Apr 23 20:00 8 -> 'anon_inode:[timerfd]' lrwx------ 1 root root 64 Apr 23 20:00 9 -> 'anon_inode:[eventpoll]'
$ sudo tail /var/snap/lxd/common/lxd/logs/openvpn/snapshot_dump_2020-04-23T22:56:36+03:00.log -n 30 (00.102359) Will take mnt namespace in the image (00.102364) Add mnt ns 12 pid 1950 (00.102373) Will take user namespace in the image (00.102377) Add user ns 13 pid 1950 (00.102387) Will take cgroup namespace in the image (00.102392) Add cgroup ns 14 pid 1950 (00.102457) Error (criu/namespaces.c:415): Can't dump nested uts namespace for 2018 (00.102464) Error (criu/namespaces.c:670): Can't make utsns id (00.102473) Unlock network (00.102478) Running network-unlock scripts iptables-restore: invalid option -- 'w' iptables-restore: line 3 failed (00.105009) Error (criu/util.c:610): exited, status=1 ip6tables-restore: invalid option -- 'w' ip6tables-restore: line 3 failed (00.106376) Error (criu/util.c:610): exited, status=1 (00.106407) Unfreezing tasks into 1 (00.106415) Unseizing 1950 into 1 (00.106428) Unseizing 2013 into 1 (00.106439) Unseizing 2018 into 1 (00.106448) Unseizing 2026 into 1 (00.106456) Unseizing 2029 into 1 (00.106464) Unseizing 2030 into 1 (00.106472) Unseizing 2031 into 1 (00.106480) Unseizing 2043 into 1 (00.106489) Unseizing 2045 into 1 (00.106498) Unseizing 2054 into 1 (00.106507) Unseizing 2251 into 1 (00.106516) Unseizing 2263 into 1 (00.106554) Error (criu/cr-dump.c:1743): Dumping FAILED.
And here are results with brand new containers. I've checked with btrfs, lvm and zfs backends - same result.
$ lxc storage create testpoollvm4G lvm size=4GB $ lxc storage create testpoolzfs2G zfs size=2GB $ lxc storage create testpoolbtrfs2G btrfs size=2GB
$ lxc launch images:alpine/edge alpineedgebtrfs --storage=testpoolbtrfs2G $ lxc launch images:alpine/edge alpineedgezfs --storage=testpoolzfs2G $ lxc launch images:alpine/edge alpineedgelvm --storage=testpoollvm4G
I have created counter script and pushed it into each container for easier state check. It's endless loop which echoes numbers into file, so i can read it.
!/bin/sh
i=0; while true; do echo $i>count; i=$(expr $i + 1); sleep 1; done
Run counter
$ lxc exec alpineedgebtrfs -- /bin/sh -ic 'nohup ./count.sh &'
Check current count
$ lxc exec alpineedgebtrfs -- /bin/sh -c 'cat count'
$ lxc stop alpineedgezfs --stateful <- stopped just fine $ lxc start alpineedgezfs
Error: Migrate: Failed to run: /snap/lxd/current/bin/lxd forkmigrate alpineedgezfs /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/alpineedgezfs/lxc.conf /var/snap/lxd/common/lxd/containers/alpineedgezfs/state true: Try
lxc info --show-log alpineedgezfsfor more info$ lxc info --show-log alpineedgezfs Name: alpineedgezfs Location: none Remote: unix:// Architecture: x86_64 Created: 2020/04/23 18:54 UTC Status: Stopped Type: container Profiles: default
Log: lxc alpineedgezfs 20200423185818.806 ERROR criu - criu.c:criu_ok:872 - Found un-dumpable network: phys (eth0)
At the same time i can create snapshots without problems and restore them. If i restore while container is running there is no errors, but counter don't count any more, i.e. process died. If i restore after stopping it, then i receive this same error and i have no internet connection. Lan and it's services like smb, upnp are accessible though.
After some investigation i've found out that it adds default paths to host route table: Before starting stateful container:
$ ip route default via 192.168.1.254 dev wlp4s0 proto dhcp src 192.168.1.152 metric 303 192.168.1.0/24 dev wlp4s0 proto dhcp scope link src 192.168.1.152 metric 303 192.168.100.0/24 dev virbr01 proto kernel scope link src 192.168.100.165
After (i called it multiple times):
$ ip route default via 192.168.100.165 dev vethac23679b proto dhcp src 192.168.100.153 metric 210 default via 192.168.100.165 dev veth5ca5d774 proto dhcp src 192.168.100.153 metric 212 default via 192.168.100.165 dev veth21d0dbf5 proto dhcp src 192.168.100.153 metric 214 default via 192.168.100.165 dev veth937b77d8 proto dhcp src 192.168.100.153 metric 216 default via 192.168.1.254 dev wlp4s0 proto dhcp src 192.168.1.152 metric 303 192.168.1.0/24 dev wlp4s0 proto dhcp scope link src 192.168.1.152 metric 303 192.168.100.0/24 dev virbr01 proto kernel scope link src 192.168.100.165 192.168.100.0/24 dev vethac23679b proto dhcp scope link src 192.168.100.153 metric 210 192.168.100.0/24 dev veth5ca5d774 proto dhcp scope link src 192.168.100.153 metric 212 192.168.100.0/24 dev veth21d0dbf5 proto dhcp scope link src 192.168.100.153 metric 214 192.168.100.0/24 dev veth937b77d8 proto dhcp scope link src 192.168.100.153 metric 216
And all interfaces don't get destroyed after error:
$ ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000 link/ether b4:2e:99:1d:d9:a4 brd ff:ff:ff:ff:ff:ff 3: wlp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000 link/ether 38:00:25:aa:aa:5f brd ff:ff:ff:ff:ff:ff 4: virbr01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether ee:40:3e:56:90:5c brd ff:ff:ff:ff:ff:ff 5: virbr01-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr01 state DOWN mode DEFAULT group default qlen 1000 link/ether 52:54:00:ec:e4:38 brd ff:ff:ff:ff:ff:ff 7: veth8313aa62@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr01 state UP mode DEFAULT group default qlen 1000 link/ether 9e:17:e0:67:38:68 brd ff:ff:ff:ff:ff:ff link-netnsid 0 9: veth2e7cfc96@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr01 state UP mode DEFAULT group default qlen 1000 link/ether c6:fc:d0:2e:45:77 brd ff:ff:ff:ff:ff:ff link-netnsid 1 10: vethac23679b@vetha4ba0b30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:d0:55:41 brd ff:ff:ff:ff:ff:ff 11: vetha4ba0b30@vethac23679b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr01 state UP mode DEFAULT group default qlen 1000 link/ether e6:a3:8b:8d:f3:af brd ff:ff:ff:ff:ff:ff 12: veth5ca5d774@veth70464ae1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:d0:55:41 brd ff:ff:ff:ff:ff:ff 13: veth70464ae1@veth5ca5d774: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr01 state UP mode DEFAULT group default qlen 1000 link/ether a2:9f:81:01:67:22 brd ff:ff:ff:ff:ff:ff 14: veth21d0dbf5@vethb5679864: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:d0:55:41 brd ff:ff:ff:ff:ff:ff 15: vethb5679864@veth21d0dbf5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr01 state UP mode DEFAULT group default qlen 1000 link/ether ca:2b:a3:a3:5d:36 brd ff:ff:ff:ff:ff:ff 16: veth937b77d8@vethedca3338: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:d0:55:41 brd ff:ff:ff:ff:ff:ff 17: vethedca3338@veth937b77d8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr01 state UP mode DEFAULT group default qlen 1000 link/ether 0e:e3:c9:f3:ac:21 brd ff:ff:ff:ff:ff:ff
adrianreber
commented
4 years ago @diabolusss You are providing too much information. It is hard to find the actual error :wink:
But I think this is one of the many problems you are seeing:
lxc alpineedgezfs 20200423185818.806 ERROR criu - criu.c:criu_ok:872 - Found un-dumpable network: phys (eth0)
I am confused. Why is there a physical device in your container. Did you put it there. For this to work LXC needs to fundamentally change its CRIU - network namespace handling.
diabolusss
commented
4 years ago Sorry, I just don't know what information could be important, so i've inserted everything that was asked before and seems to be important from my point of view. For long texts is it acceptable to put pastebin links?
I am confused. Why is there a physical device in your container. Did you put it there. For this to work LXC needs to fundamentally change its CRIU - network namespace handling. I have added bridge that was created by libvirtd.
$ lxc profile show default config: {} description: Default LXD profile devices: eth0: name: eth0 nictype: bridged parent: virbr01 type: nic root: path: / pool: stablepool50G type: disk name: default used_by: ...
$ brctl show bridge name bridge id STP enabled interfaces virbr01 8000.ee403e56905c yes veth265eabbf vethb208c158 vethb3d8ba02 vethda9f3141 virbr01-nic
<network>
<name>default-ethernet</name>
<uuid>08a305fa-e19d-4562-8cb9-78f6196e7f18</uuid>
<forward dev="wlp4s0" mode="nat">
<nat>
<port start="1024" end="65535"/>
</nat>
<interface dev="wlp4s0"/>
</forward>
<bridge name="virbr01" stp="on" delay="0"/>
<mac address="52:54:00:ec:e4:38"/>
<domain name="default-ethernet"/>
<ip address="192.168.100.165" netmask="255.255.255.0">
<dhcp>
<range start="192.168.100.128" end="192.168.100.254"/>
<host mac="52:54:00:3a:6a:17" name="archlinux" ip="192.168.100.192"/>
</dhcp>
</ip>
</network>If you start a container which network interfaces are active in the container? Can you show that?
diabolusss
commented
4 years ago @adrianreber, here you are: That is for old container with failed criu
$ lxc exec openvpn -- bash -ilc "ip link" 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:70:f9:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0
This is for new container before stateful action, because after that it don't start up
$ lxc exec alpineedgezfs -- sh -ilc "ip link" 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 00:16:3e:99:be:b1 brd ff:ff:ff:ff:ff:ff
Snorch
commented
4 years ago Also you have an error which was not discussed here yet which is also a critical one:
Error (criu/namespaces.c:415): Can't dump nested uts namespace for 1958
Error (criu/namespaces.c:415): Can't dump nested uts namespace for 22578
Error (criu/namespaces.c:415): Can't dump nested uts namespace for 2018Criu does not support nested uts namespaces if your application processes have more than one utsns you won't be able to dump it (workaround: you need to kill all tasks from other uts namspaces before dump).
-- offtop: actually supporting nested uts namespaces should be not so hard to do, you just need to: 1) call prepare_utsns for each utsns not only for root_one 2) do CLONE_NEWUTS only for the first task from utsns, others should inherit or setns (or maybe even prepare all uts namespaces before tasks creation).
diabolusss
commented
4 years ago @Snorch Thank you for your suggestion, but i wouldn't try it before second error is fixed. Simply because if stateful snapshot will be made successfully, as for second error in my case, then i don't know how to recover or drop it without reverting to a staless state saved before.
offtop: i imagined criu is working like vm snapshots out-of-the-box, but now it seems to me that it's not so easy to achieve.
adrianreber
commented
4 years ago Similar error discussed here. In one of the errors here LXC does not even start CRIU https://github.com/lxc/lxc/blob/master/src/lxc/criu.c#L872
diabolusss
commented
4 years ago @adrianreber thank you.
Recently I've noticed another bug related to starting container with un-dumpable interfaces. After i've tried to start container (that was statefully stopped) and received previously mentioned error smth is constantly adding back this broken routes after i delete them. (if i don't reboot after that). snap restart lxd doesnt help. p.s. manually putting their state to DOWN (or deleting) solves this problem.
sudo ip link set dev vethXXX down
github-actions[bot]
commented
3 years ago A friendly reminder that this issue had no activity for 30 days.
luke-jr
commented
3 years ago Any solution? Seems Proxmox VE fails to suspend containers out-of-the-box due to this issue :/
(Both ubuntu and gentoo/no-systemd)
adrianreber
commented
3 years ago @luke-jr If you have any specific error it would probably make more sense to open a new ticket. This ticket discusses multiple things so it is hard to know what you exactly mean.
Hi. I have installed criu to make stateful operations with lxd containers, but it always fails.
System info
Error on --stateful actions
I've tried to do the same with fresh new container and received similar error:
I'm new into lxd and criu, so i hope this problem is just lack of knowledge. Your help will be appreciated. As a last resort i'll try the same on a fresh new host system as this option is crucial for me...