search

checkpoint-restore / criu

Checkpoint/Restore tool
criu.org
Other
2.97k stars 596 forks source link

checkpoint broken for Podman or Docker #2011

Closed josevnz closed 1 year ago

josevnz commented 1 year ago

Description

Checkpoint feature is broken with the following error:

Steps to reproduce the issue:

  1. Run a container as root (required for PODMAN/ Docker)
podman run \
    --detach \
    --name webtop-test \
    --env PUID=1000 \
    --env PGID=1000 \
    --tz America/New_York \
    --env TITLE='Webtop test' \
    --publish 3000:3000 \
    --volume /home/josevnz/webtop/config:/config \
    lscr.io/linuxserver/webtop:ubuntu-kde

  1. Confirm is running (podman top webtop-test, podman logs --follow webtop-test

  2. podman container checkpoint webtop-test

  3. podman failes to freeze the container

    
2022-12-05T08:46:43.000805148Z: CRIU checkpointing failed -52
Please check CRIU logfile /var/lib/containers/storage/overlay-containers/64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc/userdata/dump.log

Error: /usr/bin/crun checkpoint --image-path /var/lib/containers/storage/overlay-containers/64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc/userdata/checkpoint --work-path /var/lib/containers/storage/overlay-containers/64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc/userdata 64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc failed: exit status 1


6. Docker steps render the similar failure

docker run --detach --name webtop-test --env PUID=1000 --env PGID=1000 --env TZ=America/New_York --env TITLE='Webtop test' --publish 3000:3000 --volume /home/josevnz/webtop/config2:/config lscr.io/linuxserver/webtop:ubuntu-kde

docker ps webtop-test

[root@dmaf5 ]# docker checkpoint create webtop-test checkpoint Error response from daemon: Cannot checkpoint container webtop-test: runc did not terminate successfully: criu failed: type NOTIFY errno 0 path= /run/containerd/io.containerd.runtime.v2.task/moby/bdf7c5243ada5a5d37774162fc1d86f4c72ae6b006d0df948a48576b333f012a/criu-dump.log: unknown

7. 

**Describe the results you received:**

PODMAN/ Docker crashes

**Describe the results you expected:**

container is frozen.

**Additional information you deem important (e.g. issue happens only occasionally):**

This feature doesn't work for any non trivial components. Maybe it should be removed completely from CRIU.

**CRIU logs and information:**

Will attach full log as file, below just the snippet with the issue

<details><summary>CRIU full dump/restore logs:</summary>
<p>

PODMAN

2022-12-05T08:46:43.000805148Z: CRIU checkpointing failed -52 Please check CRIU logfile /var/lib/containers/storage/overlay-containers/64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc/userdata/dump.log

Error: /usr/bin/crun checkpoint --image-path /var/lib/containers/storage/overlay-containers/64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc/userdata/checkpoint --work-path /var/lib/containers/storage/overlay-containers/64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc/userdata 64cd2721f621484346465c0fcf4d24a240179e5468bd901d161b010df3b3e2bc failed: exit status 1

(41.164806) mnt: Inspecting sharing on 1605 shared_id 0 master_id 0 (@./) (41.164819) Error (criu/mount.c:662): mnt: 1617:./usr/share/zoneinfo/Etc/UTC doesn't have a proper root mount (41.164856) Unlock network (41.164863) Running network-unlock scripts (41.172373) Unfreezing tasks into 1 (41.172395) Unseizing 5290 into 1 (41.172414) Unseizing 5300 into 1 (41.172428) Unseizing 5303 into 1 (41.172443) Unseizing 5321 into 1 (41.172456) Unseizing 5335 into 1 (41.172479) Unseizing 5322 into 1 (41.172510) Unseizing 5323 into 1 (41.172523) Unseizing 5524 into 1 (41.172537) Unseizing 5537 into 1 (41.172548) Unseizing 5324 into 1 (41.172558) Unseizing 5490 into 1 (41.172567) Unseizing 5325 into 1 (41.172578) Unseizing 5503 into 1 (41.172617) Unseizing 5326 into 1 (41.172633) Unseizing 5482 into 1 (41.172645) Unseizing 5327 into 1 (41.172655) Unseizing 5475 into 1 (41.172747) Error (criu/cr-dump.c:1768): Dumping FAILED.

</p>

<p>
DOCKER

pie: 1: Daemon waits for command (00.013536) Fetched ack: 64 64 0 (00.013551) Error (criu/lsm.c:90): Invalid selinux context kernel (00.013560) ---------------------------------------- (00.013565) Error (criu/cr-dump.c:1390): Dump core (pid: 7013) failed with -1 (00.013581) Waiting for 7013 to trap

</p>

</details>

<details><summary>Output of `criu --version`:</summary>
<p>

[root@dmaf5 ]# criu --version Version: 3.15


</p>
</details>

<details><summary>Output of `criu check --all`:</summary>
<p>

[root@dmaf5 ]# criu check --all Looks good.


</p>
</details>

**Additional environment details:**

[root@dmaf5 ]# rpm -qi podman Name : podman Epoch : 3 Version : 3.3.1 Release : 1.fc33 Architecture: x86_64 Install Date: Sat 11 Sep 2021 08:32:10 AM EDT Group : Unspecified Size : 51477155 License : ASL 2.0 Signature : RSA/SHA256, Mon 30 Aug 2021 05:41:54 PM EDT, Key ID 49fd77499570ff31 Source RPM : podman-3.3.1-1.fc33.src.rpm Build Date : Mon 30 Aug 2021 04:46:43 PM EDT Build Host : buildvm-x86-29.iad2.fedoraproject.org Packager : Fedora Project Vendor : Fedora Project URL : https://podman.io/ Bug URL : https://bugz.fedoraproject.org/podman Summary : Manage Pods, Containers and Container Images

[root@dmaf5 ]# rpm -qi docker-ce Name : docker-ce Epoch : 3 Version : 20.10.12 Release : 3.fc33 Architecture: x86_64 Install Date: Tue 14 Dec 2021 07:44:42 PM EST Group : Tools/Docker Size : 100157208 License : ASL 2.0 Signature : RSA/SHA512, Mon 13 Dec 2021 09:22:03 AM EST, Key ID c52feb6b621e9f35 Source RPM : docker-ce-20.10.12-3.fc33.src.rpm Build Date : Mon 13 Dec 2021 06:43:52 AM EST Build Host : 3bf4c068d283 Packager : Docker support@docker.com Vendor : Docker URL : https://www.docker.com Summary : The open-source application container engine Description : Docker is a product for you to build, ship and run any application as a lightweight container.

Docker containers are both hardware-agnostic and platform-agnostic. This means they can run anywhere, from your laptop to the largest cloud compute instance and everything in between - and they don't require you to use a particular language, framework or packaging system. That makes them great building blocks for deploying and scaling web apps, databases, and backend services without depending on a particular stack or provider.

josevnz commented 1 year ago

Docker + CRIU logs

(00.000000) Unable to get $HOME directory, local configuration file will not be used.(00.000096) Version: 3.15 (gitid 0)
(00.000122) Running on dmaf5 Linux 5.14.18-100.fc33.x86_64 #1 SMP Fri Nov 12 17:38:44 UTC 2021 x86_64
(00.000130) Would overwrite RPC settings with values from /etc/criu/runc.conf
(00.000168) Loaded kdat cache from /run/criu/criu.kdat
(00.000243) ========================================
(00.000252) Dumping processes (pid: 8132)
(00.000259) ========================================
(00.000272) rlimit: RLIMIT_NOFILE unlimited for self
(00.000287) Running pre-dump scripts
(00.000294)     RPC
(00.000431) irmap: Searching irmap cache in work dir
(00.000464) No irmap-cache image
(00.000472) irmap: Searching irmap cache in parent
(00.000484) irmap: No irmap cache
(00.000494) cpu: x86_family 23 x86_vendor_id AuthenticAMD x86_model_id AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
(00.000502) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 832
(00.000512) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
(00.000521) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
(00.000529) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.000854) cg-prop: Parsing controller "cpu"
(00.000869) cg-prop:    Strategy "replace"
(00.000878) cg-prop:    Property "cpu.shares"
(00.000885) cg-prop:    Property "cpu.cfs_period_us"
(00.000892) cg-prop:    Property "cpu.cfs_quota_us"
(00.000900) cg-prop:    Property "cpu.rt_period_us"
(00.000906) cg-prop:    Property "cpu.rt_runtime_us"
(00.000913) cg-prop: Parsing controller "memory"
(00.000920) cg-prop:    Strategy "replace"
(00.000928) cg-prop:    Property "memory.limit_in_bytes"
(00.000935) cg-prop:    Property "memory.memsw.limit_in_bytes"
(00.000942) cg-prop:    Property "memory.swappiness"
(00.000949) cg-prop:    Property "memory.soft_limit_in_bytes"
(00.000955) cg-prop:    Property "memory.move_charge_at_immigrate"
(00.000962) cg-prop:    Property "memory.oom_control"
(00.000969) cg-prop:    Property "memory.use_hierarchy"
(00.000976) cg-prop:    Property "memory.kmem.limit_in_bytes"
(00.000982) cg-prop:    Property "memory.kmem.tcp.limit_in_bytes"
(00.000989) cg-prop: Parsing controller "cpuset"
(00.000996) cg-prop:    Strategy "replace"
(00.001003) cg-prop:    Property "cpuset.cpus"
(00.001010) cg-prop:    Property "cpuset.mems"
(00.001017) cg-prop:    Property "cpuset.memory_migrate"
(00.001024) cg-prop:    Property "cpuset.cpu_exclusive"
(00.001031) cg-prop:    Property "cpuset.mem_exclusive"
(00.001038) cg-prop:    Property "cpuset.mem_hardwall"
(00.001045) cg-prop:    Property "cpuset.memory_spread_page"
(00.001051) cg-prop:    Property "cpuset.memory_spread_slab"
(00.001058) cg-prop:    Property "cpuset.sched_load_balance"
(00.001070) cg-prop:    Property "cpuset.sched_relax_domain_level"
(00.001077) cg-prop: Parsing controller "blkio"
(00.001084) cg-prop:    Strategy "replace"
(00.001091) cg-prop:    Property "blkio.weight"
(00.001098) cg-prop: Parsing controller "freezer"
(00.001104) cg-prop:    Strategy "replace"
(00.001112) cg-prop: Parsing controller "perf_event"
(00.001119) cg-prop:    Strategy "replace"
(00.001126) cg-prop: Parsing controller "net_cls"
(00.001133) cg-prop:    Strategy "replace"
(00.001140) cg-prop:    Property "net_cls.classid"
(00.001147) cg-prop: Parsing controller "net_prio"
(00.001153) cg-prop:    Strategy "replace"
(00.001161) cg-prop:    Property "net_prio.ifpriomap"
(00.001167) cg-prop: Parsing controller "pids"
(00.001174) cg-prop:    Strategy "replace"
(00.001181) cg-prop:    Property "pids.max"
(00.001188) cg-prop: Parsing controller "devices"
(00.001195) cg-prop:    Strategy "replace"
(00.001202) cg-prop:    Property "devices.list"
(00.001250) Preparing image inventory (version 1)
(00.001296) Add pid ns 1 pid 8420
(00.001314) Add net ns 2 pid 8420
(00.001330) Add ipc ns 3 pid 8420
(00.001346) Add uts ns 4 pid 8420
(00.001362) Add time ns 5 pid 8420
(00.001385) Add mnt ns 6 pid 8420
(00.001401) Add user ns 7 pid 8420
(00.001416) Add cgroup ns 8 pid 8420
(00.001424) cg: Dumping cgroups for 8420
(00.001474) cg:  `- New css ID 1
(00.001485) cg:     `- [] -> [/system.slice/containerd.service] [0]
(00.001494) cg: Set 1 is criu one
(00.001542) Detected cgroup V2 freezer
(00.001550) freezing processes: 100000 attempts with 100 ms steps
(00.001575) cgroup.freeze=1
(00.001667) SEIZE 8132: success
(00.001687) SEIZE 8171: success
(00.001701) SEIZE 8174: success
(00.001713) SEIZE 8192: success
(00.001729) SEIZE 8193: success
(00.001748) SEIZE 8194: success
(00.001807) SEIZE 8195: success
(00.001824) SEIZE 8196: success
(00.001839) SEIZE 8197: success
(00.001862) SEIZE 8198: success
(00.001896) SEIZE 8206: success
(00.001909) SEIZE 8327: success
(00.001920) SEIZE 8334: success
(00.001935) SEIZE 8342: success
(00.001949) SEIZE 8352: success
(00.001961) SEIZE 8357: success
(00.001972) SEIZE 8358: success
(00.001986) SEIZE 8359: success
(00.002002) SEIZE 8360: success
(00.002038) SEIZE 8361: success
(00.002053) SEIZE 8363: success
(00.002066) SEIZE 8369: success
(00.002079) SEIZE 8381: success
(00.002555) Seized task 8132, state 1
(00.002572) seccomp: Collected tid_real 8132 mode 0x2
(00.002605) Collected (0 attempts, 0 in_progress)
(00.002658) Seized task 8171, state 0
(00.002742) seccomp: Collected tid_real 8171 mode 0x2
(00.002789) Collected (0 attempts, 0 in_progress)
(00.002826) Seized task 8174, state 0
(00.002906) seccomp: Collected tid_real 8174 mode 0x2
(00.002933) Collected (0 attempts, 0 in_progress)
(00.002960) Collected (0 attempts, 0 in_progress)
(00.002973) Collected 8174 in 1 state
(00.003011) Collected (-1 attempts, 0 in_progress)
(00.003027) Collected 8171 in 1 state
(00.003035) Seized task 8192, state 0
(00.003114) seccomp: Collected tid_real 8192 mode 0x2
(00.003140) Collected (0 attempts, 0 in_progress)
(00.003166) Seized task 8206, state 0
(00.003233) seccomp: Collected tid_real 8206 mode 0x2
(00.003255) Collected (0 attempts, 0 in_progress)
(00.003278) Collected (0 attempts, 0 in_progress)
(00.003293) Collected 8206 in 1 state
(00.003329) Collected (-1 attempts, 0 in_progress)
(00.003339) Collected 8192 in 1 state
(00.003346) Seized task 8193, state 0
(00.003418) seccomp: Collected tid_real 8193 mode 0x2
(00.003441) Collected (0 attempts, 0 in_progress)
(00.003466) Collected (0 attempts, 0 in_progress)
(00.003475) Collected 8193 in 1 state
(00.003482) Seized task 8194, state 0
(00.003542) seccomp: Collected tid_real 8194 mode 0x2
(00.003562) Collected (0 attempts, 0 in_progress)
(00.003586) Seized task 8369, state 0
(00.003646) seccomp: Collected tid_real 8369 mode 0x2
(00.003670) Collected (0 attempts, 0 in_progress)
(00.003694) Seized task 8381, state 0
(00.003750) seccomp: Collected tid_real 8381 mode 0x2
(00.003785) Collected (0 attempts, 0 in_progress)
(00.003807) Collected (0 attempts, 0 in_progress)
(00.003816) Collected 8381 in 1 state
(00.003841) Collected (-1 attempts, 0 in_progress)
(00.003854) Collected 8369 in 1 state
(00.003878) Collected (-1 attempts, 0 in_progress)
(00.003889) Collected 8194 in 1 state
(00.003896) Seized task 8195, state 0
(00.003952) seccomp: Collected tid_real 8195 mode 0x2
(00.003974) Collected (0 attempts, 0 in_progress)
(00.003997) Seized task 8342, state 0
(00.004056) seccomp: Collected tid_real 8342 mode 0x2
(00.004076) Collected (0 attempts, 0 in_progress)
(00.004097) Collected (0 attempts, 0 in_progress)
(00.004106) Collected 8342 in 1 state
(00.004130) Collected (-1 attempts, 0 in_progress)
(00.004143) Collected 8195 in 1 state
(00.004150) Seized task 8196, state 0
(00.004210) seccomp: Collected tid_real 8196 mode 0x2
(00.004235) Collected (0 attempts, 0 in_progress)
(00.004258) Seized task 8352, state 0
(00.004320) seccomp: Collected tid_real 8352 mode 0x2
(00.004357)     Seizing 8352's 8357 thread
(00.004415) seccomp: Collected tid_real 8357 mode 0x2
(00.004423)     Seizing 8352's 8358 thread
(00.004480) seccomp: Collected tid_real 8358 mode 0x2
(00.004488)     Seizing 8352's 8359 thread
(00.004543) seccomp: Collected tid_real 8359 mode 0x2
(00.004551)     Seizing 8352's 8360 thread
(00.004608) seccomp: Collected tid_real 8360 mode 0x2
(00.004615)     Seizing 8352's 8361 thread
(00.004688) seccomp: Collected tid_real 8361 mode 0x2
(00.004696)     Seizing 8352's 8363 thread
(00.004744) seccomp: Collected tid_real 8363 mode 0x2
(00.004773) Collected (-1 attempts, 0 in_progress)
(00.004819) Collected (0 attempts, 0 in_progress)
(00.004832) Collected 8352 in 1 state
(00.004847) Collected (-1 attempts, 0 in_progress)
(00.004854) Collected 8196 in 1 state
(00.004858) Seized task 8197, state 0
(00.004895) seccomp: Collected tid_real 8197 mode 0x2
(00.004908) Collected (0 attempts, 0 in_progress)
(00.004921) Seized task 8334, state 0
(00.004958) seccomp: Collected tid_real 8334 mode 0x2
(00.004970) Collected (0 attempts, 0 in_progress)
(00.004982) Collected (0 attempts, 0 in_progress)
(00.004989) Collected 8334 in 1 state
(00.005003) Collected (-1 attempts, 0 in_progress)
(00.005008) Collected 8197 in 1 state
(00.005013) Seized task 8198, state 0
(00.005049) seccomp: Collected tid_real 8198 mode 0x2
(00.005061) Collected (0 attempts, 0 in_progress)
(00.005074) Seized task 8327, state 0
(00.005111) seccomp: Collected tid_real 8327 mode 0x2
(00.005123) Collected (0 attempts, 0 in_progress)
(00.005134) Collected (0 attempts, 0 in_progress)
(00.005139) Collected 8327 in 1 state
(00.005153) Collected (-1 attempts, 0 in_progress)
(00.005158) Collected 8198 in 1 state
(00.005174) Collected (-1 attempts, 0 in_progress)
(00.005181) Collected 8132 in 1 state
(00.005204) Will take pid namespace in the image
(00.005209) Add pid ns 9 pid 8132
(00.005216) Will take net namespace in the image
(00.005220) Add net ns 10 pid 8132
(00.005228) Will take ipc namespace in the image
(00.005233) Add ipc ns 11 pid 8132
(00.005240) Will take uts namespace in the image
(00.005244) Add uts ns 12 pid 8132
(00.005257) Will take mnt namespace in the image
(00.005262) Add mnt ns 13 pid 8132
(00.005271) Will take cgroup namespace in the image
(00.005276) Add cgroup ns 14 pid 8132
(00.005812) Lock network
(00.005818) Running network-lock scripts
(00.005823)     RPC
(00.011816)     type btrfs source /dev/nvme0n1p3 mnt_id 1073 s_dev 0x20 /root/var/lib/docker/btrfs/subvolumes/498bbeb030019cf9699525b71d9ba38bd0090798fe66021b4a1743efb063f48e @ ./ flags 0x280000 options ssd,space_cache,subvolid=1711,subvol=/root/var/lib/docker/btrfs/subvolumes/498bbeb030019cf9699525b71d9ba38bd0090798fe66021b4a1743efb063f48e
(00.011833)     type proc source proc mnt_id 1074 s_dev 0x45 / @ ./proc flags 0x20000e options 
(00.011840)     type tmpfs source tmpfs mnt_id 1075 s_dev 0x46 / @ ./dev flags 0x1000002 options size=65536k,mode=755,inode64
(00.011848)     type devpts source devpts mnt_id 1076 s_dev 0x47 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666
(00.011864)     type sysfs source sysfs mnt_id 1077 s_dev 0x48 / @ ./sys flags 0x20000f options 
(00.011872)     type cgroup2 source cgroup mnt_id 1078 s_dev 0x1b /system.slice/docker-bdf7c5243ada5a5d37774162fc1d86f4c72ae6b006d0df948a48576b333f012a.scope @ ./sys/fs/cgroup flags 0x20000f options nsdelegate
(00.011879)     type mqueue source mqueue mnt_id 1079 s_dev 0x44 / @ ./dev/mqueue flags 0x20000e options 
(00.011889)     type tmpfs source shm mnt_id 1080 s_dev 0x49 / @ ./dev/shm flags 0x20000e options size=65536k,inode64
(00.011911)     type btrfs source /dev/nvme0n1p3 mnt_id 1081 s_dev 0x20 /home/josevnz/webtop/config2 @ ./config flags 0x200000 options ssd,space_cache,subvolid=256,subvol=/home
(00.011919)     type btrfs source /dev/nvme0n1p3 mnt_id 1082 s_dev 0x20 /root/var/lib/docker/containers/bdf7c5243ada5a5d37774162fc1d86f4c72ae6b006d0df948a48576b333f012a/resolv.conf @ ./etc/resolv.conf flags 0x200000 options ssd,space_cache,subvolid=257,subvol=/root
(00.011926)     type btrfs source /dev/nvme0n1p3 mnt_id 1083 s_dev 0x20 /root/var/lib/docker/containers/bdf7c5243ada5a5d37774162fc1d86f4c72ae6b006d0df948a48576b333f012a/hostname @ ./etc/hostname flags 0x200000 options ssd,space_cache,subvolid=257,subvol=/root
(00.011935)     type btrfs source /dev/nvme0n1p3 mnt_id 1084 s_dev 0x20 /root/var/lib/docker/containers/bdf7c5243ada5a5d37774162fc1d86f4c72ae6b006d0df948a48576b333f012a/hosts @ ./etc/hosts flags 0x200000 options ssd,space_cache,subvolid=257,subvol=/root
(00.011948)     type proc source proc mnt_id 995 s_dev 0x45 /bus @ ./proc/bus flags 0x20000f options 
(00.011955)     type proc source proc mnt_id 996 s_dev 0x45 /fs @ ./proc/fs flags 0x20000f options 
(00.011960)     type proc source proc mnt_id 997 s_dev 0x45 /irq @ ./proc/irq flags 0x20000f options 
(00.011980)     type proc source proc mnt_id 998 s_dev 0x45 /sys @ ./proc/sys flags 0x20000f options 
(00.011987)     type proc source proc mnt_id 999 s_dev 0x45 /sysrq-trigger @ ./proc/sysrq-trigger flags 0x20000f options 
(00.011992)     type tmpfs source tmpfs mnt_id 1000 s_dev 0x4a / @ ./proc/asound flags 0x200001 options inode64
(00.011998)     type tmpfs source tmpfs mnt_id 1001 s_dev 0x4b / @ ./proc/acpi flags 0x200001 options inode64
(00.012004)     type tmpfs source tmpfs mnt_id 1002 s_dev 0x46 /null @ ./proc/kcore flags 0x1000002 options size=65536k,mode=755,inode64
(00.012010)     type tmpfs source tmpfs mnt_id 1003 s_dev 0x46 /null @ ./proc/keys flags 0x1000002 options size=65536k,mode=755,inode64
(00.012017)     type tmpfs source tmpfs mnt_id 1004 s_dev 0x46 /null @ ./proc/latency_stats flags 0x1000002 options size=65536k,mode=755,inode64
(00.012025)     type tmpfs source tmpfs mnt_id 1005 s_dev 0x46 /null @ ./proc/timer_list flags 0x1000002 options size=65536k,mode=755,inode64
(00.012032)     type tmpfs source tmpfs mnt_id 1006 s_dev 0x4c / @ ./proc/scsi flags 0x200001 options inode64
(00.012037)     type tmpfs source tmpfs mnt_id 1007 s_dev 0x4d / @ ./sys/firmware flags 0x200001 options inode64
(00.012044) mnt: Building mountpoints tree
(00.012048) mnt:    Building plain mount tree
(00.012053) mnt:        Working on 1007->1077
(00.012058) mnt:        Working on 1006->1074
(00.012063) mnt:        Working on 1005->1074
(00.012067) mnt:        Working on 1004->1074
(00.012072) mnt:        Working on 1003->1074
(00.012077) mnt:        Working on 1002->1074
(00.012081) mnt:        Working on 1001->1074
(00.012086) mnt:        Working on 1000->1074
(00.012091) mnt:        Working on 999->1074
(00.012095) mnt:        Working on 998->1074
(00.012100) mnt:        Working on 997->1074
(00.012105) mnt:        Working on 996->1074
(00.012110) mnt:        Working on 995->1074
(00.012114) mnt:        Working on 1084->1073
(00.012119) mnt:        Working on 1083->1073
(00.012123) mnt:        Working on 1082->1073
(00.012128) mnt:        Working on 1081->1073
(00.012133) mnt:        Working on 1080->1075
(00.012138) mnt:        Working on 1079->1075
(00.012142) mnt:        Working on 1078->1077
(00.012147) mnt:        Working on 1077->1073
(00.012152) mnt:        Working on 1076->1075
(00.012156) mnt:        Working on 1075->1073
(00.012161) mnt:        Working on 1074->1073
(00.012166) mnt:        Working on 1073->994
(00.012170) mnt:    Resorting children of 1073 in mount order
(00.012175) mnt:    Resorting children of 1084 in mount order
(00.012180) mnt:    Resorting children of 1083 in mount order
(00.012184) mnt:    Resorting children of 1082 in mount order
(00.012189) mnt:    Resorting children of 1081 in mount order
(00.012194) mnt:    Resorting children of 1077 in mount order
(00.012198) mnt:    Resorting children of 1078 in mount order
(00.012203) mnt:    Resorting children of 1007 in mount order
(00.012208) mnt:    Resorting children of 1075 in mount order
(00.012213) mnt:    Resorting children of 1080 in mount order
(00.012217) mnt:    Resorting children of 1079 in mount order
(00.012222) mnt:    Resorting children of 1076 in mount order
(00.012227) mnt:    Resorting children of 1074 in mount order
(00.012231) mnt:    Resorting children of 1006 in mount order
(00.012236) mnt:    Resorting children of 1005 in mount order
(00.012241) mnt:    Resorting children of 1004 in mount order
(00.012246) mnt:    Resorting children of 1003 in mount order
(00.012250) mnt:    Resorting children of 1002 in mount order
(00.012255) mnt:    Resorting children of 1001 in mount order
(00.012259) mnt:    Resorting children of 1000 in mount order
(00.012264) mnt:    Resorting children of 999 in mount order
(00.012269) mnt:    Resorting children of 998 in mount order
(00.012274) mnt:    Resorting children of 997 in mount order
(00.012278) mnt:    Resorting children of 996 in mount order
(00.012283) mnt:    Resorting children of 995 in mount order
(00.012290) mnt: Done:
(00.012295) mnt: [./](1073->994)
(00.012299) mnt:  [./etc/hosts](1084->1073)
(00.012304) mnt:  <--
(00.012309) mnt:  [./etc/hostname](1083->1073)
(00.012313) mnt:  <--
(00.012318) mnt:  [./etc/resolv.conf](1082->1073)
(00.012323) mnt:  <--
(00.012327) mnt:  [./config](1081->1073)
(00.012332) mnt:  <--
(00.012337) mnt:  [./sys](1077->1073)
(00.012341) mnt:   [./sys/fs/cgroup](1078->1077)
(00.012346) mnt:   <--
(00.012350) mnt:   [./sys/firmware](1007->1077)
(00.012355) mnt:   <--
(00.012359) mnt:  <--
(00.012364) mnt:  [./dev](1075->1073)
(00.012369) mnt:   [./dev/shm](1080->1075)
(00.012374) mnt:   <--
(00.012378) mnt:   [./dev/mqueue](1079->1075)
(00.012383) mnt:   <--
(00.012388) mnt:   [./dev/pts](1076->1075)
(00.012392) mnt:   <--
(00.012397) mnt:  <--
(00.012401) mnt:  [./proc](1074->1073)
(00.012406) mnt:   [./proc/scsi](1006->1074)
(00.012410) mnt:   <--
(00.012415) mnt:   [./proc/timer_list](1005->1074)
(00.012420) mnt:   <--
(00.012424) mnt:   [./proc/latency_stats](1004->1074)
(00.012429) mnt:   <--
(00.012433) mnt:   [./proc/keys](1003->1074)
(00.012438) mnt:   <--
(00.012443) mnt:   [./proc/kcore](1002->1074)
(00.012448) mnt:   <--
(00.012452) mnt:   [./proc/acpi](1001->1074)
(00.012457) mnt:   <--
(00.012462) mnt:   [./proc/asound](1000->1074)
(00.012466) mnt:   <--
(00.012471) mnt:   [./proc/sysrq-trigger](999->1074)
(00.012475) mnt:   <--
(00.012480) mnt:   [./proc/sys](998->1074)
(00.012485) mnt:   <--
(00.012489) mnt:   [./proc/irq](997->1074)
(00.012494) mnt:   <--
(00.012499) mnt:   [./proc/fs](996->1074)
(00.012503) mnt:   <--
(00.012508) mnt:   [./proc/bus](995->1074)
(00.012512) mnt:   <--
(00.012517) mnt:  <--
(00.012521) mnt: <--
(00.012531) mnt: Found /dev/null mapping for ./proc/timer_list mountpoint
(00.012535) mnt: Found /dev/null mapping for ./proc/latency_stats mountpoint
(00.012540) mnt: Found /dev/null mapping for ./proc/keys mountpoint
(00.012545) mnt: Found /dev/null mapping for ./proc/kcore mountpoint
(00.012551) mnt: Found /etc/hosts mapping for ./etc/hosts mountpoint
(00.012556) mnt: Found /etc/hostname mapping for ./etc/hostname mountpoint
(00.012560) mnt: Found /etc/resolv.conf mapping for ./etc/resolv.conf mountpoint
(00.012565) mnt: Found /config mapping for ./config mountpoint
(00.012570) mnt: Inspecting sharing on 1007 shared_id 0 master_id 0 (@./sys/firmware)
(00.012575) mnt: Inspecting sharing on 1006 shared_id 0 master_id 0 (@./proc/scsi)
(00.012580) mnt: Inspecting sharing on 1005 shared_id 0 master_id 0 (@./proc/timer_list)
(00.012584) mnt:    The mount 1004 is bind for 1005 (@./proc/latency_stats -> @./proc/timer_list)
(00.012589) mnt:    The mount 1003 is bind for 1005 (@./proc/keys -> @./proc/timer_list)
(00.012594) mnt:    The mount 1002 is bind for 1005 (@./proc/kcore -> @./proc/timer_list)
(00.012598) mnt:    The mount 1075 is bind for 1005 (@./dev -> @./proc/timer_list)
(00.012603) mnt: Inspecting sharing on 1004 shared_id 0 master_id 0 (@./proc/latency_stats)
(00.012607) mnt: Inspecting sharing on 1003 shared_id 0 master_id 0 (@./proc/keys)
(00.012612) mnt: Inspecting sharing on 1002 shared_id 0 master_id 0 (@./proc/kcore)
(00.012616) mnt: Inspecting sharing on 1001 shared_id 0 master_id 0 (@./proc/acpi)
(00.012621) mnt: Inspecting sharing on 1000 shared_id 0 master_id 0 (@./proc/asound)
(00.012626) mnt: Inspecting sharing on 999 shared_id 0 master_id 0 (@./proc/sysrq-trigger)
(00.012631) mnt:    The mount 998 is bind for 999 (@./proc/sys -> @./proc/sysrq-trigger)
(00.012635) mnt:    The mount 997 is bind for 999 (@./proc/irq -> @./proc/sysrq-trigger)
(00.012640) mnt:    The mount 996 is bind for 999 (@./proc/fs -> @./proc/sysrq-trigger)
(00.012644) mnt:    The mount 995 is bind for 999 (@./proc/bus -> @./proc/sysrq-trigger)
(00.012649) mnt:    The mount 1074 is bind for 999 (@./proc -> @./proc/sysrq-trigger)
(00.012654) mnt: Inspecting sharing on 998 shared_id 0 master_id 0 (@./proc/sys)
(00.012658) mnt: Inspecting sharing on 997 shared_id 0 master_id 0 (@./proc/irq)
(00.012663) mnt: Inspecting sharing on 996 shared_id 0 master_id 0 (@./proc/fs)
(00.012669) mnt: Inspecting sharing on 995 shared_id 0 master_id 0 (@./proc/bus)
(00.012674) mnt: Inspecting sharing on 1084 shared_id 0 master_id 0 (@./etc/hosts)
(00.012678) mnt:    The mount 1083 is bind for 1084 (@./etc/hostname -> @./etc/hosts)
(00.012683) mnt:    The mount 1082 is bind for 1084 (@./etc/resolv.conf -> @./etc/hosts)
(00.012688) mnt: Inspecting sharing on 1083 shared_id 0 master_id 0 (@./etc/hostname)
(00.012692) mnt: Inspecting sharing on 1082 shared_id 0 master_id 0 (@./etc/resolv.conf)
(00.012697) mnt: Inspecting sharing on 1081 shared_id 0 master_id 0 (@./config)
(00.012702) mnt: Inspecting sharing on 1080 shared_id 0 master_id 0 (@./dev/shm)
(00.012706) mnt: Inspecting sharing on 1079 shared_id 0 master_id 0 (@./dev/mqueue)
(00.012711) mnt: Inspecting sharing on 1078 shared_id 0 master_id 0 (@./sys/fs/cgroup)
(00.012716) mnt: Inspecting sharing on 1077 shared_id 0 master_id 0 (@./sys)
(00.012720) mnt: Inspecting sharing on 1076 shared_id 0 master_id 0 (@./dev/pts)
(00.012725) mnt: Inspecting sharing on 1075 shared_id 0 master_id 0 (@./dev)
(00.012729) mnt: Inspecting sharing on 1074 shared_id 0 master_id 0 (@./proc)
(00.012734) mnt: Inspecting sharing on 1073 shared_id 0 master_id -1 (@./)
(00.012740) Collecting netns 10/8132
(00.012744) Switching to 8132's net for collecting sockets
(00.012919) unix:   Collected: ino 88553 peer_ino 0 family    1 type    1 state 10 name s
(00.012927) unix:   Collected: ino 89911 peer_ino 0 family    1 type    5 state  7 name (null)
(00.013275) inet:   Collected: ino  0x14f39 family AF_INET    type SOCK_STREAM    port     4822 state TCP_LISTEN       src_addr 0.0.0.0
(00.013646) inet:   Collected: ino  0x161fb family AF_INET6   type SOCK_STREAM    port     3350 state TCP_LISTEN       src_addr ::ffff:127.0.0.1
(00.013651) inet:   Collected: ino  0x16222 family AF_INET6   type SOCK_STREAM    port     3000 state TCP_LISTEN       src_addr ::
(00.013656) inet:   Collected: ino  0x14f33 family AF_INET6   type SOCK_STREAM    port     3389 state TCP_LISTEN       src_addr ::
(00.013719) netlink: Collect netlink sock 0x15216
(00.013725) netlink: Collect netlink sock 0x15d4e
(00.013730) netlink: Collect netlink sock 0x15d74
(00.013734) netlink: Collect netlink sock 0x15f30
(00.013739) netlink: Collect netlink sock 0x15d75
(00.013744) netlink: Collect netlink sock 0x15d50
(00.013748) netlink: Collect netlink sock 0x15d52
(00.013753) netlink: Collect netlink sock 0x15d8a
(00.013764) netlink: Collect netlink sock 0x15d51
(00.013767) netlink: Collect netlink sock 0x15d4f
(00.013777) Collecting pidns 9/8132
(00.013801) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8132
(00.013809) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8171
(00.013817) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8174
(00.013830) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8192
(00.013838) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8206
(00.013845) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8193
(00.013852) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8194
(00.013860) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8369
(00.013868) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8381
(00.013876) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8195
(00.013885) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8342
(00.013893) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8196
(00.013930) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8352
(00.013934) seccomp:     Disable filter on tid_rea 8357, will be propagated
(00.013939) seccomp:     Disable filter on tid_rea 8358, will be propagated
(00.013944) seccomp:     Disable filter on tid_rea 8359, will be propagated
(00.013949) seccomp:     Disable filter on tid_rea 8360, will be propagated
(00.013953) seccomp:     Disable filter on tid_rea 8361, will be propagated
(00.013958) seccomp:     Disable filter on tid_rea 8363, will be propagated
(00.013965) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8197
(00.013970) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8334
(00.013983) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8198
(00.013989) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 8327
(00.014114) ========================================
(00.014120) Dumping task (pid: 8132)
(00.014124) ========================================
(00.014129) Obtaining task stat ... 
(00.014156) 
(00.014160) Collecting mappings (pid: 8132)
(00.014165) ----------------------------------------
(00.014245) Dumping path for -3 fd via self 15 [/package/admin/s6-2.11.1.2/command/s6-svscan]
(00.014303) vma 608000 borrows vfi from previous 400000
(00.014351) Collected, longest area occupies 33 pages
(00.014356) 0x400000-0x409000 (36K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x1
(00.014360) 0x608000-0x60a000 (8K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x8000 reg fp  shmid: 0x1
(00.014365) 0x1a44000-0x1a45000 (4K) prot 0 flags 0x22 fdflags 0 st 0x221 off 0 reg heap ap  shmid: 0
(00.014369) 0x1a45000-0x1a46000 (4K) prot 0x3 flags 0x22 fdflags 0 st 0x221 off 0 reg heap ap  shmid: 0
(00.014374) 0x7ffd862db000-0x7ffd862fc000 (132K) prot 0x3 flags 0x122 fdflags 0 st 0x201 off 0 reg ap  shmid: 0
(00.014379) 0x7ffd863cd000-0x7ffd863d1000 (16K) prot 0x1 flags 0x22 fdflags 0 st 0x1201 off 0 reg vvar ap  shmid: 0
(00.014384) 0x7ffd863d1000-0x7ffd863d3000 (8K) prot 0x5 flags 0x22 fdflags 0 st 0x209 off 0 reg vdso ap  shmid: 0
(00.014389) 0xffffffffff600000-0xffffffffff601000 (4K) prot 0x5 flags 0x22 fdflags 0 st 0x204 off 0 vsys ap  shmid: 0
(00.014393) ----------------------------------------
(00.014398) 
(00.014402) Collecting fds (pid: 8132)
(00.014407) ----------------------------------------
(00.014424) Found 7 file descriptors
(00.014429) ----------------------------------------
(00.014438) Dump private signals of 8132
(00.014442) Dump shared signals of 8132
(00.014451) Parasite syscall_ip at 0x400000
(00.014572) Set up parasite blob using memfd
(00.014578) Putting parasite blob into 0x7f9b2bc10000->0x7fdec3d7a000
(00.014626) Dumping general registers for 8132 in native mode
(00.014632) Dumping GP/FPU registers for 8132
(00.014643) x86: xsave runtime structure
(00.014647) x86: -----------------------
(00.014652) x86: cwd:0x37f swd:0 twd:0 fop:0 mxcsr:0x1f80 mxcsr_mask:0x2ffff
(00.014656) x86: magic1:0x46505853 extended_size:836 xstate_bv:0x7 xstate_size:832
(00.014661) x86: xstate_bv: 0x7
(00.014666) x86: -----------------------
(00.014670) Putting tsock into pid 8132
(00.014727) Wait for parasite being daemonized...
(00.014732) Wait for ack 2 on daemon socket
pie: 1: Running daemon thread leader
pie: 1: __sent ack msg: 2 2 0
pie: 1: Daemon waits for command
(00.014769) Fetched ack: 2 2 0
(00.014774) Parasite 8132 has been switched to daemon mode
(00.014782) Sent msg to daemon 72 0 0
pie: 1: __fetched msg: 72 0 0
pie: 1: __sent ack msg: 72 72 0
pie: 1: Daemon waits for command
(00.014812) Wait for ack 72 on daemon socket
(00.014817) Fetched ack: 72 72 0
(00.014821) vdso: vDSO hint is reliable - omit checking
(00.014827) Sent msg to daemon 74 0 0
(00.014830) Wait for ack 74 on daemon socket
pie: 1: __fetched msg: 74 0 0
pie: 1: __sent ack msg: 74 74 0
pie: 1: Daemon waits for command
(00.014847) Fetched ack: 74 74 0
(00.014853) Sent msg to daemon 70 0 0
(00.014856) Wait for ack 70 on daemon socket
pie: 1: __fetched msg: 70 0 0
pie: 1: __sent ack msg: 70 70 0
pie: 1: Daemon waits for command
(00.014874) Fetched ack: 70 70 0
(00.014879) sid=1 pgid=1 pid=1
(00.014892) 
(00.014896) Dumping opened files (pid: 8132)
(00.014901) ----------------------------------------
(00.014906) Sent msg to daemon 71 0 0
pie: 1: __fetched msg: 71 0 0
pie: 1: __sent ack msg: 71 71 0
pie: 1: Daemon waits for command
(00.014934) Wait for ack 71 on daemon socket
(00.014939) Fetched ack: 71 71 0
(00.014958) 8132 fdinfo 0: pos:                0 flags:           100000/0
(00.014973) Dumping path for 0 fd via self 18 [/dev/null]
(00.014995) 8132 fdinfo 1: pos:                0 flags:             4001/0
(00.015001) Dumping pipe 19 with id 0x3 pipe_id 0x15206
(00.015014) 8132 fdinfo 2: pos:                0 flags:             4001/0
(00.015025) Dumping pipe 20 with id 0x4 pipe_id 0x15207
(00.015041) lockinfo: 1:1 1 8132 00:20:81724 0 EOF
(00.015049) 8132 fdinfo 3: pos:                0 flags:           104001/0x1
(00.015059) Dumping path for 3 fd via self 21 [/run/service/.s6-svscan/lock]
(00.015066) Only file size could be stored for validation for file /run/service/.s6-svscan/lock
(00.015079) 8132 fdinfo 5: pos:                0 flags:           104000/0x1
(00.015089) Dumping path for 5 fd via self 22 [/run/service/.s6-svscan/control]
(00.015095) Dumping fifo 22 with id 0x6 pipe_id 0x13f3d
(00.015099) Dumping data from pipe 0x13f3d fd 22
(00.015139) 8132 fdinfo 6: pos:                0 flags:           104001/0x1
(00.015151) Dumping path for 6 fd via self 24 [/run/service/.s6-svscan/control]
(00.015157) Dumping fifo 24 with id 0x8 pipe_id 0x13f3d
(00.015170) 8132 fdinfo 7: pos:                0 flags:             4002/0x1
(00.015189) ----------------------------------------
(00.015199) Sent msg to daemon 65 0 0
(00.015202) Wait for ack 65 on daemon socket
pie: 1: __fetched msg: 65 0 0
pie: 1: __sent ack msg: 65 65 0
pie: 1: Daemon waits for command
(00.015221) Fetched ack: 65 65 0
(00.015226) 
(00.015231) Dumping pages (type: 58 pid: 8132)
(00.015235) ----------------------------------------
(00.015241)    Private vmas 33/52 pages
(00.015250) pagemap-cache: created for pid 8132 (takes 4096 bytes)
(00.015255) page-pipe: Create page pipe for 52 segs
(00.015259) page-pipe: Will grow page pipe (iov off is 0)
(00.015275) pagemap-cache: 8132: filling VMA 400000-409000 (36K) [l:400000 h:600000]
(00.015280) pagemap-cache:  8132:           400000-409000           nr:1     cov:36864
(00.015285) pagemap-cache:  8132: simple mode [l:400000 h:409000]
(00.015293) page-pipe: Add iov to page pipe (0 iovs, 0/52 total)
(00.015298) Pagemap generated: 1 pages (0 lazy) 0 holes
(00.015302) pagemap-cache: 8132: filling VMA 608000-60a000 (8K) [l:600000 h:800000]
(00.015307) pagemap-cache:  8132:           608000-60a000           nr:1     cov:8192
(00.015311) pagemap-cache:  8132: simple mode [l:608000 h:60a000]
(00.015317) page-pipe: Add iov to page pipe (1 iovs, 1/52 total)
(00.015322) Pagemap generated: 1 pages (0 lazy) 0 holes
(00.015327) pagemap-cache: 8132: filling VMA 1a44000-1a45000 (4K) [l:1a00000 h:1c00000]
(00.015331) pagemap-cache:  8132:          1a44000-1a45000          nr:1     cov:4096
(00.015336) pagemap-cache:  8132:          1a45000-1a46000          nr:2     cov:8192
(00.015341) pagemap-cache:  8132: cache  mode [l:1a44000 h:1c00000]
(00.015346) Pagemap generated: 0 pages (0 lazy) 0 holes
(00.015350) page-pipe: Will grow page pipe (iov off is 2)
(00.015356) page-pipe: Add iov to page pipe (0 iovs, 2/52 total)
(00.015361) Pagemap generated: 1 pages (0 lazy) 0 holes
(00.015365) pagemap-cache: 8132: filling VMA 7ffd862db000-7ffd862fc000 (132K) [l:7ffd86200000 h:7ffd86400000]
(00.015371) page-pipe: Add iov to page pipe (1 iovs, 3/52 total)
(00.015376) page-pipe: Add iov to page pipe (2 iovs, 4/52 total)
(00.015381) Pagemap generated: 3 pages (0 lazy) 0 holes
(00.015385) pagemap-cache: 8132: filling VMA 7ffd863cd000-7ffd863d1000 (16K) [l:7ffd86200000 h:7ffd86400000]
(00.015390) Pagemap generated: 0 pages (0 lazy) 0 holes
(00.015395) pagemap-cache: 8132: filling VMA 7ffd863d1000-7ffd863d3000 (8K) [l:7ffd86200000 h:7ffd86400000]
(00.015400) page-pipe: Will grow page pipe (iov off is 5)
(00.015405) page-pipe: Add iov to page pipe (0 iovs, 5/52 total)
(00.015410) Pagemap generated: 2 pages (0 lazy) 0 holes
(00.015415) page-pipe: Page pipe:
(00.015420) page-pipe: * 3 pipes 6/52 iovs:
(00.015424) page-pipe:  buf 2 pages, 2 iovs, flags: 0 pipe_off: 0 :
(00.015429) page-pipe:      0x400000 1
(00.015434) page-pipe:      0x609000 1
(00.015438) page-pipe:  buf 4 pages, 3 iovs, flags: 1 pipe_off: 0 :
(00.015443) page-pipe:      0x1a45000 1
(00.015448) page-pipe:      0x7ffd862f1000 1
(00.015453) page-pipe:      0x7ffd862fa000 2
(00.015457) page-pipe:  buf 2 pages, 1 iovs, flags: 0 pipe_off: 0 :
(00.015462) page-pipe:      0x7ffd863d1000 2
(00.015467) page-pipe: * 0 holes:
(00.015475) PPB: 2 pages 2 segs 16 pipe 0 off
(00.015481) Sent msg to daemon 66 0 0
pie: 1: __fetched msg: 66 0 0
(00.015487) Wait for ack 66 on daemon socket
pie: 1: __sent ack msg: 66 66 0
pie: 1: Daemon waits for command
(00.015508) Fetched ack: 66 66 0
(00.015513) PPB: 4 pages 3 segs 16 pipe 2 off
(00.015518) Sent msg to daemon 66 0 0
(00.015523) Wait for ack 66 on daemon socket
pie: 1: __fetched msg: 66 0 0
pie: 1: __sent ack msg: 66 66 0
pie: 1: Daemon waits for command
(00.015542) Fetched ack: 66 66 0
(00.015547) PPB: 2 pages 1 segs 16 pipe 5 off
(00.015551) Sent msg to daemon 66 0 0
(00.015556) Wait for ack 66 on daemon socket
pie: 1: __fetched msg: 66 0 0
pie: 1: __sent ack msg: 66 66 0
pie: 1: Daemon waits for command
(00.015577) Fetched ack: 66 66 0
(00.015583) page-xfer: Transferring pages:
(00.015587) page-xfer:  buf 2/2
(00.015592) page-xfer:  p 0x400000 [1]
(00.015601) page-xfer:  p 0x609000 [1]
(00.015608) page-xfer:  buf 4/3
(00.015612) page-xfer:  p 0x1a45000 [1]
(00.015618) page-xfer:  p 0x7ffd862f1000 [1]
(00.015623) page-xfer:  p 0x7ffd862fa000 [2]
(00.015629) page-xfer:  buf 2/1
(00.015634) page-xfer:  p 0x7ffd863d1000 [2]
(00.015647) page-pipe: Killing page pipe
(00.015656) ----------------------------------------
(00.015661) Sent msg to daemon 65 0 0
(00.015664) Wait for ack 65 on daemon socket
pie: 1: __fetched msg: 65 0 0
pie: 1: __sent ack msg: 65 65 0
pie: 1: Daemon waits for command
(00.015681) Fetched ack: 65 65 0
(00.015687) Sent msg to daemon 67 0 0
(00.015690) Wait for ack 67 on daemon socket
pie: 1: __fetched msg: 67 0 0
pie: 1: __sent ack msg: 67 67 0
pie: 1: Daemon waits for command
(00.015716) Fetched ack: 67 67 0
(00.015722) Sent msg to daemon 68 0 0
(00.015725) Wait for ack 68 on daemon socket
pie: 1: __fetched msg: 68 0 0
pie: 1: __sent ack msg: 68 68 0
pie: 1: Daemon waits for command
(00.015744) Fetched ack: 68 68 0
(00.015749) Sent msg to daemon 69 0 0
(00.015752) Wait for ack 69 on daemon socket
pie: 1: __fetched msg: 69 0 0
pie: 1: __sent ack msg: 69 69 0
pie: 1: Daemon waits for command
(00.015781) Fetched ack: 69 69 0
(00.015784) 
(00.015789) Dumping core (pid: 8132)
(00.015794) ----------------------------------------
(00.015798) Obtaining personality ... 
(00.015812) Sent msg to daemon 64 0 0
(00.015815) Wait for ack 64 on daemon socket
pie: 1: __fetched msg: 64 0 0
pie: 1: __sent ack msg: 64 64 0
pie: 1: Daemon waits for command
(00.015865) Fetched ack: 64 64 0
(00.015880) Error (criu/lsm.c:90): Invalid selinux context kernel
(00.015886) ----------------------------------------
(00.015890) Error (criu/cr-dump.c:1390): Dump core (pid: 8132) failed with -1
(00.015906) Waiting for 8132 to trap
(00.015921) Daemon 8132 exited trapping
(00.015927) Sent msg to daemon 3 0 0
(00.015932) Force no-breakpoints restore
(00.015952) 8132 was trapped
(00.015957) 8132 (native) is going to execute the syscall 45, required is 15
(00.015975) 8132 was trapped
(00.015979) `- Expecting exit
(00.015997) 8132 was trapped
(00.016001) 8132 (native) is going to execute the syscall 186, required is 15
(00.016020) 8132 was trapped
(00.016024) `- Expecting exit
(00.016042) 8132 was trapped
(00.016046) 8132 (native) is going to execute the syscall 1, required is 15
pie: 1: __fetched msg: 3 0 0
(00.016066) 8132 was trapped
(00.016071) `- Expecting exit
(00.016088) 8132 was trapped
(00.016093) 8132 (native) is going to execute the syscall 186, required is 15
(00.016110) 8132 was trapped
(00.016115) `- Expecting exit
(00.016133) 8132 was trapped
(00.016137) 8132 (native) is going to execute the syscall 186, required is 15
(00.016155) 8132 was trapped
(00.016159) `- Expecting exit
(00.016177) 8132 was trapped
(00.016181) 8132 (native) is going to execute the syscall 1, required is 15
pie: 1: 1: new_sp=0x7fdec3d7ff88 ip 0x406513
(00.016200) 8132 was trapped
(00.016204) `- Expecting exit
(00.016222) 8132 was trapped
(00.016227) 8132 (native) is going to execute the syscall 3, required is 15
(00.016248) 8132 was trapped
(00.016253) `- Expecting exit
(00.016271) 8132 was trapped
(00.016275) 8132 (native) is going to execute the syscall 3, required is 15
(00.016297) 8132 was trapped
(00.016302) `- Expecting exit
(00.016320) 8132 was trapped
(00.016324) 8132 (native) is going to execute the syscall 15, required is 15
(00.016342) 8132 was stopped
(00.016420) Unlock network
(00.016425) Running network-unlock scripts
(00.016430)     RPC
(00.021090) Unfreezing tasks into 1
(00.021110)     Unseizing 8132 into 1
(00.021124)     Unseizing 8171 into 1
(00.021134)     Unseizing 8174 into 1
(00.021144)     Unseizing 8192 into 1
(00.021155)     Unseizing 8206 into 1
(00.021166)     Unseizing 8193 into 1
(00.021178)     Unseizing 8194 into 1
(00.021188)     Unseizing 8369 into 1
(00.021212)     Unseizing 8381 into 1
(00.021239)     Unseizing 8195 into 1
(00.021262)     Unseizing 8342 into 1
(00.021288)     Unseizing 8196 into 1
(00.021304)     Unseizing 8352 into 1
(00.021334)     Unseizing 8197 into 1
(00.021360)     Unseizing 8334 into 1
(00.021378)     Unseizing 8198 into 1
(00.021389)     Unseizing 8327 into 1
(00.021463) Error (criu/cr-dump.c:1768): Dumping FAILED.
adrianreber commented 1 year ago

Please update to the latest version of CRIU and Podman. That should resolve your problems.

josevnz commented 1 year ago

PODMAN

(00.000258) Version: 3.15 (gitid 0)
(00.000295) Running on dmaf5 Linux 5.14.18-100.fc33.x86_64 #1 SMP Fri Nov 12 17:38:44 UTC 2021 x86_64
(00.000320) Loaded kdat cache from /run/criu/criu.kdat
(00.000363) ========================================
(00.000368) Dumping processes (pid: 9117)
(00.000372) ========================================
(00.000382) rlimit: RLIMIT_NOFILE unlimited for self
(00.000391) Running pre-dump scripts
(00.000410) irmap: Searching irmap cache in work dir
(00.000429) No irmap-cache image
(00.000435) irmap: Searching irmap cache in parent
(00.000444) irmap: No irmap cache
(00.000451) cpu: x86_family 23 x86_vendor_id AuthenticAMD x86_model_id AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
(00.000458) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 832
(00.000464) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
(00.000469) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
(00.000475) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.000676) cg-prop: Parsing controller "cpu"
(00.000685) cg-prop:    Strategy "replace"
(00.000692) cg-prop:    Property "cpu.shares"
(00.000697) cg-prop:    Property "cpu.cfs_period_us"
(00.000699) cg-prop:    Property "cpu.cfs_quota_us"
(00.000702) cg-prop:    Property "cpu.rt_period_us"
(00.000706) cg-prop:    Property "cpu.rt_runtime_us"
(00.000711) cg-prop: Parsing controller "memory"
(00.000715) cg-prop:    Strategy "replace"
(00.000718) cg-prop:    Property "memory.limit_in_bytes"
(00.000723) cg-prop:    Property "memory.memsw.limit_in_bytes"
(00.000725) cg-prop:    Property "memory.swappiness"
(00.000729) cg-prop:    Property "memory.soft_limit_in_bytes"
(00.000734) cg-prop:    Property "memory.move_charge_at_immigrate"
(00.000738) cg-prop:    Property "memory.oom_control"
(00.000743) cg-prop:    Property "memory.use_hierarchy"
(00.000748) cg-prop:    Property "memory.kmem.limit_in_bytes"
(00.000752) cg-prop:    Property "memory.kmem.tcp.limit_in_bytes"
(00.000757) cg-prop: Parsing controller "cpuset"
(00.000762) cg-prop:    Strategy "replace"
(00.000766) cg-prop:    Property "cpuset.cpus"
(00.000770) cg-prop:    Property "cpuset.mems"
(00.000775) cg-prop:    Property "cpuset.memory_migrate"
(00.000780) cg-prop:    Property "cpuset.cpu_exclusive"
(00.000784) cg-prop:    Property "cpuset.mem_exclusive"
(00.000789) cg-prop:    Property "cpuset.mem_hardwall"
(00.000794) cg-prop:    Property "cpuset.memory_spread_page"
(00.000798) cg-prop:    Property "cpuset.memory_spread_slab"
(00.000803) cg-prop:    Property "cpuset.sched_load_balance"
(00.000808) cg-prop:    Property "cpuset.sched_relax_domain_level"
(00.000812) cg-prop: Parsing controller "blkio"
(00.000817) cg-prop:    Strategy "replace"
(00.000822) cg-prop:    Property "blkio.weight"
(00.000826) cg-prop: Parsing controller "freezer"
(00.000831) cg-prop:    Strategy "replace"
(00.000835) cg-prop: Parsing controller "perf_event"
(00.000839) cg-prop:    Strategy "replace"
(00.000843) cg-prop: Parsing controller "net_cls"
(00.000848) cg-prop:    Strategy "replace"
(00.000853) cg-prop:    Property "net_cls.classid"
(00.000856) cg-prop: Parsing controller "net_prio"
(00.000861) cg-prop:    Strategy "replace"
(00.000865) cg-prop:    Property "net_prio.ifpriomap"
(00.000869) cg-prop: Parsing controller "pids"
(00.000873) cg-prop:    Strategy "replace"
(00.000878) cg-prop:    Property "pids.max"
(00.000882) cg-prop: Parsing controller "devices"
(00.000887) cg-prop:    Strategy "replace"
(00.000892) cg-prop:    Property "devices.list"
(00.000918) Preparing image inventory (version 1)
(00.000944) Add pid ns 1 pid 9404
(00.000955) Add net ns 2 pid 9404
(00.000965) Add ipc ns 3 pid 9404
(00.000976) Add uts ns 4 pid 9404
(00.000986) Add time ns 5 pid 9404
(00.001002) Add mnt ns 6 pid 9404
(00.001011) Add user ns 7 pid 9404
(00.001020) Add cgroup ns 8 pid 9404
(00.001024) cg: Dumping cgroups for 9404
(00.001038) cg:  `- New css ID 1
(00.001043) cg:     `- [] -> [/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-0a5d6c65-d145-41d0-80e6-ac6bfbb9f4f9.scope] [0]
(00.001056) cg: Set 1 is criu one
(00.001096) Detected cgroup V2 freezer
(00.001101) freezing processes: 100000 attempts with 100 ms steps
(00.001130) cgroup.freeze=0
(00.001242) cgroup.freeze=1
(00.001274) freezing processes: 0 attempts done
(00.001305) SEIZE 9117: success
(00.001315) SEIZE 9127: success
(00.001322) SEIZE 9128: success
(00.001329) SEIZE 9148: success
(00.001340) SEIZE 9149: success
(00.001349) SEIZE 9150: success
(00.001358) SEIZE 9151: success
(00.001366) SEIZE 9152: success
(00.001375) SEIZE 9153: success
(00.001382) SEIZE 9154: success
(00.001391) SEIZE 9162: success
(00.001403) SEIZE 9301: success
(00.001411) SEIZE 9308: success
(00.001416) SEIZE 9317: success
(00.001424) SEIZE 9327: success
(00.001430) SEIZE 9332: success
(00.001438) SEIZE 9333: success
(00.001445) SEIZE 9334: success
(00.001453) SEIZE 9335: success
(00.001463) SEIZE 9336: success
(00.001469) SEIZE 9338: success
(00.001476) SEIZE 9343: success
(00.001487) SEIZE 9356: success
(00.001743) Seized task 9117, state 1
(00.001751) seccomp: Collected tid_real 9117 mode 0x2
(00.001769) Collected (0 attempts, 0 in_progress)
(00.001791) Seized task 9127, state 0
(00.001827) seccomp: Collected tid_real 9127 mode 0x2
(00.001842) Collected (0 attempts, 0 in_progress)
(00.001856) Seized task 9128, state 0
(00.001888) seccomp: Collected tid_real 9128 mode 0x2
(00.001901) Collected (0 attempts, 0 in_progress)
(00.001915) Collected (0 attempts, 0 in_progress)
(00.001922) Collected 9128 in 1 state
(00.001936) Collected (-1 attempts, 0 in_progress)
(00.001944) Collected 9127 in 1 state
(00.001948) Seized task 9148, state 0
(00.001985) seccomp: Collected tid_real 9148 mode 0x2
(00.001998) Collected (0 attempts, 0 in_progress)
(00.002011) Seized task 9162, state 0
(00.002041) seccomp: Collected tid_real 9162 mode 0x2
(00.002052) Collected (0 attempts, 0 in_progress)
(00.002066) Collected (0 attempts, 0 in_progress)
(00.002074) Collected 9162 in 1 state
(00.002090) Collected (-1 attempts, 0 in_progress)
(00.002099) Collected 9148 in 1 state
(00.002104) Seized task 9149, state 0
(00.002154) seccomp: Collected tid_real 9149 mode 0x2
(00.002171) Collected (0 attempts, 0 in_progress)
(00.002185) Collected (0 attempts, 0 in_progress)
(00.002192) Collected 9149 in 1 state
(00.002197) Seized task 9150, state 0
(00.002229) seccomp: Collected tid_real 9150 mode 0x2
(00.002243) Collected (0 attempts, 0 in_progress)
(00.002259) Seized task 9343, state 0
(00.002298) seccomp: Collected tid_real 9343 mode 0x2
(00.002316) Collected (0 attempts, 0 in_progress)
(00.002330) Seized task 9356, state 0
(00.002369) seccomp: Collected tid_real 9356 mode 0x2
(00.002383) Collected (0 attempts, 0 in_progress)
(00.002395) Collected (0 attempts, 0 in_progress)
(00.002403) Collected 9356 in 1 state
(00.002418) Collected (-1 attempts, 0 in_progress)
(00.002425) Collected 9343 in 1 state
(00.002439) Collected (-1 attempts, 0 in_progress)
(00.002449) Collected 9150 in 1 state
(00.002453) Seized task 9151, state 0
(00.002485) seccomp: Collected tid_real 9151 mode 0x2
(00.002501) Collected (0 attempts, 0 in_progress)
(00.002519) Seized task 9317, state 0
(00.002554) seccomp: Collected tid_real 9317 mode 0x2
(00.002565) Collected (0 attempts, 0 in_progress)
(00.002578) Collected (0 attempts, 0 in_progress)
(00.002582) Collected 9317 in 1 state
(00.002598) Collected (-1 attempts, 0 in_progress)
(00.002605) Collected 9151 in 1 state
(00.002610) Seized task 9152, state 0
(00.002647) seccomp: Collected tid_real 9152 mode 0x2
(00.002673) Collected (0 attempts, 0 in_progress)
(00.002686) Seized task 9327, state 0
(00.002717) seccomp: Collected tid_real 9327 mode 0x2
(00.002741)     Seizing 9327's 9332 thread
(00.002774) seccomp: Collected tid_real 9332 mode 0x2
(00.002780)     Seizing 9327's 9333 thread
(00.002812) seccomp: Collected tid_real 9333 mode 0x2
(00.002821)     Seizing 9327's 9334 thread
(00.002851) seccomp: Collected tid_real 9334 mode 0x2
(00.002856)     Seizing 9327's 9335 thread
(00.002884) seccomp: Collected tid_real 9335 mode 0x2
(00.002889)     Seizing 9327's 9336 thread
(00.002933) seccomp: Collected tid_real 9336 mode 0x2
(00.002940)     Seizing 9327's 9338 thread
(00.002972) seccomp: Collected tid_real 9338 mode 0x2
(00.002995) Collected (-1 attempts, 0 in_progress)
(00.003044) Collected (0 attempts, 0 in_progress)
(00.003060) Collected 9327 in 1 state
(00.003076) Collected (-1 attempts, 0 in_progress)
(00.003081) Collected 9152 in 1 state
(00.003086) Seized task 9153, state 0
(00.003203) seccomp: Collected tid_real 9153 mode 0x2
(00.003219) Collected (0 attempts, 0 in_progress)
(00.003233) Seized task 9308, state 0
(00.003272) seccomp: Collected tid_real 9308 mode 0x2
(00.003283) Collected (0 attempts, 0 in_progress)
(00.003295) Collected (0 attempts, 0 in_progress)
(00.003300) Collected 9308 in 1 state
(00.003315) Collected (-1 attempts, 0 in_progress)
(00.003322) Collected 9153 in 1 state
(00.003327) Seized task 9154, state 0
(00.003360) seccomp: Collected tid_real 9154 mode 0x2
(00.003375) Collected (0 attempts, 0 in_progress)
(00.003388) Seized task 9301, state 0
(00.003419) seccomp: Collected tid_real 9301 mode 0x2
(00.003430) Collected (0 attempts, 0 in_progress)
(00.003442) Collected (0 attempts, 0 in_progress)
(00.003447) Collected 9301 in 1 state
(00.003460) Collected (-1 attempts, 0 in_progress)
(00.003464) Collected 9154 in 1 state
(00.003482) Collected (-1 attempts, 0 in_progress)
(00.003490) Collected 9117 in 1 state
(00.003517) Will take pid namespace in the image
(00.003522) Add pid ns 9 pid 9117
(00.003531) Will take net namespace in the image
(00.003535) Add net ns 10 pid 9117
(00.003545) Will take ipc namespace in the image
(00.003550) Add ipc ns 11 pid 9117
(00.003557) Will take uts namespace in the image
(00.003562) Add uts ns 12 pid 9117
(00.003587) Will take mnt namespace in the image
(00.003592) Add mnt ns 13 pid 9117
(00.003603) Will take cgroup namespace in the image
(00.003608) Add cgroup ns 14 pid 9117
(00.004281) Lock network
(00.004290) Running network-lock scripts
(00.011678)     type overlay source overlay mnt_id 1605 s_dev 0x43 / @ ./ flags 0x200000 options lowerdir=/var/lib/containers/storage/overlay/l/U34TIRC243O3QPM2XOABF5ALPF:/var/lib/containers/storage/overlay/l/6RSHI5HVJ53Q77AWQCMHGOKEOO:/var/lib/containers/storage/overlay/l/D2QFEG4OXCLCBHD2OZMXJWNBJA:/var/lib/containers/storage/overlay/l/OBH56BQSWSITRDOVGJ2OYPTDYT:/var/lib/containers/storage/overlay/l/O3WE74VP6NQXD6YX5IH2SVEZ66:/var/lib/containers/storage/overlay/l/JBCOYL4QZ2UYHRO5KAVRV4HTHU:/var/lib/containers/storage/overlay/l/ONNHOZ323UQ3HHEDZJNXBLFBMZ:/var/lib/containers/storage/overlay/l/QBRCXCZ4QRVOQ22C7DSTKZI66H:/var/lib/containers/storage/overlay/l/SHZINFPDR44E3Q5BDHVCX4LHXM:/var/lib/containers/storage/overlay/l/INGMSRG2D3KOLB7C3MVYXARABC:/var/lib/containers/storage/overlay/l/BLFWPQYT2MQHSL7T6JZPJSUL7V:/var/lib/containers/storage/overlay/l/JG3ORHX6TG2Q3V3QDMOMAWEHC6:/var/lib/containers/storage/overlay/l/MJNUW23GKHHACATDIUP42ZNG5R:/var/lib/containers/storage/overlay/l/QI2YQH5BBLX7FZXNRDDG437FWX:/var/lib/containers/storage/overlay/l/JRDDSPCRLWZQY6PBQXKREWVFAT:/var/lib/containers/storage/overlay/l/AGDXOVEBP6L27EQJRCPA2IRAFL,upperdir=/var/lib/containers/storage/overlay/30cfa2641181337382c966c087ea910c27a57601cbe4a687d61d58146ba12fc1/diff,workdir=/var/lib/containers/storage/overlay/30cfa2641181337382c966c087ea910c27a57601cbe4a687d61d58146ba12fc1/work
(00.011698)     type sysfs source sysfs mnt_id 1606 s_dev 0x46 / @ ./sys flags 0x20000f options 
(00.011706)     type proc source proc mnt_id 1607 s_dev 0x47 / @ ./proc flags 0x20000e options 
(00.011713)     type tmpfs source tmpfs mnt_id 1608 s_dev 0x48 / @ ./dev flags 0x1000002 options size=65536k,mode=755,inode64
(00.011721)     type btrfs source /dev/nvme0n1p3 mnt_id 1609 s_dev 0x20 /home/josevnz/webtop/config2 @ ./config flags 0x200000 options ssd,space_cache,subvolid=256,subvol=/home
(00.011730)     type tmpfs source tmpfs mnt_id 1610 s_dev 0x1a /containers/storage/overlay-containers/6d3d4aa6de70bf3eddcf19c21335486264df9f06d160233c33acf68cd3ee6c1d/userdata/hostname @ ./etc/hostname flags 0x1000000 options size=3213880k,nr_inodes=819200,mode=755,inode64
(00.011748)     type mqueue source mqueue mnt_id 1611 s_dev 0x45 / @ ./dev/mqueue flags 0x20000e options 
(00.011771)     type tmpfs source tmpfs mnt_id 1612 s_dev 0x1a /containers/storage/overlay-containers/6d3d4aa6de70bf3eddcf19c21335486264df9f06d160233c33acf68cd3ee6c1d/userdata/hosts @ ./etc/hosts flags 0x1000000 options size=3213880k,nr_inodes=819200,mode=755,inode64
(00.011779)     type devpts source devpts mnt_id 1613 s_dev 0x49 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666
(00.011796)     type tmpfs source tmpfs mnt_id 1614 s_dev 0x1a /containers/storage/overlay-containers/6d3d4aa6de70bf3eddcf19c21335486264df9f06d160233c33acf68cd3ee6c1d/userdata/localtime @ ./usr/share/zoneinfo/Etc/UTC flags 0x1000000 options size=3213880k,nr_inodes=819200,mode=755,inode64
(00.011807)     type tmpfs source tmpfs mnt_id 1615 s_dev 0x1a /containers/storage/overlay-containers/6d3d4aa6de70bf3eddcf19c21335486264df9f06d160233c33acf68cd3ee6c1d/userdata/.containerenv @ ./run/.containerenv flags 0x1000000 options size=3213880k,nr_inodes=819200,mode=755,inode64
(00.011817)     type tmpfs source tmpfs mnt_id 1616 s_dev 0x1a /containers/storage/overlay-containers/6d3d4aa6de70bf3eddcf19c21335486264df9f06d160233c33acf68cd3ee6c1d/userdata/run/secrets @ ./run/secrets flags 0x1000000 options size=3213880k,nr_inodes=819200,mode=755,inode64
(00.011842)     type tmpfs source tmpfs mnt_id 1617 s_dev 0x1a /containers/storage/overlay-containers/6d3d4aa6de70bf3eddcf19c21335486264df9f06d160233c33acf68cd3ee6c1d/userdata/resolv.conf @ ./etc/resolv.conf flags 0x1000000 options size=3213880k,nr_inodes=819200,mode=755,inode64
(00.011849)     type tmpfs source shm mnt_id 1618 s_dev 0x42 / @ ./dev/shm flags 0x200000 options size=64000k,inode64
(00.011856)     type cgroup2 source cgroup2 mnt_id 1619 s_dev 0x1b /machine.slice/libpod-6d3d4aa6de70bf3eddcf19c21335486264df9f06d160233c33acf68cd3ee6c1d.scope/container @ ./sys/fs/cgroup flags 0x20000f options nsdelegate
(00.011863)     type tmpfs source tmpfs mnt_id 1620 s_dev 0x4a / @ ./proc/acpi flags 0x200001 options size=0k,inode64
(00.011871)     type devtmpfs source devtmpfs mnt_id 1621 s_dev 0x5 /null @ ./proc/kcore flags 0x1000000 options size=8007864k,nr_inodes=2001966,mode=755,inode64
(00.011880)     type devtmpfs source devtmpfs mnt_id 1622 s_dev 0x5 /null @ ./proc/keys flags 0x1000000 options size=8007864k,nr_inodes=2001966,mode=755,inode64
(00.011887)     type devtmpfs source devtmpfs mnt_id 1623 s_dev 0x5 /null @ ./proc/latency_stats flags 0x1000000 options size=8007864k,nr_inodes=2001966,mode=755,inode64
(00.011894)     type devtmpfs source devtmpfs mnt_id 1624 s_dev 0x5 /null @ ./proc/timer_list flags 0x1000000 options size=8007864k,nr_inodes=2001966,mode=755,inode64
(00.011901)     type tmpfs source tmpfs mnt_id 1625 s_dev 0x4b / @ ./proc/scsi flags 0x200001 options size=0k,inode64
(00.011918)     type tmpfs source tmpfs mnt_id 1626 s_dev 0x4c / @ ./sys/firmware flags 0x200001 options size=0k,inode64
(00.011925)     type tmpfs source tmpfs mnt_id 1627 s_dev 0x4d / @ ./sys/fs/selinux flags 0x200001 options size=0k,inode64
(00.011931)     type tmpfs source tmpfs mnt_id 1628 s_dev 0x4e / @ ./sys/dev/block flags 0x200001 options size=0k,inode64
(00.011938)     type proc source proc mnt_id 1629 s_dev 0x47 /asound @ ./proc/asound flags 0x200001 options 
(00.011945)     type proc source proc mnt_id 1630 s_dev 0x47 /bus @ ./proc/bus flags 0x200001 options 
(00.011951)     type proc source proc mnt_id 1631 s_dev 0x47 /fs @ ./proc/fs flags 0x200001 options 
(00.011958)     type proc source proc mnt_id 1632 s_dev 0x47 /irq @ ./proc/irq flags 0x200001 options 
(00.011963)     type proc source proc mnt_id 1633 s_dev 0x47 /sys @ ./proc/sys flags 0x200001 options 
(00.011972)     type proc source proc mnt_id 1634 s_dev 0x47 /sysrq-trigger @ ./proc/sysrq-trigger flags 0x200001 options 
(00.011979) mnt: Building mountpoints tree
(00.011984) mnt:    Building plain mount tree
(00.011989) mnt:        Working on 1634->1607
(00.011993) mnt:        Working on 1633->1607
(00.011998) mnt:        Working on 1632->1607
(00.012002) mnt:        Working on 1631->1607
(00.012007) mnt:        Working on 1630->1607
(00.012019) mnt:        Working on 1629->1607
(00.012023) mnt:        Working on 1628->1606
(00.012028) mnt:        Working on 1627->1606
(00.012033) mnt:        Working on 1626->1606
(00.012037) mnt:        Working on 1625->1607
(00.012042) mnt:        Working on 1624->1607
(00.012047) mnt:        Working on 1623->1607
(00.012052) mnt:        Working on 1622->1607
(00.012056) mnt:        Working on 1621->1607
(00.012074) mnt:        Working on 1620->1607
(00.012078) mnt:        Working on 1619->1606
(00.012082) mnt:        Working on 1618->1608
(00.012087) mnt:        Working on 1617->1605
(00.012091) mnt:        Working on 1616->1605
(00.012096) mnt:        Working on 1615->1605
(00.012101) mnt:        Working on 1614->1605
(00.012106) mnt:        Working on 1613->1608
(00.012110) mnt:        Working on 1612->1605
(00.012174) mnt:        Working on 1611->1608
(00.012179) mnt:        Working on 1610->1605
(00.012183) mnt:        Working on 1609->1605
(00.012188) mnt:        Working on 1608->1605
(00.012193) mnt:        Working on 1607->1605
(00.012198) mnt:        Working on 1606->1605
(00.012202) mnt:        Working on 1605->1466
(00.012207) mnt:    Resorting children of 1605 in mount order
(00.012212) mnt:    Resorting children of 1614 in mount order
(00.012216) mnt:    Resorting children of 1617 in mount order
(00.012221) mnt:    Resorting children of 1616 in mount order
(00.012226) mnt:    Resorting children of 1615 in mount order
(00.012230) mnt:    Resorting children of 1612 in mount order
(00.012235) mnt:    Resorting children of 1610 in mount order
(00.012240) mnt:    Resorting children of 1609 in mount order
(00.012244) mnt:    Resorting children of 1608 in mount order
(00.012249) mnt:    Resorting children of 1618 in mount order
(00.012254) mnt:    Resorting children of 1613 in mount order
(00.012259) mnt:    Resorting children of 1611 in mount order
(00.012263) mnt:    Resorting children of 1607 in mount order
(00.012268) mnt:    Resorting children of 1634 in mount order
(00.012273) mnt:    Resorting children of 1633 in mount order
(00.012277) mnt:    Resorting children of 1632 in mount order
(00.012282) mnt:    Resorting children of 1631 in mount order
(00.012287) mnt:    Resorting children of 1630 in mount order
(00.012292) mnt:    Resorting children of 1629 in mount order
(00.012296) mnt:    Resorting children of 1625 in mount order
(00.012301) mnt:    Resorting children of 1624 in mount order
(00.012305) mnt:    Resorting children of 1623 in mount order
(00.012310) mnt:    Resorting children of 1622 in mount order
(00.012315) mnt:    Resorting children of 1621 in mount order
(00.012320) mnt:    Resorting children of 1620 in mount order
(00.012324) mnt:    Resorting children of 1606 in mount order
(00.012329) mnt:    Resorting children of 1628 in mount order
(00.012334) mnt:    Resorting children of 1627 in mount order
(00.012338) mnt:    Resorting children of 1619 in mount order
(00.012343) mnt:    Resorting children of 1626 in mount order
(00.012348) mnt: Done:
(00.012352) mnt: [./](1605->1466)
(00.012357) mnt:  [./usr/share/zoneinfo/Etc/UTC](1614->1605)
(00.012361) mnt:  <--
(00.012366) mnt:  [./etc/resolv.conf](1617->1605)
(00.012371) mnt:  <--
(00.012375) mnt:  [./run/secrets](1616->1605)
(00.012380) mnt:  <--
(00.012385) mnt:  [./run/.containerenv](1615->1605)
(00.012390) mnt:  <--
(00.012394) mnt:  [./etc/hosts](1612->1605)
(00.012399) mnt:  <--
(00.012404) mnt:  [./etc/hostname](1610->1605)
(00.012408) mnt:  <--
(00.012413) mnt:  [./config](1609->1605)
(00.012418) mnt:  <--
(00.012422) mnt:  [./dev](1608->1605)
(00.012427) mnt:   [./dev/shm](1618->1608)
(00.012432) mnt:   <--
(00.012436) mnt:   [./dev/pts](1613->1608)
(00.012441) mnt:   <--
(00.012445) mnt:   [./dev/mqueue](1611->1608)
(00.012450) mnt:   <--
(00.012455) mnt:  <--
(00.012460) mnt:  [./proc](1607->1605)
(00.012464) mnt:   [./proc/sysrq-trigger](1634->1607)
(00.012469) mnt:   <--
(00.012474) mnt:   [./proc/sys](1633->1607)
(00.012478) mnt:   <--
(00.012483) mnt:   [./proc/irq](1632->1607)
(00.012488) mnt:   <--
(00.012492) mnt:   [./proc/fs](1631->1607)
(00.012497) mnt:   <--
(00.012502) mnt:   [./proc/bus](1630->1607)
(00.012506) mnt:   <--
(00.012511) mnt:   [./proc/asound](1629->1607)
(00.012515) mnt:   <--
(00.012526) mnt:   [./proc/scsi](1625->1607)
(00.012530) mnt:   <--
(00.012535) mnt:   [./proc/timer_list](1624->1607)
(00.012540) mnt:   <--
(00.012545) mnt:   [./proc/latency_stats](1623->1607)
(00.012549) mnt:   <--
(00.012554) mnt:   [./proc/keys](1622->1607)
(00.012558) mnt:   <--
(00.012563) mnt:   [./proc/kcore](1621->1607)
(00.012568) mnt:   <--
(00.012573) mnt:   [./proc/acpi](1620->1607)
(00.012577) mnt:   <--
(00.012582) mnt:  <--
(00.012587) mnt:  [./sys](1606->1605)
(00.012591) mnt:   [./sys/dev/block](1628->1606)
(00.012596) mnt:   <--
(00.012600) mnt:   [./sys/fs/selinux](1627->1606)
(00.012605) mnt:   <--
(00.012610) mnt:   [./sys/fs/cgroup](1619->1606)
(00.012614) mnt:   <--
(00.012619) mnt:   [./sys/firmware](1626->1606)
(00.012624) mnt:   <--
(00.012629) mnt:  <--
(00.012633) mnt: <--
(00.012647) mnt: Found /proc/timer_list mapping for ./proc/timer_list mountpoint
(00.012651) mnt: Found /proc/latency_stats mapping for ./proc/latency_stats mountpoint
(00.012656) mnt: Found /proc/keys mapping for ./proc/keys mountpoint
(00.012660) mnt: Found /proc/kcore mapping for ./proc/kcore mountpoint
(00.012664) mnt: Found /dev/shm mapping for ./dev/shm mountpoint
(00.012669) mnt: Found /etc/resolv.conf mapping for ./etc/resolv.conf mountpoint
(00.012674) mnt: Found /run/secrets mapping for ./run/secrets mountpoint
(00.012679) mnt: Found /run/.containerenv mapping for ./run/.containerenv mountpoint
(00.012683) mnt: Found /etc/hosts mapping for ./etc/hosts mountpoint
(00.012688) mnt: Found /etc/hostname mapping for ./etc/hostname mountpoint
(00.012691) mnt: Found /config mapping for ./config mountpoint
(00.012697) mnt: Inspecting sharing on 1634 shared_id 0 master_id 0 (@./proc/sysrq-trigger)
(00.012702) mnt:    The mount 1633 is bind for 1634 (@./proc/sys -> @./proc/sysrq-trigger)
(00.012707) mnt:    The mount 1632 is bind for 1634 (@./proc/irq -> @./proc/sysrq-trigger)
(00.012712) mnt:    The mount 1631 is bind for 1634 (@./proc/fs -> @./proc/sysrq-trigger)
(00.012716) mnt:    The mount 1630 is bind for 1634 (@./proc/bus -> @./proc/sysrq-trigger)
(00.012721) mnt:    The mount 1629 is bind for 1634 (@./proc/asound -> @./proc/sysrq-trigger)
(00.012725) mnt:    The mount 1607 is bind for 1634 (@./proc -> @./proc/sysrq-trigger)
(00.012729) mnt: Inspecting sharing on 1633 shared_id 0 master_id 0 (@./proc/sys)
(00.012734) mnt: Inspecting sharing on 1632 shared_id 0 master_id 0 (@./proc/irq)
(00.012739) mnt: Inspecting sharing on 1631 shared_id 0 master_id 0 (@./proc/fs)
(00.012743) mnt: Inspecting sharing on 1630 shared_id 0 master_id 0 (@./proc/bus)
(00.012748) mnt: Inspecting sharing on 1629 shared_id 0 master_id 0 (@./proc/asound)
(00.012753) mnt: Inspecting sharing on 1628 shared_id 0 master_id 0 (@./sys/dev/block)
(00.012757) mnt: Inspecting sharing on 1627 shared_id 0 master_id 0 (@./sys/fs/selinux)
(00.012762) mnt: Inspecting sharing on 1626 shared_id 0 master_id 0 (@./sys/firmware)
(00.012766) mnt: Inspecting sharing on 1625 shared_id 0 master_id 0 (@./proc/scsi)
(00.012771) mnt: Inspecting sharing on 1624 shared_id 0 master_id 0 (@./proc/timer_list)
(00.012776) mnt:    The mount 1623 is bind for 1624 (@./proc/latency_stats -> @./proc/timer_list)
(00.012781) mnt:    The mount 1622 is bind for 1624 (@./proc/keys -> @./proc/timer_list)
(00.012784) mnt:    The mount 1621 is bind for 1624 (@./proc/kcore -> @./proc/timer_list)
(00.012787) mnt: Inspecting sharing on 1623 shared_id 0 master_id 0 (@./proc/latency_stats)
(00.012792) mnt: Inspecting sharing on 1622 shared_id 0 master_id 0 (@./proc/keys)
(00.012797) mnt: Inspecting sharing on 1621 shared_id 0 master_id 0 (@./proc/kcore)
(00.012801) mnt: Inspecting sharing on 1620 shared_id 0 master_id 0 (@./proc/acpi)
(00.012805) mnt: Inspecting sharing on 1619 shared_id 0 master_id 0 (@./sys/fs/cgroup)
(00.012810) mnt: Inspecting sharing on 1618 shared_id 0 master_id 0 (@./dev/shm)
(00.012818) mnt: Inspecting sharing on 1617 shared_id 0 master_id 0 (@./etc/resolv.conf)
(00.012821) mnt:    The mount 1616 is bind for 1617 (@./run/secrets -> @./etc/resolv.conf)
(00.012825) mnt:    The mount 1615 is bind for 1617 (@./run/.containerenv -> @./etc/resolv.conf)
(00.012833) mnt:    The mount 1614 is bind for 1617 (@./usr/share/zoneinfo/Etc/UTC -> @./etc/resolv.conf)
(00.012837) mnt:    The mount 1612 is bind for 1617 (@./etc/hosts -> @./etc/resolv.conf)
(00.012842) mnt:    The mount 1610 is bind for 1617 (@./etc/hostname -> @./etc/resolv.conf)
(00.012846) mnt: Inspecting sharing on 1616 shared_id 0 master_id 0 (@./run/secrets)
(00.012850) mnt: Inspecting sharing on 1615 shared_id 0 master_id 0 (@./run/.containerenv)
(00.012857) mnt: Inspecting sharing on 1614 shared_id 0 master_id 0 (@./usr/share/zoneinfo/Etc/UTC)
(00.012862) mnt: Inspecting sharing on 1613 shared_id 0 master_id 0 (@./dev/pts)
(00.012866) mnt: Inspecting sharing on 1612 shared_id 0 master_id 0 (@./etc/hosts)
(00.012871) mnt: Inspecting sharing on 1611 shared_id 0 master_id 0 (@./dev/mqueue)
(00.012877) mnt: Inspecting sharing on 1610 shared_id 0 master_id 0 (@./etc/hostname)
(00.012882) mnt: Inspecting sharing on 1609 shared_id 0 master_id 0 (@./config)
(00.012886) mnt: Inspecting sharing on 1608 shared_id 0 master_id 0 (@./dev)
(00.012888) mnt: Inspecting sharing on 1607 shared_id 0 master_id 0 (@./proc)
(00.012892) mnt: Inspecting sharing on 1606 shared_id 0 master_id 0 (@./sys)
(00.012895) mnt: Inspecting sharing on 1605 shared_id 0 master_id 0 (@./)
(00.012900) Error (criu/mount.c:662): mnt: 1614:./usr/share/zoneinfo/Etc/UTC doesn't have a proper root mount
(00.012917) Unlock network
(00.012922) Running network-unlock scripts
(00.019529) Unfreezing tasks into 1
(00.019547)     Unseizing 9117 into 1
(00.019558)     Unseizing 9127 into 1
(00.019567)     Unseizing 9128 into 1
(00.019580)     Unseizing 9148 into 1
(00.019587)     Unseizing 9162 into 1
(00.019597)     Unseizing 9149 into 1
(00.019606)     Unseizing 9150 into 1
(00.019620)     Unseizing 9343 into 1
(00.019626)     Unseizing 9356 into 1
(00.019638)     Unseizing 9151 into 1
(00.019646)     Unseizing 9317 into 1
(00.019653)     Unseizing 9152 into 1
(00.019664)     Unseizing 9327 into 1
(00.019705)     Unseizing 9153 into 1
(00.019712)     Unseizing 9308 into 1
(00.019719)     Unseizing 9154 into 1
(00.019724)     Unseizing 9301 into 1
(00.019758) Error (criu/cr-dump.c:1768): Dumping FAILED.
josevnz commented 1 year ago

Hello @adrianreber,

What version of CRIU should I use? I can try to upgrade, want to make sure I'm using the right one.

Thanks.

adrianreber commented 1 year ago

Just use the latest. I seems you are using Fedora 33 which went EOL over a year ago.

The latest Podman error log you are posting could be solved by not specifying --tz America/New_York.

josevnz commented 1 year ago

Just use the latest. I seems you are using Fedora 33 which went EOL over a year ago.

Will do.

The latest Podman error log you are posting could be solved by not specifying --tz America/New_York.

I pass --tz or --env TZ= in Docker because it is needed in this case. CRIU checkpoint should not crash because of that.

But for sake of argument I tried that too and it dies with a new error message now:

(00.016250) Fetched ack: 64 64 0
(00.016266) Error (criu/lsm.c:90): Invalid selinux context kernel
(00.016272) ----------------------------------------
(00.016277) Error (criu/cr-dump.c:1390): Dump core (pid: 10500) failed with -1
(00.016299) Waiting for 10500 to trap
...
(00.022875)     Unseizing 10537 into 1
(00.022882)     Unseizing 10684 into 1
(00.022912) Error (criu/cr-dump.c:1768): Dumping FAILED.

As you suggested will try again with a newer version of the tools and OS.

Please do not close the issue yet as I want to confirm this will go away with the new version.

adrianreber commented 1 year ago

Just use the latest. I seems you are using Fedora 33 which went EOL over a year ago.

Will do.

The latest Podman error log you are posting could be solved by not specifying --tz America/New_York.

I pass --tz or --env TZ= in Docker because it is needed in this case. CRIU checkpoint should not crash because of that.

Well, it does not crash. The problem is just that the timezone handling is done in a way that cannot be handled by CRIU. This has already been discussed in Podman. But the mountpoint of the timezone is not correctly specified in config.json and so runc/crun do not have the correct information to tell CRIU about this mountpoint.

We would love to have a fix for it. Please contribute a fix.

But for sake of argument I tried that too and it dies with a new error message now:

(00.016250) Fetched ack: 64 64 0
(00.016266) Error (criu/lsm.c:90): Invalid selinux context kernel
(00.016272) ----------------------------------------
(00.016277) Error (criu/cr-dump.c:1390): Dump core (pid: 10500) failed with -1
(00.016299) Waiting for 10500 to trap
...
(00.022875)     Unseizing 10537 into 1
(00.022882)     Unseizing 10684 into 1
(00.022912) Error (criu/cr-dump.c:1768): Dumping FAILED.

As you suggested will try again with a newer version of the tools and OS.

Please do not close the issue yet as I want to confirm this will go away with the new version.

It seems you have a non standard setup of SELinux. This should be fixed in newer versions of CRIU.

But if you require a timezone in your container, this is not something CRIU can fix as long as the container is not correctly setup by the Podman/Docker. So far these discussion did not lead anywhere, but if you have an idea how to solve it please provide a fix.

josevnz commented 1 year ago

I did not create that Image but I'll crack it open to see what they are doing there, will report my findings here.

I was about to test on another distro (Ubuntu Focal 20.04) but then I saw this issue related to the OS, so i will stick with Fedora for the testing.

adrianreber commented 1 year ago

Yes, latest Fedora or RHEL clone should work really well. Depending on the Ubuntu kernel it can work if it is new enough. On 22.04 you have to watch out to install a newer version of CRIU from our repository. The CRIU version in Ubuntu is too old and actually completely broken due to missing support for restartable sequences.

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

josevnz commented 1 year ago

Hello,

Upgraded to the following versions of docker, podman and criu (enabled also docker experimental settings on docker),

Docker version

[josevnz@dmaf5 ~]$ rpm -qi docker-ce 
Name        : docker-ce
Epoch       : 3
Version     : 20.10.22
Release     : 3.fc37
Architecture: x86_64
Install Date: Fri 30 Dec 2022 07:56:35 PM EST
Group       : Tools/Docker
Size        : 87739886
License     : ASL 2.0
Signature   : RSA/SHA512, Fri 16 Dec 2022 03:39:02 AM EST, Key ID c52feb6b621e9f35
Source RPM  : docker-ce-20.10.22-3.fc37.src.rpm
Build Date  : Thu 15 Dec 2022 05:26:12 PM EST
Build Host  : 6e7c0d90a1d1
Packager    : Docker <support@docker.com>
Vendor      : Docker
URL         : https://www.docker.com
Summary     : The open-source application container engine
Description :
Docker is a product for you to build, ship and run any application as a
lightweight container.

Docker containers are both hardware-agnostic and platform-agnostic. This means
they can run anywhere, from your laptop to the largest cloud compute instance
and everything in between - and they don't require you to use a particular
language, framework or packaging system. That makes them great building blocks
for deploying and scaling web apps, databases, and backend services without
depending on a particular stack or provider.

Criu version

[josevnz@dmaf5 ~]$ rpm -qi criu
criu       criu-libs  
[josevnz@dmaf5 ~]$ rpm -qi criu
Name        : criu
Version     : 3.17.1
Release     : 3.fc37
Architecture: x86_64
Install Date: Sat 05 Nov 2022 04:43:39 AM EDT
Group       : Unspecified
Size        : 1545609
License     : GPLv2
Signature   : RSA/SHA256, Wed 20 Jul 2022 08:33:11 PM EDT, Key ID f55ad3fb5323552a
Source RPM  : criu-3.17.1-3.fc37.src.rpm
Build Date  : Wed 20 Jul 2022 07:59:41 PM EDT
Build Host  : buildhw-x86-07.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://criu.org/
Bug URL     : https://bugz.fedoraproject.org/criu
Summary     : Tool for Checkpoint/Restore in User-space
Description :
criu is the user-space part of Checkpoint/Restore in User-space
(CRIU), a project to implement checkpoint/restore functionality for
Linux in user-space.

Podman version

[josevnz@dmaf5 ~]$ rpm -qi podman
Name        : podman
Epoch       : 4
Version     : 4.3.1
Release     : 1.fc37
Architecture: x86_64
Install Date: Fri 30 Dec 2022 09:21:59 PM EST
Group       : Unspecified
Size        : 43050129
License     : ASL 2.0 and BSD and ISC and MIT and MPLv2.0
Signature   : RSA/SHA256, Fri 11 Nov 2022 11:35:55 AM EST, Key ID f55ad3fb5323552a
Source RPM  : podman-4.3.1-1.fc37.src.rpm
Build Date  : Fri 11 Nov 2022 10:01:24 AM EST
Build Host  : buildhw-x86-09.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://podman.io/
Bug URL     : https://bugz.fedoraproject.org/podman
Summary     : Manage Pods, Containers and Container Images
Description :
podman (Pod Manager) is a fully featured container engine that is a simple
daemonless tool.  podman provides a Docker-CLI comparable command line that
eases the transition from other container engines and allows the management of
pods, containers and images.  Simply put: alias docker=podman.
Most podman commands can be run as a regular user, without requiring
additional privileges.

podman uses Buildah(1) internally to create container images.
Both tools share image (not container) storage, hence each can use or
manipulate images (but not containers) created by the other.

Manage Pods, Containers and Container Images
podman Simple management tool for pods, containers and images

Got the following error with Docker:

[josevnz@dmaf5 ~]$ docker run     --detach     --name webtop-test     --env PUID=1000     --env PGID=1000     --env TZ=America/New_York     --env TITLE='Webtop test'     --publish 3000:3000     --volume /home/josevnz/webtop/config2:/config     lscr.io/linuxserver/webtop:ubuntu-kde
[josevnz@dmaf5 ~]$ docker ps
CONTAINER ID   IMAGE                                   COMMAND                  CREATED          STATUS                        PORTS                                                 NAMES
8e6a55bfae2a   lscr.io/linuxserver/webtop:ubuntu-kde   "/init"                  43 seconds ago   Up 42 seconds                 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp, 3389/tcp   webtop-test
[josevnz@dmaf5 ~]$ docker logs  webtop-test
[custom-init] No custom services found, skipping...
[migrations] started
[migrations] no migrations found

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/

Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    1000
User gid:    1000
-------------------------------------

Generating 2048 bit rsa key...

ssl_gen_key_xrdp1 ok

saving to rsakeys.ini

..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..+....+.....+......+.............+.........+...+..+...+..................+....+......+..+...+................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.........+...+..+.+.....+...+....+........+............+.+..............+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+...+..+.+.........+...+..+......+...+.........+...............+.............+.....+....+.....+.+..+...............+......+......+......+...+.+.....+......+............+..........+.....+...+..........+.........+...+..................+............+...+..+..............................+.+...........+......+...+.+...........+...+.+......+........+.......+........+....+...+.....+...............+.......+...+..............+.............+...............+..+.+...+......+........+...+...............+.......+..+....+.....+....+........+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
[custom-init] No custom files found, skipping...
guacd[204]: INFO:   Guacamole proxy daemon (guacd) version 1.1.0 started
guacd[204]: INFO:   Listening on host 0.0.0.0, port 4822
guacd[204]: INFO:   Guacamole connection closed during handshake
Starting guacamole-lite websocket server
listening on *:3000
[guac-init] Auto start not set, application start on login
[ls.io-init] done.
[josevnz@dmaf5 ~]$ docker checkpoint create  webtop-test checkpoint
Error response from daemon: Cannot checkpoint container webtop-test: runc did not terminate successfully: exit status 1: criu failed: type NOTIFY errno 0 path= /run/containerd/io.containerd.runtime.v2.task/moby/8e6a55bfae2afedb5cf79ee39c4cc05f7cd8e32100323d7f593e7a6c66cab7d3/criu-dump.log: unknown

From the log file:

(00.726002) mnt:        1290: 1f:/home/josevnz/webtop/config2 @ ./config
(00.726008) mnt:        1289: 61:/ @ ./dev/shm
(00.731678) mnt:        1288: 5c:/ @ ./dev/mqueue
(00.731740) mnt:        1287: 1a:/ @ ./sys/fs/cgroup
(00.731745) mnt:        1286: 60:/ @ ./sys
(00.731750) mnt:        1285: 5f:/ @ ./dev/pts
(00.731754) mnt:        1284: 5e:/ @ ./dev
(00.731759) mnt: Mount is not fully visible ./dev(1284)
(00.731803) mnt:        mount has children ./dev(1284)
(00.737816) mnt:        1283: 5d:/ @ ./proc
(00.737839) mnt:        1282: 1f:/root/var/lib/docker/btrfs/subvolumes/200ebbf8ab6ab1dfd46167cc3e33fb1c2fddacde7d7334b32663874da3fdd25b @ ./
(00.737885) Dumping file-locks
(00.737891) Error (criu/file-lock.c:110): Some file locks are hold by dumping tasks! You can try --file-locks to dump them.
(00.737949) Unlock network
(00.737954) Running network-unlock scripts
(00.737958)     RPC
(00.754236) Unfreezing tasks into 1
(00.754252)     Unseizing 18029 into 1
(00.754261)     Unseizing 18075 into 1
(00.754269)     Unseizing 18077 into 1
(00.754278)     Unseizing 18096 into 1
(00.754286)     Unseizing 18110 into 1
(00.754294)     Unseizing 18097 into 1
(00.754300)     Unseizing 18098 into 1
(00.754306)     Unseizing 18317 into 1
(00.754313)     Unseizing 18328 into 1
(00.754320)     Unseizing 18099 into 1
(00.754327)     Unseizing 18270 into 1
(00.754336)     Unseizing 18100 into 1
(00.754343)     Unseizing 18299 into 1
(00.754374)     Unseizing 18101 into 1
(00.754382)     Unseizing 18260 into 1
(00.754388)     Unseizing 18102 into 1
(00.754394)     Unseizing 18253 into 1
(00.754441) Error (criu/cr-dump.c:2053): Dumping FAILED.

Podman error

[josevnz@dmaf5 ~]$ sudo -i
[sudo] password for josevnz: 
[root@dmaf5 ~]# podman run \
    --detach \
    --name webtop-test \
    --env PUID=1000 \
    --env PGID=1000 \
    --tz America/New_York \
    --env TITLE='Webtop test' \
    --publish 3000:3000 \
    --volume /home/josevnz/webtop/config:/config \
    lscr.io/linuxserver/webtop:ubuntu-kde
Trying to pull lscr.io/linuxserver/webtop:ubuntu-kde...
Getting image source signatures
Copying blob 2f5c2978749a done  
Copying blob 8eebfc527709 done  
Copying blob 96464c4b8240 done  
Copying blob cbba887b2540 done  
Copying blob 274402f9efdb done  
Copying blob a15ce2a609e0 done  
Copying blob 4416ec73a8df done  
Copying blob d78d143d868a done  
Copying blob b3fd396573c7 done  
Copying blob cfa496bce4e3 done  
Copying blob 92c76390af24 done  
Copying blob da6b6383c539 done  
Copying blob 964f23c9979e done  
Copying blob 8e742e7730bb done  
Copying blob e47ddd62fe0d done  
Copying blob c379c3f6fe6b done  
Copying config 481d084330 done  
Writing manifest to image destination
Storing signatures
856379f32514148b48aa73308cfd53c94c5fc4014534a33f4fcc7c5555aceb77
[root@dmaf5 ~]# podman ps
CONTAINER ID  IMAGE                                  COMMAND     CREATED        STATUS             PORTS                   NAMES
856379f32514  lscr.io/linuxserver/webtop:ubuntu-kde              9 seconds ago  Up 10 seconds ago  0.0.0.0:3000->3000/tcp  webtop-test
[root@dmaf5 ~]# podman container checkpoint webtop-test
2023-01-05T02:53:52.535876Z: CRIU checkpointing failed -52.  Please check CRIU logfile /var/lib/containers/storage/overlay-containers/856379f32514148b48aa73308cfd53c94c5fc4014534a33f4fcc7c5555aceb77/userdata/dump.log
Error: `/usr/bin/crun checkpoint --image-path /var/lib/containers/storage/overlay-containers/856379f32514148b48aa73308cfd53c94c5fc4014534a33f4fcc7c5555aceb77/userdata/checkpoint --work-path /var/lib/containers/storage/overlay-containers/856379f32514148b48aa73308cfd53c94c5fc4014534a33f4fcc7c5555aceb77/userdata 856379f32514148b48aa73308cfd53c94c5fc4014534a33f4fcc7c5555aceb77` failed: exit status 1

From the log file:

(00.021433) mnt: Inspecting sharing on 1804 shared_id 0 master_id 0 (@./)
(00.021459) Error (criu/mount.c:753): mnt: 1809:./usr/share/zoneinfo/Etc/UTC doesn't have a proper root mount
(00.021496) Unlock network
(00.021504) Running network-unlock scripts
(00.041280) Unfreezing tasks into 1
(00.041317)     Unseizing 20205 into 1
(00.041333)     Unseizing 20216 into 1
(00.041343)     Unseizing 20217 into 1
(00.041353)     Unseizing 20237 into 1
(00.041364)     Unseizing 20251 into 1
(00.041374)     Unseizing 20238 into 1
(00.041385)     Unseizing 20239 into 1
(00.041399)     Unseizing 20240 into 1
(00.041411)     Unseizing 20241 into 1
(00.041427)     Unseizing 20242 into 1
(00.041438)     Unseizing 20243 into 1
(00.041482) Error (criu/cr-dump.c:2053): Dumping FAILED.

So two different type of errors (Docker versus Podman). I can try a different container to test this use case, any recommendations?

adrianreber commented 1 year ago

The docker error can be solved by echo file-locks >> /etc/criu/runc.conf.

The Podman error can be solved by removing the timezone parameter. The timezone bind mount is not configured correctly by Podman and so CRIU fails during checkpointing. Podman should have a parameter --file-locks which would solve the error you see in Docker in case you see it in Podman.

josevnz commented 1 year ago

Hello,

I got it to work with Docker, here are the steps:

sudo mkdir -p /etc/criu/
sudo echo file-locks >> /etc/criu/runc.conf
ocker run     --detach     --name webtop-test     --env PUID=1000     --env PGID=1000     --env TZ=America/New_York     --env TITLE='Webtop test'     --publish 3000:3000     --volume /home/josevnz/webtop/config2:/config     lscr.io/linuxserver/webtop:ubuntu-kde
docker checkpoint create  webtop-test checkpoint1 # No crash
docker checkpoint ls webtop-test # Shows checkpoint1
docker start --checkpoint checkpoint1 webtop-test # Bring it back
docker logs --follow webtop-test # Shows the container running

As for podman using --file-locks, no luck. It is interesting than Docker managed to dodge the bullet:

root@dmaf5 ~]# podman container checkpoint --file-locks webtop-test 
2023-01-05T09:43:22.959145Z: CRIU checkpointing failed -52.  Please check CRIU logfile /var/lib/containers/storage/overlay-containers/bc1375ab4317d5f3a667c52d7eee6a07bc035d20b28e35f50015be3ecb8f65ae/userdata/dump.log
Error: `/usr/bin/crun checkpoint --image-path /var/lib/containers/storage/overlay-containers/bc1375ab4317d5f3a667c52d7eee6a07bc035d20b28e35f50015be3ecb8f65ae/userdata/checkpoint --work-path /var/lib/containers/storage/overlay-containers/bc1375ab4317d5f3a667c52d7eee6a07bc035d20b28e35f50015be3ecb8f65ae/userdata --file-locks bc1375ab4317d5f3a667c52d7eee6a07bc035d20b28e35f50015be3ecb8f65ae` failed: exit status 1

Log message:

00.016607) mnt: Inspecting sharing on 1936 shared_id 0 master_id 0 (@./)
(00.016625) Error (criu/mount.c:753): mnt: 1941:./usr/share/zoneinfo/Etc/UTC doesn't have a proper root mount
(00.016671) Unlock network
(00.016682) Running network-unlock scripts
(00.044621) Unfreezing tasks into 1

I see someone else pointed out this issue, initially tough Podman was the culprit then came back to Criu.

It seems than the takeaway from this problem is:

  1. Use the latest and greatest versions of Docker, Podman, Criu and Linux kernel (see examples above)
  2. In case of failure, start playing with some options on criu (file-lock on this case)
  3. May still not be enough to some containers (podman)

I'm closing this issue, hopefully this. Also thanks for spending your time looking into this, I'm more hopeful now that we can use checkpoints in our applications after getting this complex container to work.