search

checkpoint-restore / criu

Checkpoint/Restore tool
criu.org
Other
2.76k stars 559 forks source link

Criu restore failed when calling lxc-checkpoint #2381

Closed chenxy1988 closed 2 months ago

chenxy1988 commented 3 months ago

Hi criu team,

The lxc-checkpoint using criu to do the checkpoint/restore function. In my setup, lxc version is 5.0.3, criu version is 3.19. When I using this script to test this feature(see blow link), the criu reported an error that restore was failed. Could you help to check whether my system missing lib/kernel features that the failure happens? thanks!

https://github.com/lxc/lxc/blob/main/src/tests/lxc-test-checkpoint-restore

Error mssage as below:

(02.083926)      1: mnt-v2: can_mount_now_v2: true as 275 is external
(02.084212)      1: mnt-v2: Mount 275 is detected as dir-mount
(02.084324)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000275 for 275
(02.084702)      1: mnt-v2:     Mounting unsupported @275 (0)
(02.084809)      1: mnt-v2:     Bind /lib to /tmp/.criu.mntns.50SvLA/mnt-0000000275
(02.086261)      1: Error (/usr/src/debug/criu/3.19.0+git/criu/mount-v2.c:324): mnt-v2: Failed to move_mount from /lib to /tmp/.criu.mntns.50SvLA/mnt-0000000275: Invalid argument
(02.097948) Error (/usr/src/debug/criu/3.19.0+git/criu/cr-restore.c:2557): Restoring FAILED.

Steps to reproduce the issue:

  1. Build & install lxc and criu to system(my environment is Yocto)
  2. Run lxc-checkpoint script (https://github.com/lxc/lxc/blob/main/src/tests/lxc-test-checkpoint-restore)

Describe the results you received: The script do checkpoint without error, but failed to restore.

(02.083032)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000279 for 279
(02.083926)      1: mnt-v2: can_mount_now_v2: true as 275 is external
(02.084212)      1: mnt-v2: Mount 275 is detected as dir-mount
(02.084324)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000275 for 275
(02.084702)      1: mnt-v2:     Mounting unsupported @275 (0)
(02.084809)      1: mnt-v2:     Bind /lib to /tmp/.criu.mntns.50SvLA/mnt-0000000275
(02.086261)      1: Error (/usr/src/debug/criu/3.19.0+git/criu/mount-v2.c:324): mnt-v2: Failed to move_mount from /lib to /tmp/.criu.mntns.50SvLA/mnt-0000000275: Invalid argument
(02.097948) Error (/usr/src/debug/criu/3.19.0+git/criu/cr-restore.c:2557): Restoring FAILED.

Describe the results you expected: It should be normal without error.

Additional information you deem important (e.g. issue happens only occasionally):

CRIU logs and information: Full restore log attached here:

root@qemux86-64:/# cat /tmp/checkpoint/restore.log 
(00.000000) CRIU run id = 0xeffffffc0000a744
(00.001227) Version: 3.19 (gitid v3.19)
(00.002402) Running on qemux86-64 Linux 6.6.20-yocto-standard #1 SMP PREEMPT_DYNAMIC Sun Mar  3 16:28:22 UTC 2024 x86_64
(00.003867) Loaded kdat cache from /run/criu.kdat
(00.018094) Will dump/restore TCP connections
(00.018667) Will allow link remaps on FS
(00.019853) rlimit: RLIMIT_NOFILE unlimited for self
(00.031154) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel Xeon E3-12xx v2 (Ivy Bridge)
(00.031796) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 832
(00.032313) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
(00.033093) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
(00.033614) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:0 xgetbv1:0 xsaves:0
(00.036774) kernel pid_max=4194304
(00.036960) Reading image tree
(00.039966) Add mnt ns 13 pid 1
(00.040214) Add net ns 10 pid 1
(00.040623) Add pid ns 9 pid 1
(00.042045) pstree pid_max=4
(00.042964) Will restore in 6c020000 namespaces
(00.043645) NS mask to use 6c020000
(00.046295) Collecting 51/56 (flags 3)
(00.046905) No memfd.img image
(00.047220)  `- ... done
(00.047712) Collecting 40/54 (flags 2)
(00.049076) Collected [sbin/init] ID 0x1
(00.049411) Collected [dev/null] ID 0x2
(00.049765) Collected [.] ID 0x3
(00.049890) Collected [.] ID 0x4
(00.050956) unix:  `- Got id 0x5 ino 81238 type SOCK_DGRAM state TCP_CLOSE peer 0 (name /dev/log dir -)
(00.051838) Collected [dev/null] ID 0x6
(00.052110) Collected [var/log/messages] ID 0x7
(00.052228) Collected [.] ID 0x8
(00.052335) Collected [.] ID 0x9
(00.052661)  `- ... done
(00.052798) Collecting 46/68 (flags 0)
(00.053127) No remap-fpath.img image
(00.053226)  `- ... done
(00.053691) No apparmor.img image
(00.055204) cg: Preparing cgroups yard (cgroups restore mode 0x8)
(00.057087) cg: Opening .criu.cgyard.YF1n8M as cg yard
(00.057876) cg:         Making controller dir .criu.cgyard.YF1n8M/unified ()
(00.059101) cg: Determined cgroup dir unified/lxc.payload.lxc-test-criu already exist
(00.064939) Running pre-restore scripts
(00.066319) net: Saved netns fd for links restore
(00.066949) cg: cgroud: Daemon started
(00.069126) mnt: Reading mountpoint images (id 13 pid 1)
(00.070387) mnt:                Will mount 50 from /.lxc-boot-id
(00.071040) mnt:                Will mount 50 @ /tmp/.criu.mntns.50SvLA/mnt-0000000050 /proc/sys/kernel/random/boot_id
(00.071149) mnt:        Read 50 mp @ /proc/sys/kernel/random/boot_id
(00.071441) mnt:                Will mount 281 from /ptmx
(00.071862) mnt:                Will mount 281 @ /tmp/.criu.mntns.50SvLA/mnt-0000000281 /dev/ptmx
(00.071945) mnt:        Read 281 mp @ /dev/ptmx
(00.072072) mnt:                Will mount 280 from /
(00.072217) mnt:                Will mount 280 @ /tmp/.criu.mntns.50SvLA/mnt-0000000280 /dev/pts
(00.072294) mnt:        Read 280 mp @ /dev/pts
(00.072410) mnt:                Will mount 279 from /
(00.072700) mnt:                Will mount 279 @ /tmp/.criu.mntns.50SvLA/mnt-0000000279 /sys/fs/cgroup
(00.072780) mnt:        Read 279 mp @ /sys/fs/cgroup
(00.073187) mnt:                Will mount 278 from /usr/lib64 (E)
(00.073379) mnt:                Will mount 278 @ /tmp/.criu.mntns.50SvLA/mnt-0000000278 /usr/lib64
(00.073633) mnt:        Read 278 mp @ /usr/lib64
(00.073912) mnt:                Will mount 277 from /lib64 (E)
(00.074066) mnt:                Will mount 277 @ /tmp/.criu.mntns.50SvLA/mnt-0000000277 /lib64
(00.074145) mnt:        Read 277 mp @ /lib64
(00.074281) mnt:                Will mount 276 from /usr/lib (E)
(00.074386) mnt:                Will mount 276 @ /tmp/.criu.mntns.50SvLA/mnt-0000000276 /usr/lib
(00.074602) mnt:        Read 276 mp @ /usr/lib
(00.074758) mnt:                Will mount 275 from /lib (E)
(00.074891) mnt:                Will mount 275 @ /tmp/.criu.mntns.50SvLA/mnt-0000000275 /lib
(00.074968) mnt:        Read 275 mp @ /lib
(00.075085) mnt:                Will mount 274 from /
(00.075215) mnt:                Will mount 274 @ /tmp/.criu.mntns.50SvLA/mnt-0000000274 /dev/mqueue
(00.075291) mnt:        Read 274 mp @ /dev/mqueue
(00.075647) mnt:                Will mount 273 from /
(00.075777) mnt:                Will mount 273 @ /tmp/.criu.mntns.50SvLA/mnt-0000000273 /dev/shm
(00.075851) mnt:        Read 273 mp @ /dev/shm
(00.076036) mnt:                Will mount 272 from /
(00.076157) mnt:                Will mount 272 @ /tmp/.criu.mntns.50SvLA/mnt-0000000272 /sys
(00.076231) mnt:        Read 272 mp @ /sys
(00.076346) mnt:                Will mount 271 from /sysrq-trigger
(00.076624) mnt:                Will mount 271 @ /tmp/.criu.mntns.50SvLA/mnt-0000000271 /proc/sysrq-trigger
(00.076715) mnt:        Read 271 mp @ /proc/sysrq-trigger
(00.076837) mnt:                Will mount 270 from /sys
(00.076947) mnt:                Will mount 270 @ /tmp/.criu.mntns.50SvLA/mnt-0000000270 /proc/sys
(00.077019) mnt:        Read 270 mp @ /proc/sys
(00.077133) mnt:                Will mount 269 from /sys/net
(00.077377) mnt:                Will mount 269 @ /tmp/.criu.mntns.50SvLA/mnt-0000000269 /proc/sys/net
(00.077657) mnt:        Read 269 mp @ /proc/sys/net
(00.077784) mnt:                Will mount 268 from /
(00.077896) mnt:                Will mount 268 @ /tmp/.criu.mntns.50SvLA/mnt-0000000268 /proc
(00.077968) mnt:        Read 268 mp @ /proc
(00.078083) mnt:                Will mount 267 from /
(00.078191) mnt:                Will mount 267 @ /tmp/.criu.mntns.50SvLA/mnt-0000000267 /dev
(00.078262) mnt:        Read 267 mp @ /dev
(00.078376) mnt:                Will mount 266 from /var/lib/lxc/lxc-test-criu/rootfs
(00.078637) mnt:                Will mount 266 @ /tmp/.criu.mntns.50SvLA/mnt-0000000266 /
(00.078725) mnt:        Read 266 mp @ /
(00.079036) mnt: Building mountpoints tree
(00.079143) mnt:        Building plain mount tree
(00.079254) mnt:                Working on 266->47
(00.079412) mnt:                Working on 267->266
(00.079800) mnt:                Working on 268->266
(00.079895) mnt:                Working on 269->270
(00.079978) mnt:                Working on 270->268
(00.080058) mnt:                Working on 271->268
(00.080136) mnt:                Working on 272->266
(00.080225) mnt:                Working on 273->267
(00.080301) mnt:                Working on 274->267
(00.080376) mnt:                Working on 275->266
(00.080595) mnt:                Working on 276->266
(00.080680) mnt:                Working on 277->266
(00.080758) mnt:                Working on 278->266
(00.080834) mnt:                Working on 279->272
(00.080910) mnt:                Working on 280->267
(00.080985) mnt:                Working on 281->267
(00.081061) mnt:                Working on 50->270
(00.081188) mnt:        Resorting children of 266 in mount order
(00.081695) mnt:        Resorting children of 276 in mount order
(00.081841) mnt:        Resorting children of 278 in mount order
(00.081930) mnt:        Resorting children of 267 in mount order
(00.082012) mnt:        Resorting children of 273 in mount order
(00.082090) mnt:        Resorting children of 274 in mount order
(00.082166) mnt:        Resorting children of 280 in mount order
(00.082242) mnt:        Resorting children of 281 in mount order
(00.082318) mnt:        Resorting children of 268 in mount order
(00.082394) mnt:        Resorting children of 270 in mount order
(00.082608) mnt:        Resorting children of 50 in mount order
(00.082693) mnt:        Resorting children of 269 in mount order
(00.082771) mnt:        Resorting children of 271 in mount order
(00.082847) mnt:        Resorting children of 272 in mount order
(00.082923) mnt:        Resorting children of 279 in mount order
(00.082999) mnt:        Resorting children of 275 in mount order
(00.083088) mnt:        Resorting children of 277 in mount order
(00.083189) mnt: Done:
(00.083333) mnt: [/](266->47)
(00.083766) mnt:  [/usr/lib](276->266)
(00.083916) mnt:  <--
(00.084022) mnt:  [/usr/lib64](278->266)
(00.084112) mnt:  <--
(00.084204) mnt:  [/dev](267->266)
(00.084284) mnt:   [/dev/shm](273->267)
(00.084364) mnt:   <--
(00.084599) mnt:   [/dev/mqueue](274->267)
(00.084700) mnt:   <--
(00.084780) mnt:   [/dev/pts](280->267)
(00.084861) mnt:   <--
(00.084938) mnt:   [/dev/ptmx](281->267)
(00.085017) mnt:   <--
(00.085093) mnt:  <--
(00.085167) mnt:  [/proc](268->266)
(00.085247) mnt:   [/proc/sys](270->268)
(00.085326) mnt:    [/proc/sys/kernel/random/boot_id](50->270)
(00.085405) mnt:    <--
(00.085614) mnt:    [/proc/sys/net](269->270)
(00.085703) mnt:    <--
(00.085779) mnt:   <--
(00.085854) mnt:   [/proc/sysrq-trigger](271->268)
(00.085933) mnt:   <--
(00.086008) mnt:  <--
(00.086082) mnt:  [/sys](272->266)
(00.086161) mnt:   [/sys/fs/cgroup](279->272)
(00.086239) mnt:   <--
(00.086314) mnt:  <--
(00.086387) mnt:  [/lib](275->266)
(00.086592) mnt:  <--
(00.086677) mnt:  [/lib64](277->266)
(00.086758) mnt:  <--
(00.086833) mnt: <--
(00.087155) mnt:        The mount 275 is bind for 266 (@/lib -> @/)
(00.087334) mnt:        The mount 276 is bind for 266 (@/usr/lib -> @/)
(00.087606) mnt:        The mount 277 is bind for 266 (@/lib64 -> @/)
(00.087713) mnt:        The mount 278 is bind for 266 (@/usr/lib64 -> @/)
(00.087947) mnt:        The mount  50 is bind for 267 (@/proc/sys/kernel/random/boot_id -> @/dev)
(00.088066) mnt:        The mount 269 is bind for 268 (@/proc/sys/net -> @/proc)
(00.088159) mnt:        The mount 270 is bind for 268 (@/proc/sys -> @/proc)
(00.088245) mnt:        The mount 271 is bind for 268 (@/proc/sysrq-trigger -> @/proc)
(00.088385) mnt:        The mount 281 is bind for 280 (@/dev/ptmx -> @/dev/pts)
(00.088740) mnt: Start with 266:/
(00.089304) mnt-v2: Inspecting sharing on 266 shared_id 216 master_id 1 (@/)
(00.089656) mnt-v2: Inspecting sharing on 267 shared_id 217 master_id 0 (@/dev)
(00.089795) mnt-v2: Inspecting sharing on 268 shared_id 222 master_id 0 (@/proc)
(00.089893) mnt-v2: Inspecting sharing on 269 shared_id 224 master_id 0 (@/proc/sys/net)
(00.089982) mnt-v2: Inspecting sharing on 270 shared_id 223 master_id 0 (@/proc/sys)
(00.090069) mnt-v2: Inspecting sharing on 271 shared_id 225 master_id 0 (@/proc/sysrq-trigger)
(00.090156) mnt-v2: Inspecting sharing on 272 shared_id 226 master_id 0 (@/sys)
(00.090242) mnt-v2: Inspecting sharing on 273 shared_id 218 master_id 0 (@/dev/shm)
(00.090327) mnt-v2: Inspecting sharing on 274 shared_id 219 master_id 0 (@/dev/mqueue)
(00.090412) mnt-v2: Inspecting sharing on 275 shared_id 228 master_id 1 (@/lib)
(00.090668) mnt-v2: Inspecting sharing on 276 shared_id 229 master_id 1 (@/usr/lib)
(00.090758) mnt-v2: Inspecting sharing on 277 shared_id 230 master_id 1 (@/lib64)
(00.090844) mnt-v2: Inspecting sharing on 278 shared_id 231 master_id 1 (@/usr/lib64)
(00.091039) mnt-v2: Inspecting sharing on 279 shared_id 227 master_id 0 (@/sys/fs/cgroup)
(00.091136) mnt-v2: Inspecting sharing on 280 shared_id 220 master_id 0 (@/dev/pts)
(00.091223) mnt-v2: Inspecting sharing on 281 shared_id 221 master_id 0 (@/dev/ptmx)
(00.091309) mnt-v2: Inspecting sharing on 50 shared_id 217 master_id 0 (@/proc/sys/kernel/random/boot_id)
(00.092105) mnt-v2: Detected external slavery for shared group (231, 1) with source /usr/lib64
(00.092367) mnt: Mountpoint 266 (@/) moved to the root yard
(00.094749) No pidns-9.img image
(00.098328) Warn  (/usr/src/debug/criu/3.19.0+git/criu/cr-restore.c:1300): Set CLONE_PARENT | CLONE_NEWPID but it might cause restore problem,because not all kernels support such clone flags combinations!
(00.098864) Forking task with 1 pid (flags 0x6c028000)
(00.099088) Creating process using clone3()
(00.102156) PID: real 42823 virt 1
(00.104717) Wait until namespaces are created
(00.134765)      1: timens: monotonic -820 73172585
(00.135874)      1: timens: boottime -820 72130811
(00.138938) Running setup-namespaces scripts
(00.140839)      1: cg: setting cgns prefix to /lxc.payload.lxc-test-criu
(00.144010)      1: cg: Move into 2
(00.144298)      1: cg:   `-> unified//lxc.payload.lxc-test-criu/cgroup.procs
(00.145409)      1: Calling restore_sid() for init
(00.145785)      1: Restoring 1 to 1 sid
(00.147937)      1: Collecting 44/37 (flags 2)
(00.148326)      1: No tty-info.img image
(00.148671)      1:  `- ... done
(00.148873)      1: Collecting 45/51 (flags 0)
(00.149202)      1: No tty-data.img image
(00.149333)      1:  `- ... done
(00.149694)      1: Restoring namespaces 1 flags 0x6c028000
(00.163347)      1: kernel/hostname nr 13
(00.166910)      1: kernel/domainname nr 6
(00.182842)      1: Restoring IPC namespace
(00.183195)      1: Restoring IPC variables
(00.197053)      1: Restoring IPC shared memory
(00.197426)      1: No ipcns-shm-11.img image
(00.197894)      1: Restoring IPC message queues
(00.198258)      1: No ipcns-msg-11.img image
(00.198640)      1: Restoring IPC semaphores sets
(00.199018)      1: No ipcns-sem-11.img image
(00.276797)      1: net: Try to restore a link 10:3:eth0
(00.276961)      1: net: Restoring link eth0 type 2
(00.277242)      1: net: Restoring netdev eth0 idx 3
(00.277802)      1: net: Restore ll addr (00:../6) for device
(00.294876)      1: net: Skip eth0/accept_local, coincides with default
(00.295022)      1: net: Skip eth0/accept_redirects, coincides with default
(00.295121)      1: net: Skip eth0/accept_source_route, coincides with default
(00.295275)      1: net: Skip eth0/arp_accept, coincides with default
(00.295376)      1: net: Skip eth0/arp_announce, coincides with default
(00.295669)      1: net: Skip eth0/arp_filter, coincides with default
(00.295782)      1: net: Skip eth0/arp_ignore, coincides with default
(00.295869)      1: net: Skip eth0/arp_notify, coincides with default
(00.295955)      1: net: Skip eth0/bootp_relay, coincides with default
(00.296040)      1: net: Skip eth0/disable_policy, coincides with default
(00.296123)      1: net: Skip eth0/disable_xfrm, coincides with default
(00.296206)      1: net: Skip eth0/force_igmp_version, coincides with default
(00.296289)      1: net: Skip eth0/forwarding, coincides with default
(00.296372)      1: net: Skip eth0/igmpv2_unsolicited_report_interval, coincides with default
(00.296630)      1: net: Skip eth0/igmpv3_unsolicited_report_interval, coincides with default
(00.296737)      1: net: Skip eth0/log_martians, coincides with default
(00.296825)      1: net: Skip eth0/medium_id, coincides with default
(00.296909)      1: net: Skip eth0/promote_secondaries, coincides with default
(00.296992)      1: net: Skip eth0/proxy_arp, coincides with default
(00.297075)      1: net: Skip eth0/proxy_arp_pvlan, coincides with default
(00.297158)      1: net: Skip eth0/route_localnet, coincides with default
(00.297242)      1: net: Skip eth0/rp_filter, coincides with default
(00.297325)      1: net: Skip eth0/secure_redirects, coincides with default
(00.297409)      1: net: Skip eth0/send_redirects, coincides with default
(00.297686)      1: net: Skip eth0/shared_media, coincides with default
(00.297780)      1: net: Skip eth0/src_valid_mark, coincides with default
(00.297864)      1: net: Skip eth0/tag, coincides with default
(00.297948)      1: net: Skip eth0/ignore_routes_with_linkdown, coincides with default
(00.298033)      1: net: Skip eth0/drop_gratuitous_arp, coincides with default
(00.298116)      1: net: Skip eth0/drop_unicast_in_l2_multicast, coincides with default
(00.298276)      1: net: Skip eth0/accept_dad, coincides with default
(00.298374)      1: net: Skip eth0/accept_ra, coincides with default
(00.298618)      1: net: Skip eth0/accept_ra_defrtr, coincides with default
(00.298723)      1: net: Skip eth0/accept_ra_from_local, coincides with default
(00.298812)      1: net: Skip eth0/accept_ra_min_hop_limit, coincides with default
(00.298897)      1: net: Skip eth0/accept_ra_mtu, coincides with default
(00.298981)      1: net: Skip eth0/accept_ra_pinfo, coincides with default
(00.299101)      1: net: Skip eth0/accept_ra_rtr_pref, coincides with default
(00.299186)      1: net: Skip eth0/accept_redirects, coincides with default
(00.299271)      1: net: Skip eth0/accept_source_route, coincides with default
(00.299355)      1: net: Skip eth0/autoconf, coincides with default
(00.299692)      1: net: Skip eth0/dad_transmits, coincides with default
(00.299815)      1: net: Skip eth0/disable_ipv6, coincides with default
(00.299905)      1: net: Skip eth0/drop_unicast_in_l2_multicast, coincides with default
(00.299989)      1: net: Skip eth0/drop_unsolicited_na, coincides with default
(00.300074)      1: net: Skip eth0/force_mld_version, coincides with default
(00.300158)      1: net: Skip eth0/force_tllao, coincides with default
(00.300242)      1: net: Skip eth0/forwarding, coincides with default
(00.300325)      1: net: Skip eth0/hop_limit, coincides with default
(00.300407)      1: net: Skip eth0/ignore_routes_with_linkdown, coincides with default
(00.300780)      1: net: Skip eth0/keep_addr_on_down, coincides with default
(00.300875)      1: net: Skip eth0/max_addresses, coincides with default
(00.300960)      1: net: Skip eth0/max_desync_factor, coincides with default
(00.301044)      1: net: Skip eth0/mldv1_unsolicited_report_interval, coincides with default
(00.301238)      1: net: Skip eth0/mldv2_unsolicited_report_interval, coincides with default
(00.301356)      1: net: Skip eth0/ndisc_notify, coincides with default
(00.301678)      1: net: Skip eth0/proxy_ndp, coincides with default
(00.301785)      1: net: Skip eth0/regen_max_retry, coincides with default
(00.301875)      1: net: Skip eth0/router_probe_interval, coincides with default
(00.301967)      1: net: Skip eth0/router_solicitation_delay, coincides with default
(00.302053)      1: net: Skip eth0/router_solicitation_interval, coincides with default
(00.302145)      1: net: Skip eth0/router_solicitations, coincides with default
(00.302268)      1: net: Skip eth0/suppress_frag_ndisc, coincides with default
(00.302360)      1: net: Skip eth0/temp_prefered_lft, coincides with default
(00.302658)      1: net: Skip eth0/temp_valid_lft, coincides with default
(00.302773)      1: net: Skip eth0/use_oif_addrs_only, coincides with default
(00.302875)      1: net: Skip eth0/use_tempaddr, coincides with default
(00.303725)      1: net: Try to restore a link 10:1:lo
(00.303857)      1: net: Restoring link lo type 1
(00.346806)      1: net:        Running ip addr restore  
Error: ipv4: Address already assigned.
Error: ipv6: address already assigned.
(00.483813)      1: net:        Running ip route restore  
(00.601009)      1: net:        Running ip route restore  
RTNETLINK answers: File exists
RTNETLINK answers: File exists
(00.747313)      1: net:        Running ip rule flush  
(00.859379)      1: net:        Running ip rule delete table local
(00.985969)      1: net:        Running ip rule restore  
(01.108047)      1: net:        Running iptables-restore -w for iptables-restore -w
(01.320884)      1: net:        Running ip6tables-restore -w for ip6tables-restore -w
(01.568354)      1: No nftables-10.img image
(01.569011)      1: net: Skipping nft restore, no image
(01.569842)      1: No netns-ct-10.img image
(01.570738)      1: No netns-exp-10.img image
(01.577217)      1: mnt: Restoring mount namespace
(01.582020)      1: mnt: Move the root to /tmp/.criu.mntns.50SvLA
(01.591698)      1: mnt: Start with 0:(null)
(01.592184)      1: mnt-v2: can_mount_now_v2: true as 266 is global root
(01.592330)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000266 for 266
(01.592862)      1: mnt-v2:     Mounting unsupported @266 (0)
(01.595647)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000266 for 266
(01.596820)      1: mnt-v2: can_mount_now_v2: true as 276 is external
(01.597334)      1: mnt-v2: Mount 276 is detected as dir-mount
(01.597736)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000276 for 276
(01.598019)      1: mnt-v2:     Mounting unsupported @276 (0)
(01.598277)      1: mnt-v2:     Bind /usr/lib to /tmp/.criu.mntns.50SvLA/mnt-0000000276
(01.599606)      1: mnt-v2:             Propagate 276 to 275
(01.600619)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000276 for 276
(01.601398)      1: mnt-v2: can_mount_now_v2: true as 278 is external
(01.601759)      1: mnt-v2: Mount 278 is detected as dir-mount
(01.601855)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000278 for 278
(01.602121)      1: mnt-v2:     Mounting unsupported @278 (0)
(01.602219)      1: mnt-v2:     Bind /usr/lib64 to /tmp/.criu.mntns.50SvLA/mnt-0000000278
(01.603094)      1: mnt-v2:             Propagate 278 to 277
(01.604111)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000278 for 278
(01.605120)      1: mnt-v2: Mount 267 is detected as dir-mount
(01.605223)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000267 for 267
(01.605630)      1: mnt-v2:     Mounting tmpfs @267 (0)
(01.830019)      1: mnt-v2:             Propagate 267 to 50
(01.832351)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000267 for 267
(01.835076)      1: mnt-v2: Mount 273 is detected as dir-mount
(01.835291)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000273 for 273
(01.836226)      1: mnt-v2:     Mounting tmpfs @273 (0)
(02.044161)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000273 for 273
(02.046930)      1: mnt-v2: Mount 274 is detected as dir-mount
(02.047151)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000274 for 274
(02.047887)      1: mnt-v2:     Mounting mqueue @274 (0)
(02.051110)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000274 for 274
(02.052433)      1: mnt-v2: Mount 280 is detected as dir-mount
(02.052704)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000280 for 280
(02.052981)      1: mnt-v2:     Mounting devpts @280 (0)
(02.054364)      1: mnt-v2:             Propagate 280 to 281
(02.055685)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000280 for 280
(02.056661)      1: mnt-v2: Mount 281 is detected as file-mount
(02.056775)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000281 for 281
(02.057424)      1: mnt-v2:     Mounting devpts @281 (0)
(02.057910)      1: mnt-v2:     Bind /tmp/.criu.mntns.50SvLA/mnt-0000000280/ptmx to /tmp/.criu.mntns.50SvLA/mnt-0000000281
(02.059761)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000281 for 281
(02.060917)      1: mnt-v2: Mount 268 is detected as dir-mount
(02.061020)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000268 for 268
(02.061271)      1: mnt-v2:     Mounting proc @268 (0)
(02.062109)      1: mnt-v2:             Propagate 268 to 271
(02.062298)      1: mnt-v2:             Propagate 268 to 270
(02.062386)      1: mnt-v2:             Propagate 268 to 269
(02.063355)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000268 for 268
(02.064625)      1: mnt-v2: Mount 270 is detected as dir-mount
(02.064738)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000270 for 270
(02.065017)      1: mnt-v2:     Mounting proc @270 (0)
(02.065125)      1: mnt-v2:     Bind /tmp/.criu.mntns.50SvLA/mnt-0000000268/sys to /tmp/.criu.mntns.50SvLA/mnt-0000000270
(02.066775)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000270 for 270
(02.068065)      1: mnt-v2: Mount 50 is detected as file-mount
(02.068170)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000050 for 50
(02.068639)      1: mnt-v2:     Mounting tmpfs @50 (0)
(02.068791)      1: mnt-v2:     Bind /tmp/.criu.mntns.50SvLA/mnt-0000000267/.lxc-boot-id to /tmp/.criu.mntns.50SvLA/mnt-0000000050
(02.070278)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000050 for 50
(02.071606)      1: mnt-v2: Mount 269 is detected as dir-mount
(02.071713)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000269 for 269
(02.071963)      1: mnt-v2:     Mounting proc @269 (0)
(02.072070)      1: mnt-v2:     Bind /tmp/.criu.mntns.50SvLA/mnt-0000000268/sys/net to /tmp/.criu.mntns.50SvLA/mnt-0000000269
(02.073611)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000269 for 269
(02.074614)      1: mnt-v2: Mount 271 is detected as file-mount
(02.074727)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000271 for 271
(02.075089)      1: mnt-v2:     Mounting proc @271 (0)
(02.075213)      1: mnt-v2:     Bind /tmp/.criu.mntns.50SvLA/mnt-0000000268/sysrq-trigger to /tmp/.criu.mntns.50SvLA/mnt-0000000271
(02.076899)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000271 for 271
(02.077873)      1: mnt-v2: Mount 272 is detected as dir-mount
(02.077976)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000272 for 272
(02.078218)      1: mnt-v2:     Mounting sysfs @272 (0)
(02.079998)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000272 for 272
(02.081076)      1: mnt-v2: Mount 279 is detected as dir-mount
(02.081181)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000279 for 279
(02.081617)      1: mnt-v2:     Mounting cgroup2 @279 (0)
(02.083032)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000279 for 279
(02.083926)      1: mnt-v2: can_mount_now_v2: true as 275 is external
(02.084212)      1: mnt-v2: Mount 275 is detected as dir-mount
(02.084324)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.50SvLA/mnt-0000000275 for 275
(02.084702)      1: mnt-v2:     Mounting unsupported @275 (0)
(02.084809)      1: mnt-v2:     Bind /lib to /tmp/.criu.mntns.50SvLA/mnt-0000000275
(02.086261)      1: Error (/usr/src/debug/criu/3.19.0+git/criu/mount-v2.c:324): mnt-v2: Failed to move_mount from /lib to /tmp/.criu.mntns.50SvLA/mnt-0000000275: Invalid argument
(02.097948) Error (/usr/src/debug/criu/3.19.0+git/criu/cr-restore.c:2557): Restoring FAILED.
(02.104948) Error (/usr/src/debug/criu/3.19.0+git/criu/cgroup.c:1970): cg: cgroupd: recv req error: No such file or directory
adrianreber commented 3 months ago

This could be related to the new mount support which was added in one of the later versions of CRIU and lxc was never adapted to handle it correctly. This is just a guess, but you could try to activate the old mount code during restore.

chenxy1988 commented 2 months ago

@adrianreber , Thanks for your info, I'll try to downgrade the criu version to compare whether it was a lxc issue. Close this due to I cannot confirm when I can provide the test result.