search

checkpoint-restore / criu

Checkpoint/Restore tool
criu.org
Other
2.93k stars 587 forks source link

Building CRIU for ARM Android #1195

Closed mwerensteijn closed 4 years ago

mwerensteijn commented 4 years ago

Hi all,

For a student project I am trying to build CRIU for ARM Android. I am using the latest Android NDK and I can succesfully build protobuf, protobuf-c, libnet and libnl.

I am trying to build the latest CRIU release 3.14 with commands: ARCH=arm \ CROSS_COMPILE=arm-linux-androideabi- \ CFLAGS="-Wno-macro-redefinedpkg-config --cflags libprotobuf-c-D CONFIG_HAS_TCP_REPAIR_WINDOW -D CONFIG_HAS_TCP_REPAIR " \ LDFLAGS="pkg-config --libs libprotobuf-c`pkg-config --libs libnl-3.0 pkg-config --libs libnet" \ make -j$(nproc) criu`

Building succeeds until: criu/Makefile.packages:36: Can not find some of the required libraries criu/Makefile.packages:37: Make sure the following packages are installed criu/Makefile.packages:38: RPM based distros: protobuf protobuf-c protobuf-c-devel protobuf-compiler protobuf-devel protobuf-python libnl3-devel libcap-devel python2-future criu/Makefile.packages:39: DEB based distros: libprotobuf-dev libprotobuf-c-dev protobuf-c-compiler protobuf-compiler python-protobuf libnl-3-dev libcap-dev criu/Makefile.packages:40: To run tests the following packages are needed criu/Makefile.packages:41: RPM based distros: libaio-devel python2-pyyaml criu/Makefile.packages:42: DEB based distros: python-yaml libaio-dev criu/Makefile.packages:43: *** Compilation aborted. Stop. make[1]: *** [criu/Makefile.packages:49: check-packages] Error 2 make: *** [Makefile:241: criu] Error 2.

For readability I have attached the build log: build.log

I already have installed all debian packages except for python-future which is not available. However Python does support import future, so I don't think that is the program.

I suspect the issue to be with any of the following libraries as it tries to compile the PROGRAM_STUB in Makefile.packages with libraries '-lprotobuf-c -ldl -lnl-3 -lsoccr -Lsoccr/ -lnet', probably to check whether they are all available.

Am I missing -ldl? Or is there something else I am missing? Does anybody have experience with building CRIU for Android ARM?

Any help is welcome,

Kind regards, Mitchell Werensteijn

adrianreber commented 4 years ago

The strange thing is that I do not actually see the build error any where. If you do make V=1 it should be possible to see all the commands. Also do not use -j to make sure the last message is the actual error. With multiple parallel make runs it is sometimes a bit harder to find the correct error.

There is also this ticket were somebody tried to build CRIU recently for android: https://github.com/checkpoint-restore/criu/issues/1172

mwerensteijn commented 4 years ago

Hi Adrian,

Thank you for your response. The difference to that ticket is that I am trying to build for ARM and not x86.

I tried to see how far the compilation process would go by commenting out the: $(error Compilation aborted) in Makefile.packages:check-packages-failed

I solved some other unresolved defines and I can now compile a lot further. The full build log: build_log.txt

I do have some new undefined references now: include/common/scm-code.c:54: error: undefined reference to '__aeabi_memclr8' include/common/scm-code.c:62: error: undefined reference to '__aeabi_memcpy4' include/common/scm-code.c:78: error: undefined reference to '__aeabi_memclr8' include/common/scm-code.c:114: error: undefined reference to '__aeabi_memcpy4'
compel/plugins/std/log.c:39: error: undefined reference to '__aeabi_memset' compel/plugins/std/log.c:65: error: undefined reference to '__aeabi_memcpy' compel/plugins/std/log.c:39: error: undefined reference to '__aeabi_memset' compel/plugins/std/log.c:72: error: undefined reference to '__aeabi_memcpy' compel/plugins/std/log.c:84: error: undefined reference to '__aeabi_memcpy' arm-linux-androideabi-ld: internal error in do_relocate_sections, at /buildbot/src/android/binutils/toolchain/binutils/binutils-2.27/gold/arm.cc:6687

I have tried to change the linker flags to load libc, but this does not make a difference: ARCH=arm CROSS_COMPILE=arm-linux-androideabi- \ CFLAGS="-Wno-macro-redefined `pkg-config --cflags libprotobuf-c` -D ANDROID_PLATFORM=android-27 -D CONFIG_HAS_TCP_REPAIR_WINDOW -D CONFIG_HAS_TCP_REPAIR" \ LDFLAGS="`pkg-config --libs libprotobuf-c` `pkg-config --libs libnl-3.0` `pkg-config --libs libnet` -static $SYSROOT/usr/lib/arm-linux-androideabi/27/libc.a " \ make criu

Do you have any idea how I can solve the undefined references to the _aeabi functions?

Thank you for your help,

Kind regards, Mitchell

adrianreber commented 4 years ago

Do you have any idea how I can solve the undefined references to the _aeabi functions?

No, but googling for it, I saw it also on other projects trying to compile for Android. So it seems to be a 'normal' Android problem which should be possible to solve. As far as I know, there is no Android expert in the CRIU community. Not sure we can help you with this.

0x7f454c46 commented 4 years ago

Do you have any idea how I can solve the undefined references to the _aeabi functions?

Yes, introduce compel/arch/arm/plugins/std/memcpy.S (see x86 as an example). You can probably reuse code from kernel, see arch/arm/lib/memcpy.S or glibc sysdeps/arm/armv7/multiarch/memcpy_impl.S see aliases for __memcpy_arm.

I have tried to change the linker flags to load libc, but this does not make a difference

Yes, that won't help: the issue is in compiler-generated references to memcpy() in parasite, which can't dynamically link to libc.

mwerensteijn commented 4 years ago

Thank you both for your comments. For now I have created empty functions so the compiler errors are gone so I can see how many other compilation issues there are. Your solution sounds great, thanks for the pointers.

mwerensteijn commented 4 years ago

I implemented the functions and Criu is now compiling succesfully. I am not able to checkpoint and restore a simple loop binary as there are some errors:

./criu check --all                                                                                                                    
Warn  (criu/net.c:3137): Unable to get tun network namespace
Error (criu/libnetlink.c:55): -95 reported by netlink: Operation not supported on transport endpoint
Error (criu/net.c:3453): Unable to create a veth pair: -95
Warn  (criu/net.c:3475): NSID isn't reported for network links
Error (criu/uffd.c:272): uffd: Lazy pages are not available: Function not implemented
Error (criu/cr-check.c:512): /proc/sys/kernel/sem_next_id is inaccessible: No such file or directory
Does not look good.

Adding more verbosity shows:

./criu check --all -v4                                                                                                                  
(00.000015) Version: 3.14 (gitid v3.14-13-gc4886d3e)
(00.000059) Running on localhost Linux 4.14.177-g928602c85866-dirty #1 SMP PREEMPT Thu Sep 3 14:06:25 UTC 2020 armv8l
(00.000088) File /run/criu.kdat does not exist
(00.000203) sockets: Probing sock diag modules
(00.001040) sockets: Done probing
(00.037754) Pagemap is fully functional
(00.038155) Found anon-shmem device at 1
(00.038295) Reset 23576's dirty tracking
(00.038499)  ... done
(00.038695) Dirty tracking support is OFF
(00.039175) Found task size of bf000000
(00.072280) Warn  (criu/net.c:3137): Unable to get tun network namespace
(00.319802) Restoring netdev veth idx 10
(00.320572) Error (criu/libnetlink.c:55): -95 reported by netlink: Operation not supported on transport endpoint
(00.320635) Error (criu/net.c:3453): Unable to create a veth pair: -95
(00.321440) Warn  (criu/net.c:3475): NSID isn't reported for network links
(00.321967) Error (criu/uffd.c:272): uffd: Lazy pages are not available: Function not implemented
(00.323304) vdso: Kernel doesn't premap vDSO - probably CONFIG_VDSO is not set
(00.323730) The new mount API (fsopen, fsmount) isn't supported
(00.323838) Time namespaces are not supported.
(00.324020) Adjust mmap_min_addr 0x8000 -> 0x10000
(00.324084) Found mmap_min_addr 0x10000
(00.324177) files stat: fs/nr_open 1048576
(00.324432) Add pid ns 1 pid 23576
(00.324542) Add net ns 2 pid 23576
(00.324650) Add ipc ns 3 pid 23576
(00.324760) Add uts ns 4 pid 23576
(00.324846) Add time ns 5 pid 23576
(00.324958) Add mnt ns 6 pid 23576
(00.325066) Add user ns 7 pid 23576
(00.325177) Add cgroup ns 8 pid 23576
(00.325646)     type rootfs source rootfs mnt_id 1 s_dev 0x2 / @ ./ flags 0x1100001 options seclabel,size=932800k,nr_inodes=233200
(00.325820)     type tmpfs source tmpfs mnt_id 17 s_dev 0x12 / @ ./dev flags 0x300002 options seclabel,mode=755
(00.325996)     type devpts source devpts mnt_id 18 s_dev 0x13 / @ ./dev/pts flags 0x300000 options seclabel,mode=600,ptmxmode=000
(00.326305)     type proc source proc mnt_id 19 s_dev 0x5 / @ ./proc flags 0x300000 options gid=3009,hidepid=2
(00.326458)     type sysfs source sysfs mnt_id 20 s_dev 0x14 / @ ./sys flags 0x300000 options seclabel
(00.326610)     type selinuxfs source selinuxfs mnt_id 21 s_dev 0x10 / @ ./sys/fs/selinux flags 0x300000 options 
(00.326813)     type tmpfs source tmpfs mnt_id 22 s_dev 0x15 / @ ./mnt flags 0x30000e options seclabel,mode=755,gid=1000
(00.326971)     type ext4 source /dev/block/platform/soc/f723d000.dwmmc0/by-name/system mnt_id 23 s_dev 0xb300009 / @ ./system flags 0x300001 options seclabel
(00.327131)     type ext4 source /dev/block/platform/soc/f723d000.dwmmc0/by-name/vendor mnt_id 24 s_dev 0xb300007 / @ ./vendor flags 0x300001 options seclabel
(00.327292)     type cgroup source none mnt_id 25 s_dev 0x16 / @ ./acct flags 0x30000e options cpuacct
(00.327455)     type cgroup source none mnt_id 26 s_dev 0x17 / @ ./dev/memcg flags 0x30000e options memory
(00.327776)     type configfs source none mnt_id 27 s_dev 0xe / @ ./config flags 0x30000e options 
(00.327939)     type cgroup source none mnt_id 28 s_dev 0x18 / @ ./dev/cpuctl flags 0x30000e options cpu
(00.328208)     type cgroup source none mnt_id 29 s_dev 0x19 / @ ./dev/cpuset flags 0x30000e options cpuset,noprefix,release_agent=/sbin/cpuset_release_agent
(00.328519)     type cgroup2 source cg2_bpf mnt_id 30 s_dev 0x1a / @ ./dev/cg2_bpf flags 0x30000e options 
(00.328647)     type bpf source bpf mnt_id 31 s_dev 0x1b / @ ./sys/fs/bpf flags 0x30000e options 
(00.328774)     type pstore source pstore mnt_id 32 s_dev 0x1c / @ ./sys/fs/pstore flags 0x30000e options seclabel
(00.328931)     type ext4 source /dev/block/platform/soc/f723d000.dwmmc0/by-name/cache mnt_id 33 s_dev 0xb300008 / @ ./cache flags 0x100400 options seclabel,discard,noauto_da_alloc,data=ordered
(00.329069)     type ext4 source /dev/block/platform/soc/f723d000.dwmmc0/by-name/userdata mnt_id 34 s_dev 0xb30000a / @ ./data flags 0x100406 options seclabel,discard,noauto_da_alloc,data=ordered
(00.329205)     type tmpfs source tmpfs mnt_id 35 s_dev 0x15 /runtime/default @ ./storage flags 0x28000e options seclabel,mode=755,gid=1000
(00.329360)     type functionfs source adb mnt_id 36 s_dev 0x1d / @ ./dev/usb-ffs/adb flags 0x300000 options 
(00.329498)     type functionfs source mtp mnt_id 37 s_dev 0x1e / @ ./dev/usb-ffs/mtp flags 0x300000 options 
(00.329633)     type functionfs source ptp mnt_id 38 s_dev 0x1f / @ ./dev/usb-ffs/ptp flags 0x300000 options 
(00.329757) mnt: Building mountpoints tree
(00.329811) mnt:    Building plain mount tree
(00.329863) mnt:        Working on 38->17
(00.329925) mnt:        Working on 37->17
(00.329949) mnt:        Working on 36->17
(00.329969) mnt:        Working on 35->1
(00.330130) mnt:        Working on 34->1
(00.330175) mnt:        Working on 33->1
(00.330197) mnt:        Working on 32->20
(00.330217) mnt:        Working on 31->20
(00.330236) mnt:        Working on 30->17
(00.330256) mnt:        Working on 29->17
(00.330275) mnt:        Working on 28->17
(00.330293) mnt:        Working on 27->1
(00.330312) mnt:        Working on 26->17
(00.330330) mnt:        Working on 25->1
(00.330348) mnt:        Working on 24->1
(00.330365) mnt:        Working on 23->1
(00.330384) mnt:        Working on 22->1
(00.330402) mnt:        Working on 21->20
(00.330420) mnt:        Working on 20->1
(00.330437) mnt:        Working on 19->1
(00.330455) mnt:        Working on 18->17
(00.330474) mnt:        Working on 17->1
(00.330491) mnt:        Working on 1->1
(00.330510) mnt:    Resorting children of 1 in mount order
(00.330532) mnt:    Resorting children of 35 in mount order
(00.330550) mnt:    Resorting children of 34 in mount order
(00.330569) mnt:    Resorting children of 33 in mount order
(00.330585) mnt:    Resorting children of 27 in mount order
(00.330603) mnt:    Resorting children of 25 in mount order
(00.330620) mnt:    Resorting children of 24 in mount order
(00.330638) mnt:    Resorting children of 23 in mount order
(00.330655) mnt:    Resorting children of 22 in mount order
(00.330673) mnt:    Resorting children of 20 in mount order
(00.330691) mnt:    Resorting children of 32 in mount order
(00.330708) mnt:    Resorting children of 31 in mount order
(00.330725) mnt:    Resorting children of 21 in mount order
(00.330743) mnt:    Resorting children of 19 in mount order
(00.330763) mnt:    Resorting children of 17 in mount order
(00.330785) mnt:    Resorting children of 38 in mount order
(00.330803) mnt:    Resorting children of 37 in mount order
(00.330820) mnt:    Resorting children of 36 in mount order
(00.330838) mnt:    Resorting children of 30 in mount order
(00.330865) mnt:    Resorting children of 29 in mount order
(00.330884) mnt:    Resorting children of 28 in mount order
(00.330901) mnt:    Resorting children of 26 in mount order
(00.330920) mnt:    Resorting children of 18 in mount order
(00.330940) mnt: Done:
(00.330956) mnt: [./](1->1)
(00.330974) mnt:  [./storage](35->1)
(00.330991) mnt:  <--
(00.331010) mnt:  [./data](34->1)
(00.331029) mnt:  <--
(00.331047) mnt:  [./cache](33->1)
(00.331065) mnt:  <--
(00.331083) mnt:  [./config](27->1)
(00.331101) mnt:  <--
(00.331120) mnt:  [./acct](25->1)
(00.331138) mnt:  <--
(00.331156) mnt:  [./vendor](24->1)
(00.331175) mnt:  <--
(00.331226) mnt:  [./system](23->1)
(00.331246) mnt:  <--
(00.331295) mnt:  [./mnt](22->1)
(00.331346) mnt:  <--
(00.331390) mnt:  [./sys](20->1)
(00.331410) mnt:   [./sys/fs/pstore](32->20)
(00.331429) mnt:   <--
(00.331448) mnt:   [./sys/fs/bpf](31->20)
(00.331468) mnt:   <--
(00.331520) mnt:   [./sys/fs/selinux](21->20)
(00.331538) mnt:   <--
(00.331585) mnt:  <--
(00.331635) mnt:  [./proc](19->1)
(00.331690) mnt:  <--
(00.331732) mnt:  [./dev](17->1)
(00.331755) mnt:   [./dev/usb-ffs/ptp](38->17)
(00.331774) mnt:   <--
(00.331817) mnt:   [./dev/usb-ffs/mtp](37->17)
(00.331869) mnt:   <--
(00.331911) mnt:   [./dev/usb-ffs/adb](36->17)
(00.331930) mnt:   <--
(00.331950) mnt:   [./dev/cg2_bpf](30->17)
(00.331969) mnt:   <--
(00.331989) mnt:   [./dev/cpuset](29->17)
(00.332012) mnt:   <--
(00.332032) mnt:   [./dev/cpuctl](28->17)
(00.332078) mnt:   <--
(00.332130) mnt:   [./dev/memcg](26->17)
(00.332181) mnt:   <--
(00.332223) mnt:   [./dev/pts](18->17)
(00.332242) mnt:   <--
(00.332260) mnt:  <--
(00.332280) mnt: <--
(00.332735) unix:   Collected: ino 13347 peer_ino 0 family    1 type    1 state 10 name /dev/socket/logd
(00.332765) unix:   Collected: ino 13352 peer_ino 0 family    1 type    5 state 10 name /dev/socket/logdr
(00.332791) unix:   Collected: ino 13355 peer_ino 0 family    1 type    2 state  7 name /dev/socket/logdw
(00.332816) unix:   Collected: ino 13626 peer_ino 0 family    1 type    5 state 10 name /dev/socket/tombstoned_crash
(00.332844) unix:   Collected: ino 13631 peer_ino 0 family    1 type    5 state 10 name /dev/socket/tombstoned_intercept
(00.332868) unix:   Collected: ino 13634 peer_ino 0 family    1 type    5 state 10 name /dev/socket/tombstoned_java_trace
(00.332894) unix:   Collected: ino 7758 peer_ino 0 family    1 type    1 state 10 name /dev/socket/property_service
(00.332918) unix:   Collected: ino 8032 peer_ino 0 family    1 type    2 state  7 name /dev/socket/statsdw
(00.332943) unix:   Collected: ino 10423 peer_ino 0 family    1 type    1 state 10 name /dev/socket/adbd
(00.332968) unix:   Collected: ino 302 peer_ino 0 family    1 type    5 state 10 name 
(00.332992) unix:   Collected: ino 8423 peer_ino 0 family    1 type    5 state 10 name /dev/socket/lmkd
(00.333015) unix:   Collected: ino 8571 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333040) unix:   Collected: ino 150459 peer_ino 150460 family    1 type    1 state  1 name (null)
(00.333068) unix:   Collected: ino 7756 peer_ino 7757 family    1 type    1 state  1 name (null)
(00.333093) unix:   Collected: ino 13665 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333117) unix:   Collected: ino 7768 peer_ino 7767 family    1 type    5 state  1 name (null)
(00.333143) unix:   Collected: ino 13779 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333168) unix:   Collected: ino 7770 peer_ino 7769 family    1 type    5 state  1 name (null)
(00.333193) unix:   Collected: ino 360 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333218) unix:   Collected: ino 388 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333243) unix:   Collected: ino 312 peer_ino 313 family    1 type    1 state  1 name (null)
(00.333266) unix:   Collected: ino 11656 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333290) unix:   Collected: ino 13397 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333317) unix:   Collected: ino 10522 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333340) unix:   Collected: ino 304 peer_ino 303 family    1 type    1 state  1 name (null)
(00.333365) unix:   Collected: ino 8079 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333389) unix:   Collected: ino 249 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333413) unix:   Collected: ino 8406 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333438) unix:   Collected: ino 300 peer_ino 301 family    1 type    1 state  1 name (null)
(00.333462) unix:   Collected: ino 13607 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333487) unix:   Collected: ino 9461 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333512) unix:   Collected: ino 10360 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333537) unix:   Collected: ino 7834 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333561) unix:   Collected: ino 150460 peer_ino 150459 family    1 type    1 state  1 name (null)
(00.333588) unix:   Collected: ino 8577 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333618) unix:   Collected: ino 8623 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333645) unix:   Collected: ino 7757 peer_ino 7756 family    1 type    1 state  1 name (null)
(00.333670) unix:   Collected: ino 7767 peer_ino 7768 family    1 type    5 state  1 name (null)
(00.333695) unix:   Collected: ino 7769 peer_ino 7770 family    1 type    5 state  1 name (null)
(00.333720) unix:   Collected: ino 13743 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333745) unix:   Collected: ino 380 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333770) unix:   Collected: ino 305 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333793) unix:   Collected: ino 313 peer_ino 312 family    1 type    1 state  1 name (null)
(00.333819) unix:   Collected: ino 13418 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333845) unix:   Collected: ino 301 peer_ino 300 family    1 type    1 state  1 name (null)
(00.333868) unix:   Collected: ino 10489 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333893) unix:   Collected: ino 12571 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333917) unix:   Collected: ino 303 peer_ino 304 family    1 type    1 state  1 name (null)
(00.333942) unix:   Collected: ino 8415 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333965) unix:   Collected: ino 10390 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.333990) unix:   Collected: ino 44 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.334014) unix:   Collected: ino 9250 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.334039) unix:   Collected: ino 10338 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.334093) unix:   Collected: ino 11346 peer_ino 13355 family    1 type    2 state  7 name (null)
(00.334339) inet:   Collected: ino   0xdcb8 family AF_INET    type SOCK_DGRAM     port    48561 state TCP_ESTABLISHED  src_addr 127.0.0.1
(00.334951) sockets: Sockects collect procedure family AF_INET proto IPPROTO_RAW: -2
(00.335730) sockets: Sockects collect procedure family AF_INET6 proto IPPROTO_RAW: -2
(00.336026) netlink: Collect netlink sock 0x3
(00.336063) netlink: Collect netlink sock 0x9f91c
(00.336076) netlink: Collect netlink sock 0x2001
(00.336087) netlink: Collect netlink sock 0x2003
(00.336097) netlink: Collect netlink sock 0x285f
(00.336109) netlink: Collect netlink sock 0x1e4b
(00.336120) netlink: Collect netlink sock 0x1de8
(00.336130) netlink: Collect netlink sock 0x1ea1
(00.336195) netlink: Collect netlink sock 0x1ea1
(00.336252) netlink: Collect netlink sock 0x285f
(00.336308) netlink: Collect netlink sock 0x1e4b
(00.336361) netlink: Collect netlink sock 0x1de3
(00.336414) netlink: Collect netlink sock 0x343e
(00.336468) netlink: Collect netlink sock 0x1da4
(00.336520) netlink: Collect netlink sock 0x401
(00.336572) netlink: Collect netlink sock 0x2002
(00.336634) netlink: Collect netlink sock 0x1ea2
(00.336687) netlink: Collect netlink sock 0x2407
(00.336740) netlink: Collect netlink sock 0x4
(00.336793) netlink: Collect netlink sock 0x2481
(00.336845) netlink: Collect netlink sock 0x35e9
(00.336899) netlink: Collect netlink sock 0x35e9
(00.336953) netlink: Collect netlink sock 0x2481
(00.337009) netlink: Collect netlink sock 0x1ea2
(00.337065) netlink: Collect netlink sock 0x2407
(00.337119) netlink: Collect netlink sock 0x402
(00.337842) Eventfd fdinfo works OK (13 vs 13)
(00.338238) Epoll fdinfo works OK
(00.355008) Inotify fdinfo works OK
(00.355147) Unaligned vmsplice works OK
(00.356739) Error (criu/cr-check.c:512): /proc/sys/kernel/sem_next_id is inaccessible: No such file or directory
(00.356816) Does not look good.

I have made sure that I have compiled the kernel with the correct CONFIG parameters listed at: https://criu.org/Linux_kernel

Do you have any idea where to start?

adrianreber commented 4 years ago

Are you running this as root?

mwerensteijn commented 4 years ago

Yes I am running this as root.

adrianreber commented 4 years ago

Can you share the log of a failed checkpoint?

mwerensteijn commented 4 years ago

Sure, this is the output of checkpointing a simple loop printing binary:

# ./criu dump -t 3352
Warn  (criu/net.c:3137): Unable to get tun network namespace
Error (criu/libnetlink.c:55): -95 reported by netlink: Operation not supported on transport endpoint
Error (criu/net.c:3453): Unable to create a veth pair: -95
Warn  (criu/net.c:3475): NSID isn't reported for network links
Error (criu/uffd.c:272): uffd: Lazy pages are not available: Function not implemented
Error (compel/src/lib/infect.c:792): Can't rmap memfd for parasite blob
Error (criu/cr-dump.c:1281): Can't infect (pid: 3352) with parasite
Error (criu/cr-dump.c:1764): Dumping FAILED.

Adding --shell-job gives the same output.

mwerensteijn commented 4 years ago

It looks like the returned -95's are not from libnet but from the kernel?: #define EOPNOTSUPP 95 /* Operation not supported on transport endpoint */

https://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/Errors/unix_system_errors.html

adrianreber commented 4 years ago

This is relevant error message

Error (compel/src/lib/infect.c:792): Can't rmap memfd for parasite blob
Error (criu/cr-dump.c:1281): Can't infect (pid: 3352) with parasite
mwerensteijn commented 4 years ago

Can it be because I compiled CRIU as 32 bit but running it on a 64 bit arm host? I found this thread online with the same issue: https://lists.openvz.org/pipermail/criu/2017-December/040081.html

The loop binary I am running is also compiled as 32 bit.

adrianreber commented 4 years ago

Ah, please try everything first as 64bit to make sure. Not sure how good CRIU works in mixed environments.

mwerensteijn commented 4 years ago

Until now it doesn't seem to be easy to build it with the Android aarch64 compiler. I get quite a lot of compiler errors. I remember that was the reason I compiled it with the 32 bit compiler. I'll have another look tomorrow, to check if it is doable in the time I have for my assignment.

mwerensteijn commented 4 years ago

Hey Adrian,

Good news! Yesterday evening I got a lot of compiler errors, many of them in assembly code so I thought it was because aarch64 wasn't completely supported yet. I checked the build log more closer and found out that there where still a lot of softlinks to arch/arm source files, so it was compiling the wrong files. I did perform clean, but perhaps I should have simply completely removed the criu directory.

Anyways, after setting up the repo again, rebuilding protobuf and the other libraries for arm64 and fixing some other smaller errors criu compiled succesfully for aarch64 with the android compiler:

file criu_64                                         
criu_64: ELF executable, 64-bit LSB arm64, static, for Android 28, built by NDK r21d (6528147), not stripped

And it worked! I can succesfully checkpoint and restore a simple binary!

I still see the following errors, but it does work:

./criu_64 dump -t 3029 --shell-job                     
Warn  (criu/net.c:3137): Unable to get tun network namespace
Error (criu/libnetlink.c:55): -95 reported by netlink: Operation not supported on transport endpoint
Error (criu/net.c:3453): Unable to create a veth pair: -95
Warn  (criu/net.c:3475): NSID isn't reported for network links
Error (criu/uffd.c:272): uffd: Lazy pages are not available: Function not implemented

And the same during restoring:

./criu_64 restore --shell-job                        
Warn  (criu/net.c:3137): Unable to get tun network namespace
Error (criu/libnetlink.c:55): -95 reported by netlink: Operation not supported on transport endpoint
Error (criu/net.c:3453): Unable to create a veth pair: -95
Warn  (criu/net.c:3475): NSID isn't reported for network links
Error (criu/uffd.c:272): uffd: Lazy pages are not available: Function not implemented

Thanks for all your help and pointers. Do you know if there are any limitations caused by these warnings / errors?

adrianreber commented 4 years ago

Warn (criu/net.c:3137): Unable to get tun network namespace

Error (criu/libnetlink.c:55): -95 reported by netlink: Operation not supported on transport endpoint
Error (criu/net.c:3453): Unable to create a veth pair: -95
Warn  (criu/net.c:3475): NSID isn't reported for network links

This errors/warnings are relevant for restoring processes in a network namespace if you are using certain network namespace features of CRIU.

Error (criu/uffd.c:272): uffd: Lazy pages are not available: Function not implemented

This is relevant if you want to do post-copy (also called lazy) migration. That relies on usefaultfd which does not seem enabled in your kernel.

Not knowing what you want to do, but seeing that your test was successful, this should not be a problem.

mwerensteijn commented 4 years ago

I am researching migrating apps from the normal world to the secure world with ARM trustzone. Seeing if it is possible to checkpoint apps (simple binaries for now) in the normal world and continue execution in the secure world so that it can run protected from memory dumping attacks.

Thank you for all your help. This issue is solved!

xy815661276 commented 4 years ago

Nice to see that works,I'm trying to use criu on android.Unlike you using android-ndk, I use patchelf to patch the compiled dynamic criu.Can you provide your compilation script? I have used android-ndk to compile successfully before, but it cannot run successfully. My implementation is here： https://github.com/xy815661276/Android-criu

ScarletCarpet commented 1 year ago

Nice to see that works,I'm trying to use criu on android.Unlike you using android-ndk, I use patchelf to patch the compiled dynamic criu.Can you provide your compilation script? I have used android-ndk to compile successfully before, but it cannot run successfully. My implementation is here： https://github.com/xy815661276/Android-criu

hi, i'm working on android criu also, i can not compile criu causing some error such as header-missing, redefinition, etc. i wonder if you have run it on android successfully. best apperiated if you reply to me.