search

sahib / rmlint

Extremely fast tool to remove duplicates and other lint from your filesystem
http://rmlint.rtfd.org
GNU General Public License v3.0
1.86k stars 128 forks source link

Compiling develop branch on Raspberry Pi 4 - immintrin.h missing - ARM - x86 issue #547

Closed james-cook closed 1 year ago

james-cook commented 2 years ago

I am trying to avoid the problems I was having in 2020 with xattrs giving strange results (https://github.com/sahib/rmlint/issues/436) and seemingly solved in https://github.com/sahib/rmlint/issues/439

To do this I am trying to compile the current develop branch and following the instructions at https://rmlint.readthedocs.io/en/latest/install.html

sudo apt install git scons python3-sphinx python3-nose gettext build-essential
sudo apt install libelf-dev libglib2.0-dev libblkid-dev libjson-glib-1.0 libjson-glib-dev

scons config seems fine. But scons DEBUG=1 leads to a fatal error:

Compiling ==> lib/checksums/blake3/blake3_portable.c
lib/checksums/blake3/blake3_dispatch.c:85:5: warning: 'get_cpu_features' defined but not used [-Wunused-function]
     get_cpu_features(void) {
     ^~~~~~~~~~~~~~~~
Compiling ==> lib/checksums/blake3/blake3_sse2.c
lib/checksums/blake3/blake3_sse2.c:3:10: fatal error: immintrin.h: No such file or directory
 #include <immintrin.h>
          ^~~~~~~~~~~~~
compilation terminated.
scons: *** [lib/checksums/blake3/blake3_sse2.o] Error 1
scons: building terminated because of errors.

Blake3 is new and only present in the develop branch (I have had no problems compiling master 2.10.1) From (just as examples) https://github.com/suijingfeng/vkQuake3/issues/14#issuecomment-738144064 it looks like immintrin.h is for x86 only(?)

james-cook commented 2 years ago

More output from the scons DEBUG=1 command:

As a result of "Unable to get cpuinfo, maybe install py-cpuinfo" I did install py-cpuinfo:

 dpkg -l | grep py-cpu
ii  py-cpuinfo                     4.0.0-1                             all          Python script for getting CPU info

which seems to have removed the 1st warning but I still get the 2nd error, and I am still advised to install py-cpuinfo although it is now installed...

 scons DEBUG=1
scons: Reading SConscript files ...
Checking whether the C compiler works... (cached) yes
Checking for git revision... (cached) yes
Checking for pkg-config... (cached) yes
Checking for glib-2.0 >= 2.32... (cached) yes
Checking for gio-unix-2.0... (cached) yes
Checking for blkid... (cached) yes
Checking for json-glib-1.0... (cached) yes
Checking for -std=c11 support...(cached) yes
Checking for cygwin environment...(cached) Linux/raspberrypi/5.10.63-v7l+/#1496 SMP Wed Dec 1 15:58:56 GMT 2021/armv7l/(cached) no
Checking whether _mm_crc32_u64 is declared... (cached) no
Checking for GCC version... (cached) 8
Checking whether __builtin_cpu_supports is declared... (cached) no
Checking whether blkid_devno_to_wholedisk is declared... (cached) yes
Checking for existence of /sys/block... (cached) yes
Checking for C header file libelf.h... (cached) yes
Checking for C library libelf... (cached) yes
Checking for C type struct fiemap... (cached) yes
Checking for C function getxattr()... (cached) yes
Checking for C function setxattr()... (cached) yes
Checking for C function removexattr()... (cached) yes
Checking for C function listxattr()... (cached) yes
Checking for C function lgetxattr()... (cached) yes
Checking for C function lsetxattr()... (cached) yes
Checking for C function lremovexattr()... (cached) yes
Checking for C function llistxattr()... (cached) yes
Checking size of off_t ... (cached) yes
Checking for C function stat64()... (cached) yes
Checking whether G_CHECKSUM_SHA512 is declared... (cached) yes
Checking for C header file locale.h... (cached) yes
Checking for C header file linux/limits.h... (cached) yes
Checking whether posix_fadvise is declared... (cached) yes
Checking for C header file linux/btrfs.h... (cached) yes
Checking for C header file linux/fs.h... (cached) yes
Checking for C header file sys/utsname.h... (cached) yes
Checking for C header file sys/sysmacros.h... (cached) yes
==> Checking CPU checksum and vector extensions...
   Unable to get cpuinfo, maybe install py-cpuinfo
    AVX512F: 0
    AVX512VL: 0
    AVX2: 0
    SSE4_1: 0
    SSE2: 0
Compiling in debug mode
Using compiler optimisation -Og (to change, run scons with O=[0|1|2|3|s|fast])
Running with --jobs=4
Not compiling blake3 avx2 options
Not compiling blake3 avx512 options
Not compiling blake3 sse41 options
Not compiling blake3 sse2 options
Building rmlint
scons: done reading SConscript files.
scons: Building targets ...
build_config_template(["lib/config.h"], ["lib/config.h.in"])
Compiling ==> lib/checksums/blake3/blake3_sse2.c
gcc -I/usr/include/json-glib-1.0 -I/usr/include/gio-unix-2.0 -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/glib-2.0 -I/usr/lib/arm-linux-gnueabihf/glib-2.0/include -c -o lib/checksums/blake3/blake3_sse2_x86-64_unix.o lib/checksums/blake3/blake3_sse2_x86-64_unix.S
gcc -I/usr/include/json-glib-1.0 -I/usr/include/gio-unix-2.0 -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/uuid -I/usr/include/glib-2.0 -I/usr/lib/arm-linux-gnueabihf/glib-2.0/include -c -o lib/checksums/blake3/blake3_sse2_x86-64_windows_gnu.o lib/checksums/blake3/blake3_sse2_x86-64_windows_gnu.S
lib/checksums/blake3/blake3_sse2.c:3:10: fatal error: immintrin.h: No such file or directory
 #include <immintrin.h>
          ^~~~~~~~~~~~~
compilation terminated.
lib/checksums/blake3/blake3_sse2_x86-64_unix.S: Assembler messages:
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:15: Error: unknown pseudo-op: `.intel_syntax'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:31: Error: expression too complex -- `push r15'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:32: Error: expression too complex -- `push r14'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:33: Error: expression too complex -- `push r13'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:34: Error: expression too complex -- `push r12'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:35: Error: expression too complex -- `push rbx'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:36: Error: expression too complex -- `push rbp'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:37: Error: ARM register expected -- `mov rbp,rsp'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:38: Error: ARM register expected -- `sub rsp,360'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:39: Error: ARM register expected -- `and rsp,0xFFFFFFFFFFFFFFC0'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:40: Error: ARM register expected -- `neg r9d'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:41: Error: bad instruction `movd xmm0,r9d'
lib/checksums/blake3/blake3_sse2_x86-64_unix.S:42: Error: bad instruction `pshufd xmm0,xmm0,0x00'
cebtenzzre commented 2 years ago

Looks like a copy-paste error, try changing "sse41" to "sse2" in lib/SConscript here:

diff --git a/lib/SConscript b/lib/SConscript
index 3ae5154b..7acfb356 100644
--- a/lib/SConscript
+++ b/lib/SConscript
@@ -112,7 +112,7 @@ if not env['HAVE_SSE4_1']:

 if not env['HAVE_SSE2']:
     print('Not compiling blake3 sse2 options')
-    blake3 = [f for f in blake3 if 'sse41' not in f.path]
+    blake3 = [f for f in blake3 if 'sse2' not in f.path]
     env.Append(CCFLAGS=['-DBLAKE3_NO_SSE2'])

 library = env.Library(
james-cook commented 2 years ago

That does work, and rmlint is compiled and installed :)

However I am now getting this error:

rmlint --progress -S dma -s -1TB --keep-all-tagged DIR1 // DIR2
?¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦?                Traversing (25831 usable files / 4013 + 2 ignored files / folders)
**
ERROR:lib/pathtricia.c:80:rm_node_check_inode: assertion failed: (node->inode != RM_NO_INODE)
ERROR: Aborting due to a fatal error. (signal received: Aborted)
ERROR: Please file a bug report (See rmlint -h)

Should I file a separate report? (I have gone back to master to check and it compiles and runs without error with this same command)

cebtenzzre commented 2 years ago

Go ahead and open a separate issue for the assertion failure. It would be helpful if you could run rmlint in gdb (gdb --args rmlint ...) and print a backtrace. It seems like it should actually be impossible without some kind of corruption so building with ASAN would also be useful (CFLAGS='-fsanitize=address' LDFLAGS='-fsanitize=address' scons DEBUG=1).

james-cook commented 2 years ago

https://github.com/sahib/rmlint/issues/549

Go ahead and open a separate issue for the assertion failure. It would be helpful if you could run rmlint in gdb (gdb --args rmlint ...) and print a backtrace. It seems like it should actually be impossible without some kind of corruption so building with ASAN would also be useful (CFLAGS='-fsanitize=address' LDFLAGS='-fsanitize=address' scons DEBUG=1).

See https://github.com/sahib/rmlint/issues/549

james-cook commented 2 years ago

For me this is closed, please reopen if you don't agree :)