Fails to build on i386 and s390x

nileshpatra commented 2 years ago

Hi,

Thanks for your work on isal. The corresponding debian package (version 2.30) does not build on i386 and s390x

In i386, there's a number of:

raid/raid_multibinary_i32.s:44: error: instruction not supported in 32-bit mode
raid/raid_multibinary_i32.s:111: ... from macro `mbin_interface' defined here
raid/raid_multibinary_i32.s:45: error: instruction not supported in 32-bit mode
raid/raid_multibinary_i32.s:111: ... from macro `mbin_interface' defined here
raid/raid_multibinary_i32.s:46: error: instruction not supported in 32-bit mode
raid/raid_multibinary_i32.s:111: ... from macro `mbin_interface' defined here
raid/raid_multibinary_i32.s:47: error: instruction not supported in 32-bit mode
raid/raid_multibinary_i32.s:111: ... from macro `mbin_interface' defined here

Full log here

In s390x, tests segfault.

/usr/bin/make --no-print-directory check-TESTS
FAIL: erasure_code/gf_vect_mul_test
FAIL: erasure_code/erasure_code_test
PASS: erasure_code/gf_inverse_test
FAIL: erasure_code/erasure_code_update_test
PASS: raid/xor_gen_test
PASS: raid/pq_gen_test
PASS: raid/xor_check_test
PASS: raid/pq_check_test
PASS: crc/crc16_t10dif_test
PASS: crc/crc16_t10dif_copy_test
PASS: crc/crc64_funcs_test
PASS: crc/crc32_funcs_test
./build-aux/test-driver: line 112: 2937012 Segmentation fault      "$@" >> "$log_file" 2>&1
FAIL: igzip/igzip_rand_test
PASS: igzip/igzip_wrapper_hdr_test
PASS: igzip/checksum32_funcs_test
PASS: mem/mem_zero_detect_test
============================================================================
Testsuite summary for libisal 2.30.0
============================================================================
# TOTAL: 16
# PASS:  12
# SKIP:  0
# XFAIL: 0
# FAIL:  4
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to sg.support.isal@intel.com

Full log here

Would you please consider fixing these? Thanks, Nilesh

gbtucker commented 2 years ago

Thanks for reporting @nileshpatra. Anyone at ibm such as @penghongbo willing to look at s390x?

nileshpatra commented 2 years ago

Hi Greg, I'm not sure if that question is geared towards me. I'm personally not involved with IBM in any capacity, rather, trying to improve this package in debian so that $stuff depending on it is also usable for these archs. I however have access to some arch specific VMs, and I could help "test" fixes there.

Thanks for your work again, Nilesh

gbtucker commented 2 years ago

Thanks @nileshpatra. No, I don't have access to IBM HW either. I can cross-compile for powerpc64le and this passes check tests.

$ make -f Makefile.unx host_cpu=ppc64le CC=powerpc64le-linux-gcc LDFLAGS=-static check
  ---> Building …  ppc64le
...  done: Pass
Finished running check

@penghongbo contributed the ppc code so I was hoping could test.

penghongbo commented 2 years ago

Sorry, I used my other account.

I don't have access to S390. I am trying to see whether any team inside IBM can take a look. But I didn't get a confirm by now.

penghongbo commented 2 years ago

Sorry, I don't get confirmation from the Z team who would like to take over. Let's see whether someone in community can help on this issue.

iii-i commented 2 years ago

I'm working on fixing s390 test failures. The WIP branch is here: https://github.com/iii-i/isa-l/tree/s390

a16bitsysop commented 2 years ago

I had a look at i386 on alpine linux

If I use ./configure --host=i586-linux-gnu

and the following patch:

-- a/configure.ac
+++ b/configure.ac
@@ -223,7 +223,12 @@
   fi

   case $host_os in
-       *linux*)  arch=linux   yasm_args="-f elf64";;
+       *linux*)  arch=linux
+                 if test "$CPU" = "x86_32"; then
+                   yasm_args="-f elf32"
+                 else
+                   yasm_args="-f elf64"
+                 fi;;        *darwin*) arch=darwin  yasm_args="-f macho64 --prefix=_ ";;
        *netbsd*) arch=netbsd  yasm_args="-f elf64";;        *mingw*)  arch=mingw   yasm_args="-f win64";;

I don’t get the 32bit/64bit errors, but I get

  CCLD     libisal.la
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: erasure_code/.libs/ec_multibinary.o: warning: relocation in read-only section `.text'
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: warning: creating DT_TEXTREL in a shared object
  CCLD     programs/igzip
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: ./.libs/libisal.so: undefined reference to `gf_5vect_dot_prod_avx'
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: ./.libs/libisal.so: undefined reference to `gf_6vect_dot_prod_avx2'
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: ./.libs/libisal.so: undefined reference to `gf_6vect_dot_prod_sse'
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: ./.libs/libisal.so: undefined reference to `gf_5vect_dot_prod_avx2'
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: ./.libs/libisal.so: undefined reference to `gf_6vect_dot_prod_avx'
/usr/lib/gcc/i586-alpine-linux-musl/11.2.1/../../../../i586-alpine-linux-musl/bin/ld: ./.libs/libisal.so: undefined reference to `gf_5vect_dot_prod_sse'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:3301: programs/igzip] Error 1

a16bitsysop commented 2 years ago

I found another way to compile on alpine and avoid textrels, use the generic files for x86:

--- a/Makefile.am
+++ b/Makefile.am
@@ -64,8 +64,7 @@
 endif

 if CPU_X86_32
-libisal_la_SOURCES += ${lsrc_x86_32}
-other_tests += ${other_tests_x86_32}
+libisal_la_SOURCES += ${lsrc_base_aliases}
 endif

 if CPU_AARCH64

The configure patch is then no longer required.

So it will likely take a performance hit from when it did compile, but then x86_32 code could be removed if it does not compile anymore.

milesgranger commented 2 years ago

Hi. Curious if this is the issue relevant for the following error when trying powerpc64le and s390x. Seems like none of .asm are being picked up.

Reproduce: docker run --rm -v $(pwd):/isa-l muslcc/x86_64:s390x-linux-musl sh -c "apk add make nasm && cd /isa-l && make install prefix=./install host_cpu=s390x -f Makefile.unx CFLAGS_s390x=-fPIC LDFLAGS=-static"

Full output

> fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz (1/2) Installing make (4.3-r0) (2/2) Installing nasm (2.15.05-r0) Executing busybox-1.33.1-r6.trigger OK: 10 MiB in 17 packages mkdir -p install/lib mkdir -p install/include/isa-l mkdir -p install/bin mkdir -p install/share/man/man1 mkdir -p bin ---> Building erasure_code/ec_base.c s390x ---> Building raid/raid_base.c s390x ---> Building crc/crc_base.c s390x ---> Building crc/crc64_base.c s390x ---> Building igzip/igzip.c s390x ---> Building igzip/hufftables_c.c s390x ---> Building igzip/igzip_base.c s390x ---> Building igzip/igzip_icf_base.c s390x ---> Building igzip/adler32_base.c s390x ---> Building igzip/flatten_ll.c s390x ---> Building igzip/encode_df.c s390x ---> Building igzip/igzip_icf_body.c s390x ---> Building igzip/huff_codes.c s390x ---> Building igzip/igzip_inflate.c s390x ---> Building mem/mem_zero_detect_base.c s390x ---> Creating Lib bin/isa-l.a ---> Building shared erasure_code/ec_base.c s390x ---> Building shared raid/raid_base.c s390x ---> Building shared crc/crc_base.c s390x ---> Building shared crc/crc64_base.c s390x ---> Building shared igzip/igzip.c s390x ---> Building shared igzip/hufftables_c.c s390x ---> Building shared igzip/igzip_base.c s390x ---> Building shared igzip/igzip_icf_base.c s390x ---> Building shared igzip/adler32_base.c s390x ---> Building shared igzip/flatten_ll.c s390x ---> Building shared igzip/encode_df.c s390x ---> Building shared igzip/igzip_icf_body.c s390x ---> Building shared igzip/huff_codes.c s390x ---> Building shared igzip/igzip_inflate.c s390x ---> Building shared mem/mem_zero_detect_base.c s390x ---> Creating Shared Lib bin/libisal.so ---> Building Programs programs/igzip DEBUG s390x /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: /tmp/ccoIddeJ.o: in function `compress_file': /isa-l/programs/igzip_cli.c:683: undefined reference to `crc32_gzip_refl' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip.o): in function `update_checksum': (.text+0x378): undefined reference to `isal_adler32' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x3c4): undefined reference to `crc32_gzip_refl' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip.o): in function `isal_deflate_pass': (.text+0x1368): undefined reference to `isal_deflate_body' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x1380): undefined reference to `isal_deflate_finish' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip.o): in function `isal_deflate_icf_pass': (.text+0x1624): undefined reference to `isal_deflate_icf_finish_lvl3' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x192c): undefined reference to `encode_deflate_icf' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x1c80): undefined reference to `isal_deflate_icf_finish_lvl1' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x1c90): undefined reference to `isal_deflate_icf_finish_lvl2' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip.o): in function `isal_adler32_bam1': (.text+0x1ffc): undefined reference to `isal_adler32' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip.o): in function `isal_write_gzip_header': (.text+0x2408): undefined reference to `crc32_gzip_refl' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip.o): in function `isal_deflate_hash': (.text+0x26a0): undefined reference to `isal_deflate_hash_lvl0' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x26f8): undefined reference to `isal_deflate_hash_lvl2' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x2758): undefined reference to `isal_deflate_hash_lvl1' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x27b8): undefined reference to `isal_deflate_hash_lvl3' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip.o): in function `isal_deflate_process_dict': (.text+0x28c4): undefined reference to `isal_deflate_hash_lvl0' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x2938): undefined reference to `isal_deflate_hash_lvl1' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x296c): undefined reference to `isal_deflate_hash_lvl3' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x299c): undefined reference to `isal_deflate_hash_lvl2' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip_icf_body.o): in function `icf_body_hash1_fillgreedy_lazy': (.text+0xabc): undefined reference to `set_long_icf_fg' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip_icf_body.o): in function `icf_body_lazyhash1_fillgreedy_greedy': (.text+0xc28): undefined reference to `gen_icf_map_lh1' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0xc50): undefined reference to `set_long_icf_fg' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip_icf_body.o): in function `isal_deflate_icf_body': (.text+0xd68): undefined reference to `isal_deflate_icf_body_lvl3' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0xd80): undefined reference to `isal_deflate_icf_body_lvl2' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0xda0): undefined reference to `isal_deflate_icf_body_lvl1' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(huff_codes.o): in function `isal_create_hufftables': (.text+0x13f8): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x140c): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x1884): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x1898): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x1f2c): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x1f40): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x2748): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x275c): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x27f0): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x2804): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x2c0c): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x2c20): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(huff_codes.o): in function `isal_create_hufftables_subset': (.text+0x356c): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x3580): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x39e8): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x39fc): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x408c): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x40a0): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x48a8): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x48bc): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x4918): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x492c): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x49c0): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x49d4): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x4dec): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x4e00): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(huff_codes.o): in function `create_hufftables_icf': (.text+0x5854): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x5868): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x5cc0): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x5cd4): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x66f0): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x6704): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x6758): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x676c): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x682c): undefined reference to `build_heap' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x6840): undefined reference to `build_huff_tree' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip_inflate.o): in function `update_checksum': (.text+0x2b0): undefined reference to `crc32_gzip_refl' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip_inflate.o): in function `isal_read_gzip_header': (.text+0x40b4): undefined reference to `crc32_gzip_refl' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x4334): undefined reference to `crc32_gzip_refl' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip_inflate.o): in function `isal_inflate_stateless': (.text+0x4d08): undefined reference to `decode_huffman_code_block_stateless' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: bin/isa-l.a(igzip_inflate.o): in function `isal_inflate': (.text+0x5228): undefined reference to `decode_huffman_code_block_stateless' /lib/gcc/powerpc64le-linux-musl/11.2.1/../../../../powerpc64le-linux-musl/bin/ld: (.text+0x5328): undefined reference to `decode_huffman_code_block_stateless' collect2: error: ld returned 1 exit status make: *** [make.inc:252: programs/igzip] Error 1

Meanwhile the same command for x86_64 and aarch64 works fine. ie: docker run --rm -v $(pwd):/isa-l muslcc/x86_64:aarch64-linux-musl sh -c "apk add make nasm && cd /isa-l && make install prefix=./install host_cpu=aarch64 -f Makefile.unx CFLAGS_aarch64=-fPIC LDFLAGS=-static"

a16bitsysop commented 2 years ago

it is in alpine edge for ppc64le already, and this MR https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/35679 succeeded in building for s390x

it’s only in edge as it’s still in the testing repo

intel / isa-l

Fails to build on i386 and s390x #188