Illegal instruction bug (on Arch Linux)

[andreixk]

I'm running lrzip on linux with 2gb ram, 40Gb of harddrive space, 1.4Ghz octacore CPU.

Linux version:

$ uname -a
Linux mysrv 4.9.30-1-ARCH #1 SMP PREEMPT Sun Jun 4 01:08:38 UTC 2017 armv7l GNU/Linux

$ lrzip -V
lrzip version 0.631

When trying to compress a 50Mb pdf file lrzip encounters an error:

$ lrzip -v -z doc.pdf 
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 8
Detected 2090811392 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Verbose
Temporary Directory set as: ./
Compression mode is: ZPAQ. LZO Compressibility testing enabled
Heuristically Computed Compression Window: 13 = 1300MB
Output filename is: doc.pdf.lrz
File size: 58584688
Will take 1 pass
Beginning rzip pre-processing phase
Illegal instruction (core dumped)

[diodelass]

This is probably caused by libzpaq attempting to use JIT features that are only supported on x86, while your system appears to be ARM. When compiling zpaq's own utility for ARM, one needs to add -DNOJIT to the compiler flags, or a similar error is produced at runtime. I don't have enough experience with makefiles to know where to put compiler flags for libzpaq here, but it might just be a matter of passing in -DNOJIT to get this working on non-x86 architectures.

[pete4abw]

True, @diodelass, @andreixk . From the ZPAQ README:

Normally you can use "make" to compile for Unix, Linux, or Mac OS/X or compile like this:

g++ -O3 -march=native -Dunix zpaq.cpp libzpaq.cpp -pthread -o zpaq

To compile for non x86 or x86-64 hardware use option -DNOJIT Some compilers complain about "-march=native" option. If so, take it out.

A few fixes will be in order. In configure.ac a test for $host will have to be made for no x86 architectures. A new Makefile.am will have to be made for libzpaq so that it can interpret the variable and set -DNOJIT in the compiler flags. Then, the main Makefile.am will have to be changed to take out libzpaq from compilation. Something like this:

check output of AC_CANONICAL_HOST set a ZPAQ_COMPILE flag in ZPAQ Makefile.am, test flag and set -DNOJIT as a compile flag

Also will look at Issue #133 for updating ZPAQ which is in version 7 now.

[pete4abw]

@andreixk , I'm interested to know if this patch will work. This is a provisional fix changing only one line in Makefile.am and emitting some configure variables in configure.ac.

1. download and extract the source
2. apply the attached patch zcat arm.patch.gz | patch -p1
3. run ./autogen.sh
4. run make
5. test with ./lrzip -vv somefile (I want max verbosity output)
6. respond here with output

If it works, let me know. Then we can bake it in to configure.ac.

[diodelass]

Apologies for taking so long to respond.

make fails to complete, and emits the following output:

make  all-recursive
make[1]: Entering directory '[...]/lrzip'
Making all in lzma
make[2]: Entering directory '[...]/lrzip/lzma'
Making all in C
make[3]: Entering directory '[...]/lrzip/lzma/C'
  CC       LzFindMt.lo
  CC       LzmaDec.lo
  CC       7zCrcT8.lo
  CC       LzFind.lo
  CC       LzmaEnc.lo
  CC       LzmaLib.lo
  CC       Alloc.lo
  CC       Threads.lo
nasm  -o 7zCrcOpt_asm.o ../../lzma/ASM/x86/7zCrcOpt_asm.asm
../../lzma/ASM/x86/7zCrcOpt_asm.asm:4: fatal: unable to open include file `7zAsm.asm'
make[3]: *** [Makefile:707: 7zCrcOpt_asm.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory '[...]/lrzip/lzma/C'
make[2]: *** [Makefile:430: all-recursive] Error 1
make[2]: Leaving directory '[...]/lrzip/lzma'
make[1]: *** [Makefile:958: all-recursive] Error 1
make[1]: Leaving directory '[...]/lrzip'
make: *** [Makefile:556: all] Error 2

[pete4abw]

Are you running make from the lrzip source directory?

make[1]: Entering directory '[...]/lrzip'

No changes were made to lzma... Makefiles.

[diodelass]

Tried again today, and it worked. I don't think I made any directory location errors the first time (the [...] in my previous post was just eliding my home directory path, which I don't think is relevant). In any case, it works fine now. The output of ./lrzip -vv -z lrzip:

The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 6
Detected 3979325440 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Max Verbose
Temporary Directory set as: ./
Compression mode is: ZPAQ. LZO Compressibility testing enabled
Heuristically Computed Compression Window: 25 = 2500MB
Storage time in seconds 1367943756
Output filename is: lrzip.lrz
File size: 1131128
Succeeded in testing 1131128 sized mmap for rzip pre-processing
Will take 1 pass
Chunk size: 1131128
Byte width: 3
Succeeded in testing 823214712 sized malloc for back end compression
Using up to 7 threads to compress up to 1131128 bytes each.
Beginning rzip pre-processing phase
hashsize = 4194304.  bits = 22. 64MB
430734 total hashes -- 337615 in primary bucket (78.381%)
Malloced 1326440448 for checksum ckbuf
Starting thread 0 to compress 20293 bytes from stream 0
Starting thread 1 to compress 1020127 bytes from stream 1
lzo testing OK for chunk 20293. Compressed size = 75.78% of chunk, 1 Passes
lzo testing OK for chunk 1020127. Compressed size = 27.62% of chunk, 1 Passes
Writing initial chunk bytes value 3 at 24:0%  
Writing EOF flag as 1
Writing initial header at 29
Compthread 0 seeking to 7 to store length 3
Compthread 0 seeking to 20 to write header
Thread 0 writing 9195 compressed bytes from stream 0
Compthread 0 writing data at 30
Compthread 1 seeking to 17 to store length 30%  
Compthread 1 seeking to 9225 to write header
Thread 1 writing 336506 compressed bytes from stream 1
Compthread 1 writing data at 9235
MD5: 1f977f2b9dc44f6f44ad63ee34772e6e
matches=2267 match_bytes=111001
literals=2229 literal_bytes=1020127
true_tag_positives=18686 false_tag_positives=408594
inserts=542423 match 0.109
lrzip - Compression Ratio: 3.271. Average Compression Speed:  0.250MB/s.
Total time: 00:00:04.50

Looks good to me!

[pete4abw]

Next time you have this issue, log it completely by including the path spec in your command. i.e.

/src/lrzip/ $> make all

and all the verbiage that comes with lrzip -vv.