Illegal instruction error on aarch64 while decompressing with JIT

[Semisol]

While trying to decompress an archive from an x64 system on an aarch64 system with JIT enabled, the program crashes with Illegal instruction.

If I use -DNOJIT, it works fine.

[fcorbelli]

(Short version)This is normal

Long version, aka the "spiegone"

The zpaq archives contain within them the code to be decompressed, written in a language (which is called ZPAQL) running on a kind of virtual machine (!)

You can think of it as a kind of super-reduced version of Java

ZPAQL code is normally interpreted (-DNOJIT), and this is relatively slow, but it works on any type of CPU (arm, PowerPC, M1, etc)

On Intel processors (actually not quite all of them, the oldest of the old are excluded) the default compilation uses a JIT (just-in-time), essentially translating ZPAQL code into Intel code. This makes processing faster. It only works on Intel CPUs, though.

That's why on "anything" that is not an Intel CPU (e.g. arm) you need to use -DNOJIT.

Is it possible to write a JIT for arm? Of course it is, but I will not be the one to do it. I personally use Intel (AMD actually) machines.

[Semisol]

Thank you.

[fcorbelli]

Can you please try the attached pre-release? Run the b (benchmark) command and look at the output

zpaqfranz b

with and without -DNOJIT

BTW there is an autotest command

zpaqfranz autotest -all -to /tmp/testme

then run the .sh inside /tmp/testme (about 15GB of free space is needed)

[Semisol]

I can't see the attached file.

[fcorbelli]

I can't see the attached file.

60_4c.zip Please try

[Semisol]

Benchmark doesn't seem to be affected as it only measures hash functions. Still getting an Illegal instruction error on the autotest generated script.

benchmark

JIT:

zpaqfranz v60.4c-JIT-L(2024-07-13)
I am not sure this is Intel/AMD CPU (virtual? arm?)
WARNING: non Intel/AMD CPUs should be compiled with -DNOJIT
uname aarch64
full exename seems <</home/ubuntu/zpaqfranz>>
Free RAM seems 3.597.389.824
Benchmarks: XXHASH64 XXH3 SHA-1 SHA-256 BLAKE3 CRC-32 CRC-32C WYHASH WHIRLPOOL MD5 SHA-3 NILSIMSA HIGHWAY64
Time limit          5 s  (-n X)
Chunks of     390.62 KB  (-minsize Y)

00000005 s     XXHASH64: speed (    2.43 GB/s)
00000005 s         XXH3: speed (    4.08 GB/s)
00000005 s        SHA-1: speed (  480.88 MB/s)
00000005 s      SHA-256: speed (  154.28 MB/s)
00000005 s       BLAKE3: speed (  309.07 MB/s)
00000005 s       CRC-32: speed (    7.13 GB/s)
00000005 s      CRC-32C: speed (    1.24 GB/s)
00000005 s       WYHASH: speed (    7.14 GB/s)
00000005 s    WHIRLPOOL: speed (  118.41 MB/s)
00000005 s          MD5: speed (  439.07 MB/s)
00000005 s        SHA-3: speed (  316.16 MB/s)
00000005 s     NILSIMSA: speed (    7.13 GB/s)
00000005 s    HIGHWAY64: speed (  967.10 MB/s)

Results:

   WHIRLPOOL:   118.41 MB/s (done   592.04 MB)
     SHA-256:   154.28 MB/s (done   771.72 MB)
      BLAKE3:   309.07 MB/s (done     1.51 GB)
       SHA-3:   316.16 MB/s (done     1.54 GB)
         MD5:   439.07 MB/s (done     2.14 GB)
       SHA-1:   480.88 MB/s (done     2.35 GB)
   HIGHWAY64:   967.10 MB/s (done     4.72 GB)
     CRC-32C:     1.24 GB/s (done     6.21 GB)
    XXHASH64:     2.43 GB/s (done    12.17 GB)
        XXH3:     4.08 GB/s (done    20.42 GB)
    NILSIMSA:     7.13 GB/s (done    35.65 GB)
      CRC-32:     7.13 GB/s (done    35.66 GB)
      WYHASH:     7.14 GB/s (done    35.71 GB)

franzomips single thread index 2.991  (quick CPU check, raw 2.991)
AL314 (A15) 1.7G (phy)  2     842.59 %
Atom N2800       (phy)  4     734.94 %
Celeron J4125 2G (phy)  4     229.74 %
Xeon E3 1245 V2  (vir)  4     123.86 %
Celeron N5105    (phy)  4     164.53 %
i5-6200U         (phy)  2     157.18 %
Xeon E5 2620 V4  (phy)  8     161.86 %
Xeon E5 2630 V4  (phy) 10     192.73 %
Xeon D-1541      (vir)  8     147.28 %
i5-3570          (phy)  4     101.19 %
i7-4790K         (phy)  4      91.61 %
i7-8700K         (phy)  6      89.08 %
AMD-Ryzen 7 3700X(phy)  8      91.87 %
i9-9900K         (phy)  8      78.08 %
i9-10900         (phy) 10      80.73 %
AMD-3950X        (phy) 16      71.90 %
AMD-5950X        (phy) 16      62.34 %
i9-12900KS 56400 (phy) 16      55.36 %
AMD-7950X3D      (phy) 16      54.22 %
65.008 seconds (00:01:05) (all OK)

No JIT:

zpaqfranz v60.4c-NOJIT-L(2024-07-13)
I am not sure this is Intel/AMD CPU (virtual? arm?)
uname aarch64
full exename seems <</home/ubuntu/zpaqfranz-nojit>>
Free RAM seems 3.606.810.624
Benchmarks: XXHASH64 XXH3 SHA-1 SHA-256 BLAKE3 CRC-32 CRC-32C WYHASH WHIRLPOOL MD5 SHA-3 NILSIMSA HIGHWAY64
Time limit          5 s  (-n X)
Chunks of     390.62 KB  (-minsize Y)

00000005 s     XXHASH64: speed (    2.43 GB/s)
00000005 s         XXH3: speed (    4.08 GB/s)
00000005 s        SHA-1: speed (  481.03 MB/s)
00000005 s      SHA-256: speed (  154.24 MB/s)
00000005 s       BLAKE3: speed (  308.99 MB/s)
00000005 s       CRC-32: speed (    7.13 GB/s)
00000005 s      CRC-32C: speed (    1.24 GB/s)
00000005 s       WYHASH: speed (    7.14 GB/s)
00000005 s    WHIRLPOOL: speed (  118.54 MB/s)
00000005 s          MD5: speed (  439.00 MB/s)
00000005 s        SHA-3: speed (  316.39 MB/s)
00000005 s     NILSIMSA: speed (    7.13 GB/s)
00000005 s    HIGHWAY64: speed (  965.35 MB/s)

Results:

   WHIRLPOOL:   118.54 MB/s (done   592.81 MB)
     SHA-256:   154.24 MB/s (done   771.34 MB)
      BLAKE3:   308.99 MB/s (done     1.51 GB)
       SHA-3:   316.39 MB/s (done     1.54 GB)
         MD5:   439.00 MB/s (done     2.14 GB)
       SHA-1:   481.03 MB/s (done     2.35 GB)
   HIGHWAY64:   965.35 MB/s (done     4.71 GB)
     CRC-32C:     1.24 GB/s (done     6.22 GB)
    XXHASH64:     2.43 GB/s (done    12.18 GB)
        XXH3:     4.08 GB/s (done    20.42 GB)
      CRC-32:     7.13 GB/s (done    35.65 GB)
    NILSIMSA:     7.13 GB/s (done    35.65 GB)
      WYHASH:     7.14 GB/s (done    35.71 GB)

franzomips single thread index 2.991  (quick CPU check, raw 2.991)
AL314 (A15) 1.7G (phy)  2     842.56 %
Atom N2800       (phy)  4     734.91 %
Celeron J4125 2G (phy)  4     229.73 %
Xeon E3 1245 V2  (vir)  4     123.85 %
Celeron N5105    (phy)  4     164.53 %
i5-6200U         (phy)  2     157.18 %
Xeon E5 2620 V4  (phy)  8     161.86 %
Xeon E5 2630 V4  (phy) 10     192.72 %
Xeon D-1541      (vir)  8     147.27 %
i5-3570          (phy)  4     101.19 %
i7-4790K         (phy)  4      91.61 %
i7-8700K         (phy)  6      89.07 %
AMD-Ryzen 7 3700X(phy)  8      91.86 %
i9-9900K         (phy)  8      78.08 %
i9-10900         (phy) 10      80.73 %
AMD-3950X        (phy) 16      71.90 %
AMD-5950X        (phy) 16      62.34 %
i9-12900KS 56400 (phy) 16      55.36 %
AMD-7950X3D      (phy) 16      54.22 %
65.006 seconds (00:01:05) (all OK)

[fcorbelli]

I am not sure this is Intel/AMD CPU (virtual? arm?) WARNING: non Intel/AMD CPUs should be compiled with -DNOJIT

So it worked!

It is not possible to determine exactly what type of CPU is running the code: for example, under Proxmox a generic entry appears that does not report the "real" physical CPU Therefore, I cannot do much more than WARNING, otherwise there is a risk of false positives