use ARM crypto extensions to improve ZFS encryption performance

[albertofustinoni]

System information

Type	Version/Name
Distribution Name	Armbian Buster
Distribution Version	21.05.2
Linux Kernel	5.10.34-meson64
Architecture	aarch64 (S905X3/Cortex-A55)
ZFS Version	2.0.3-1~bpo10+1
SPL Version	2.0.3-1~bpo10+1

Describe the problem you're observing

I am running ZFS on an Odroid HC4 and I'm noticing significantly worse performance with ZFS native encryption VS creating unencrypted pool over a dm-crypto block device created via LUKS.

We are talking between 40 and 70MB/s and 100% CPU usage for native crypto depending on which cypher I choose vs saturating gigabit ethernet at ~70% CPU usage when copying via Samba.

The Cortex A55 supports ARM's crypto extensions and I'm assuming dm-crypto takes advantage of them given these results

cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       477493 iterations per second for 256-bit key
PBKDF2-sha256     878204 iterations per second for 256-bit key
PBKDF2-sha512     447344 iterations per second for 256-bit key
PBKDF2-ripemd160  305529 iterations per second for 256-bit key
PBKDF2-whirlpool  114573 iterations per second for 256-bit key
argon2i       4 iterations, 477274 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 479560 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b       535.0 MiB/s       591.1 MiB/s
    serpent-cbc        128b        44.9 MiB/s        49.5 MiB/s
    twofish-cbc        128b        70.0 MiB/s        74.9 MiB/s
        aes-cbc        256b       464.2 MiB/s       547.2 MiB/s
    serpent-cbc        256b        44.9 MiB/s        49.5 MiB/s
    twofish-cbc        256b        70.0 MiB/s        74.8 MiB/s
        aes-xts        256b       566.8 MiB/s       566.2 MiB/s
    serpent-xts        256b        46.3 MiB/s        49.9 MiB/s
    twofish-xts        256b        73.6 MiB/s        76.0 MiB/s
        aes-xts        512b       526.9 MiB/s       526.4 MiB/s
    serpent-xts        512b        46.3 MiB/s        49.9 MiB/s
    twofish-xts        512b        73.6 MiB/s        76.0 MiB/s

Is this a case of ZFS not being able to do the same? Or is this related to the kernel breaking crypto acceleration for ZFS a while ago? Was the fix only for x86/AES-NI?

[rincebrain]

The only thing I can think of that might match your description (since ZFS has, to my knowledge, never used the kernel's crypto primitives, and those posts don't appear to cite anything except each other) would be when Linux decided to not play nice in the sandbox and took away the ability for non-GPL code to use SIMD, the fix for that (#9346) and #9749, which tried to reduce the additional overhead introduced by having to save and restore everything on our own.

To my knowledge, the platform-specific optimizations (for encryption) currently are indeed limited to using x86_64 instructions, though that's not an architectural restriction, just that nobody has written or ported acceleration for other platforms.

So I would not expect the aforementioned SIMD problem to have resulted in significant performance loss on aarch64, so much as there was never a highly optimized implementation running there.

If you'd like to get one, you could always try doing it yourself (and, if you're feeling nice, contributing it here) - it looks like there's some ARMv8 optimizations available in OpenSSL, which is, I believe, where the AESNI optimizations came from. (You also might want to ask @AttilaFueloep before doing that, though, in case they're already working on it in the near term, to avoid duplicated effort.)

[AttilaFueloep]

Yes, the ICP has no SIMD support for ARM. I'd suggest porting https://github.com/openssl/openssl/blob/master/crypto/modes/asm/aes-gcm-armv8_64.pl. Since I've no ARM hardware I can't do that, sorry, but I'm willing to help if someone's going to tackle that.

[albertofustinoni]

Just to have an idea of the work involved here: is it a matter of taking the Perl scripts from openssl and have them generate actual S files for the assembly compiler? Cursory glance at ZFS's aesni-gcm-x86_64.S vs OpenSSL's aesni-gcm-x86_64.pl seems to suggest so.

Poking some more around the two source trees I see a lot of pl to S file name correspondences across OpenSSL's crypto subtree and ZFS's icp one. Would doing an armv8 build of OpenSSL, which I guess would generate S files from the perl scripts, allow taking them as they are to plug into ZFS's source?

[AttilaFueloep]

is it a matter of taking the Perl scripts from openssl and have them generate actual S files for the assembly compiler

Yes, that's the first step. You can also just run run crypto/modes/asm/aes-gcm-armv8_64.pl >out.S and have a look at out.S. The next step would be to incorporate the file into the build system. Then you'd need to adjust the generated asm file since the interfaces of OpenSSL and ICP differ slightly. You can have a look here and here to see what I needed to change for x86_64. You'd also need to add the plumbing inside ICP to use the asm routine and to do so only if the required features are present (mainly module/icp/algs/modes/gcm.c). Maybe you'd also need to touch the module parameters. I'm sure I forgot some stuff but that's what I can think of right know. You could have a look at my changes to get a rough impression of what you'd have to change.

[omarkilani]

I can take a look at this and/or help out since I'm in arm64 land and was doing some testing of ZFS encryption speed there.

FWIW, testing ZFS on a ramdisk:

Device (no FS):

   READ: bw=32.4GiB/s (34.8GB/s), 32.4GiB/s-32.4GiB/s (34.8GB/s-34.8GB/s), io=1945GiB (2089GB), run=60002-60002msec
  WRITE: bw=32.4GiB/s (34.8GB/s), 32.4GiB/s-32.4GiB/s (34.8GB/s-34.8GB/s), io=1946GiB (2089GB), run=60002-60002msec

ZFS: recordsize = 8k, compression = lz4, xattr=sa, atime=off

   READ: bw=961MiB/s (1008MB/s), 961MiB/s-961MiB/s (1008MB/s-1008MB/s), io=56.3GiB (60.5GB), run=60002-60002msec
  WRITE: bw=961MiB/s (1007MB/s), 961MiB/s-961MiB/s (1007MB/s-1007MB/s), io=56.3GiB (60.4GB), run=60002-60002msec  

ZFS: recordsize = 8k, compression = lz4, xattr=sa, atime=off, encryption = on

   READ: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=16.1GiB (17.3GB), run=60002-60002msec
  WRITE: bw=274MiB/s (288MB/s), 274MiB/s-274MiB/s (288MB/s-288MB/s), io=16.1GiB (17.3GB), run=60002-60002msec

And openssl speed -evp outputs on a Ampere Alta CPU:

OpenSSL 1.1.1g FIPS  21 Apr 2020
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm     479457.19k  1353865.07k  2027970.73k  2343825.75k  2443476.99k  2457720.15k
aes-256-gcm     449805.40k  1157351.85k  1714017.54k  1955851.26k  2030267.05k  2039540.39k

[AttilaFueloep]

Yep, I'd expect some major boost from bringing in the openssl asm routines. The Intel Coffee Lake I measured on is twice as fast but that shouldn't matter much.

If you could have a look, that would be great. I'm willing to help and that would give me a reason to learn some ARM asm ;-)

[albertofustinoni]

I have zero experience with assembler, ARM or otherwise, so I wouldn't be able to do this on my own. I'm willing to help though :)

Is ZFS cross compilable? My ARM device is a single board computer with a weak CPU, so I was trying to work from my PC from Windows Subsystem for Linux/Ubuntu 20.04: I could get ZFS to compile natively (but not to run tests) but trying to cross compile with build --host aarch64-none-linux-gnu makes the build fail.

[rincebrain]

I've not tried it in the way you mean (whenever I've played with cross-compilation here, it's been with qemu-static and chroot), but I don't know of any secret sauce that would break in that case except maybe needing to explicitly specify the Linux kernel bits to compile against aren't the running kernel.

What's the build error?

[albertofustinoni]

Just tried compiling after ./configure --host aarch64-linux-gnu which complains about missing zlib-devel, even though I have the equivalent zlib1g installed.

Compiling fails with

../../libtool: fork: Invalid argument
make[3]: *** [Makefile:1325: lvm.lo] Error 254
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:708: all-recursive] Error 1
make[1]: *** [Makefile:911: all-recursive] Error 1
make: *** [Makefile:772: all] Error 2

Of interest, I'm seeing files like

  CC       algs/aes/aes_impl_aesni.lo
  CC       algs/aes/aes_impl_x86-64.lo

being compiled, which really seem out of place in an ARM build...

Using aarch64-none-linux-gnu as a host triplet seems even more out of whack, since I'm getting things like

checking for aarch64-none-linux-gnu-strip... no
checking for strip... strip

at the config stage, which to me seems like it's just using the native x86_64 toolchain.

Full outputs below:

./configure --host aarch64-linux-gnu
checking for gawk... gawk
checking metadata... git describe
checking build system type... x86_64-pc-linux-gnu
checking host system type... aarch64-unknown-linux-gnu
checking target system type... aarch64-unknown-linux-gnu
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether make supports nested variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for aarch64-linux-gnu-strip... aarch64-linux-gnu-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking whether make sets $(MAKE)... yes
checking how to print strings... printf
checking whether make supports the include directive... yes (GNU style)
checking for aarch64-linux-gnu-gcc... aarch64-linux-gnu-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether aarch64-linux-gnu-gcc accepts -g... yes
checking for aarch64-linux-gnu-gcc option to accept ISO C89... none needed
checking whether aarch64-linux-gnu-gcc understands -c and -o together... yes
checking dependency style of aarch64-linux-gnu-gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by aarch64-linux-gnu-gcc... /usr/aarch64-linux-gnu/bin/ld
checking if the linker (/usr/aarch64-linux-gnu/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/aarch64-linux-gnu-nm -B
checking the name lister (/usr/bin/aarch64-linux-gnu-nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to aarch64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/aarch64-linux-gnu/bin/ld option to reload object files... -r
checking for aarch64-linux-gnu-objdump... aarch64-linux-gnu-objdump
checking how to recognize dependent libraries... pass_all
checking for aarch64-linux-gnu-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for aarch64-linux-gnu-ar... aarch64-linux-gnu-ar
checking for archiver @FILE support... @
checking for aarch64-linux-gnu-strip... (cached) aarch64-linux-gnu-strip
checking for aarch64-linux-gnu-ranlib... aarch64-linux-gnu-ranlib
checking command to parse /usr/bin/aarch64-linux-gnu-nm -B output from aarch64-linux-gnu-gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for aarch64-linux-gnu-mt... no
checking for mt... mt
configure: WARNING: using cross tools not prefixed with host triplet
checking if mt is a manifest tool... no
checking how to run the C preprocessor... aarch64-linux-gnu-gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if aarch64-linux-gnu-gcc supports -fno-rtti -fno-exceptions... no
checking for aarch64-linux-gnu-gcc option to produce PIC... -fPIC -DPIC
checking if aarch64-linux-gnu-gcc PIC flag -fPIC -DPIC works... yes
checking if aarch64-linux-gnu-gcc static flag -static works... yes
checking if aarch64-linux-gnu-gcc supports -c -o file.o... yes
checking if aarch64-linux-gnu-gcc supports -c -o file.o... (cached) yes
checking whether the aarch64-linux-gnu-gcc linker (/usr/aarch64-linux-gnu/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for aarch64-linux-gnu-gcc... (cached) aarch64-linux-gnu-gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether aarch64-linux-gnu-gcc accepts -g... (cached) yes
checking for aarch64-linux-gnu-gcc option to accept ISO C89... (cached) none needed
checking whether aarch64-linux-gnu-gcc understands -c and -o together... (cached) yes
checking dependency style of aarch64-linux-gnu-gcc... (cached) gcc3
checking whether ln -s works... yes
checking for aarch64-linux-gnu-pkg-config... no
checking for pkg-config... no
checking dependency style of aarch64-linux-gnu-gcc... gcc3
checking whether to build with code coverage support... no
checking how to create a pax tar archive... gnutar
checking zfs author... OpenZFS
checking zfs license... CDDL
checking whether NLS is requested... yes
checking for msgfmt... /usr/bin/msgfmt
checking for gmsgfmt... /usr/bin/msgfmt
checking for xgettext... /usr/bin/xgettext
checking for msgmerge... /usr/bin/msgmerge
checking for ld... /usr/aarch64-linux-gnu/bin/ld
checking if the linker (/usr/aarch64-linux-gnu/bin/ld) is GNU ld... yes
checking for shared library run path origin... done
checking 32-bit host C ABI... no
checking for the common suffixes of directories in the library search path... lib,lib
checking zfs config... all
checking the number of available CPUs... 16
checking whether aarch64-linux-gnu-gcc supports -Wno-unused-but-set-variable... yes
checking whether aarch64-linux-gnu-gcc supports -Wno-bool-compare... yes
checking whether aarch64-linux-gnu-gcc supports -Wframe-larger-than=<size>... yes
checking whether aarch64-linux-gnu-gcc supports -Wno-format-truncation... yes
checking whether aarch64-linux-gnu-gcc supports -Wno-format-zero-length... yes
checking whether aarch64-linux-gnu-gcc supports -fno-omit-frame-pointer... yes
checking whether aarch64-linux-gnu-gcc supports -fno-ipa-sra... yes
checking whether to build with -fsanitize=address support... no
checking for system type (linux-gnu)... Linux
checking for python3... python3
checking for python version... 3.8
checking for python platform... linux
checking for python script directory... ${prefix}/lib/python3.8/site-packages
checking for python extension module directory... ${exec_prefix}/lib/python3.8/site-packages
checking for python3.8... /usr/bin/python3.8
checking for a version of Python >= '2.1.0'... yes
checking for a version of Python >= '3.4.0'... yes
checking for the distutils Python package... yes
checking for Python include path... -I/usr/include/python3.8
checking for Python library path... -L/usr/lib -lpython3.8
checking for Python site-packages path... /usr/lib/python3/dist-packages
checking python extra libraries... -lcrypt -lpthread -ldl  -lutil -lm -lm
checking python extra linking flags... -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
checking consistency of all components of python development environment... no
checking whether to enable pyzfs: ... no
checking for sed --in-place... --in-place
checking for cppcheck... no
checking for shellcheck... no
checking for checkbashisms... no
checking for CFPreferencesCopyAppValue... no
checking for CFLocaleCopyCurrent... no
checking for CFLocaleCopyPreferredLanguages... no
checking for GNU gettext in libc... yes
checking whether to use NLS... yes
checking where the gettext function comes from... libc
checking for dracut directory... /usr/lib/dracut
checking for zlib... no
checking for zlib.h... yes
checking for compress2 in -lz... no
checking for uncompress in -lz... no
checking for crc32 in -lz... no
configure: WARNING: cannot find z via pkg-config or in the standard locations
configure: error: in `/mnt/c/Users/Alberto/source/repos/zfs':
configure: error: *** zlib-devel package required
See `config.log' for more details

make -s -j8
  GEN      gitrev
Making all in include
Making all in sys
Making all in fm
Making all in fs
Making all in fs
Making all in crypto
Making all in lua
Making all in sysevent
Making all in zstd
Making all in os
Making all in linux
Making all in kernel
Making all in linux
Making all in spl
Making all in rpc
Making all in sys
Making all in zfs
Making all in sys
Making all in rpm
Making all in generic
Making all in redhat
Making all in man
Making all in man1
Making all in man5
Making all in man8
  GEN      zed.8
  GEN      zfs-mount-generator.8
Making all in scripts
Making all in lib
Making all in libavl
  CC       avl.lo
  CCLD     libavl.la
Making all in libicp
  CC       spi/kcf_spi.lo
  CC       api/kcf_ctxops.lo
  CC       api/kcf_digest.lo
  CC       api/kcf_cipher.lo
  CC       api/kcf_miscapi.lo
  CC       api/kcf_mac.lo
  CC       algs/aes/aes_impl_aesni.lo
  CC       algs/aes/aes_impl_generic.lo
  CC       algs/aes/aes_impl_x86-64.lo
  CC       algs/aes/aes_impl.lo
  CC       algs/aes/aes_modes.lo
  CC       algs/edonr/edonr.lo
  CC       algs/modes/modes.lo
  CC       algs/modes/cbc.lo
  CC       algs/modes/gcm_generic.lo
  CC       algs/modes/gcm_pclmulqdq.lo
  CC       algs/modes/gcm.lo
  CC       algs/modes/ctr.lo
  CC       algs/modes/ccm.lo
  CC       algs/modes/ecb.lo
  CC       algs/sha1/sha1.lo
  CC       algs/sha2/sha2.lo
  CC       algs/skein/skein.lo
  CC       algs/skein/skein_block.lo
  CC       algs/skein/skein_iv.lo
  CC       illumos-crypto.lo
  CC       io/aes.lo
  CC       io/edonr_mod.lo
  CC       io/sha1_mod.lo
  CC       io/sha2_mod.lo
  CC       io/skein_mod.lo
  CC       os/modhash.lo
  CC       os/modconf.lo
  CC       core/kcf_sched.lo
  CC       core/kcf_prov_lib.lo
  CC       core/kcf_callprov.lo
  CC       core/kcf_mech_tabs.lo
  CC       core/kcf_prov_tabs.lo
  CCLD     libicp.la
Making all in libshare
  CC       libshare.lo
  CC       nfs.lo
  CC       os/linux/nfs.lo
  CC       os/linux/smb.lo
  CCLD     libshare.la
copying selected object files to avoid basename conflicts...
Making all in libspl
Making all in include
Making all in ia32
Making all in sys
Making all in rpc
Making all in sys
Making all in dktp
Making all in util
Making all in os
Making all in linux
Making all in sys
  CC       assert.lo
  CC       atomic.lo
  CC       getexecname.lo
  CC       list.lo
  CC       mkdirp.lo
  CC       page.lo
  CC       strlcat.lo
  CC       strlcpy.lo
  CC       timestamp.lo
  CC       os/linux/getexecname.lo
  CC       os/linux/gethostid.lo
  CC       os/linux/getmntany.lo
  CC       os/linux/zone.lo
  CCLD     libspl_assert.la
  CCLD     libspl.la
copying selected object files to avoid basename conflicts...
Making all in libtpool
  CC       thread_pool.lo
  CCLD     libtpool.la
Making all in libzstd
  CC       lib/zstd.lo
  CC       zfs_zstd.lo
  CCLD     libzstd.la
Making all in libefi
  CC       rdwr_efi.lo
  CCLD     libefi.la
Making all in libnvpair
  CC       libnvpair.lo
  CC       libnvpair_json.lo
  CC       nvpair_alloc_system.lo
  CC       nvpair_alloc_fixed.lo
  CC       nvpair.lo
  CC       fnvpair.lo
  CCLD     libnvpair.la
Making all in libzutil
  CC       zutil_device_path.lo
  CC       zutil_import.lo
  CC       zutil_nicenum.lo
  CC       zutil_pool.lo
  CC       os/linux/zutil_device_path_os.lo
  CC       os/linux/zutil_import_os.lo
  CC       os/linux/zutil_compat.lo
  CCLD     libzutil.la
Making all in libunicode
  CC       u8_textprep.lo
  CC       uconv.lo
  CCLD     libunicode.la
Making all in libuutil
  CC       uu_alloc.lo
  CC       uu_avl.lo
  CC       uu_ident.lo
  CC       uu_list.lo
  CC       uu_misc.lo
  CC       uu_pname.lo
  CC       uu_string.lo
  CCLD     libuutil.la
Making all in libzfs_core
  CC       libzfs_core.lo
  CCLD     libzfs_core.la
Making all in libzfs
  CC       libzfs_changelist.lo
  CC       libzfs_config.lo
  CC       libzfs_crypto.lo
  CC       libzfs_dataset.lo
  CC       libzfs_diff.lo
  CC       libzfs_import.lo
  CC       libzfs_iter.lo
  CC       libzfs_mount.lo
  CC       libzfs_pool.lo
  CC       libzfs_sendrecv.lo
  CC       libzfs_status.lo
  CC       libzfs_util.lo
  CC       os/linux/libzfs_mount_os.lo
  CC       os/linux/libzfs_pool_os.lo
  CC       os/linux/libzfs_sendrecv_os.lo
  CC       os/linux/libzfs_util_os.lo
  CC       algs/sha2/sha2.lo
  CC       cityhash.lo
  CC       zfeature_common.lo
  CC       zfs_comutil.lo
  CC       zfs_deleg.lo
  CC       zfs_fletcher.lo
  CC       zfs_fletcher_aarch64_neon.lo
  CC       zfs_fletcher_avx512.lo
  CC       zfs_fletcher_intel.lo
  CC       zfs_fletcher_sse.lo
  CC       zfs_fletcher_superscalar.lo
  CC       zfs_fletcher_superscalar4.lo
  CC       zfs_namecheck.lo
  CC       zfs_prop.lo
  CC       zpool_prop.lo
  CC       zprop_common.lo
  CCLD     libzfs.la
Making all in libzpool
  CC       kernel.lo
  CC       taskq.lo
  CC       util.lo
  CC       zfeature_common.lo
  CC       zfs_comutil.lo
  CC       zfs_deleg.lo
  CC       zfs_fletcher.lo
  CC       zfs_fletcher_aarch64_neon.lo
  CC       zfs_fletcher_avx512.lo
  CC       zfs_fletcher_intel.lo
  CC       zfs_fletcher_sse.lo
  CC       zfs_fletcher_superscalar.lo
  CC       zfs_fletcher_superscalar4.lo
  CC       zfs_namecheck.lo
  CC       zfs_prop.lo
  CC       zpool_prop.lo
  CC       zprop_common.lo
  CC       abd.lo
  CC       abd_os.lo
  CC       aggsum.lo
  CC       arc.lo
  CC       arc_os.lo
  CC       blkptr.lo
  CC       bplist.lo
  CC       bpobj.lo
  CC       bptree.lo
  CC       btree.lo
  CC       bqueue.lo
  CC       cityhash.lo
  CC       dbuf.lo
  CC       dbuf_stats.lo
  CC       ddt.lo
  CC       ddt_zap.lo
  CC       dmu.lo
  CC       dmu_diff.lo
  CC       dmu_object.lo
  CC       dmu_objset.lo
  CC       dmu_recv.lo
  CC       dmu_redact.lo
  CC       dmu_send.lo
  CC       dmu_traverse.lo
  CC       dmu_tx.lo
  CC       dmu_zfetch.lo
  CC       dnode.lo
  CC       dnode_sync.lo
  CC       dsl_bookmark.lo
  CC       dsl_dataset.lo
  CC       dsl_deadlist.lo
  CC       dsl_deleg.lo
  CC       dsl_dir.lo
  CC       dsl_crypt.lo
  CC       dsl_pool.lo
  CC       dsl_prop.lo
  CC       dsl_scan.lo
  CC       dsl_synctask.lo
  CC       dsl_destroy.lo
  CC       dsl_userhold.lo
  CC       edonr_zfs.lo
  CC       hkdf.lo
  CC       fm.lo
  CC       gzip.lo
  CC       lzjb.lo
  CC       lz4.lo
  CC       metaslab.lo
  CC       mmp.lo
  CC       multilist.lo
  CC       objlist.lo
  CC       pathname.lo
  CC       range_tree.lo
  CC       refcount.lo
  CC       rrwlock.lo
  CC       sa.lo
  CC       sha256.lo
  CC       skein_zfs.lo
  CC       spa.lo
  CC       spa_boot.lo
  CC       spa_checkpoint.lo
  CC       spa_config.lo
  CC       spa_errlog.lo
  CC       spa_history.lo
  CC       spa_log_spacemap.lo
  CC       spa_misc.lo
  CC       spa_stats.lo
  CC       space_map.lo
  CC       space_reftree.lo
  CC       txg.lo
  CC       trace.lo
  CC       uberblock.lo
  CC       unique.lo
  CC       vdev.lo
  CC       vdev_cache.lo
  CC       vdev_draid.lo
  CC       vdev_draid_rand.lo
  CC       vdev_file.lo
  CC       vdev_indirect_births.lo
  CC       vdev_indirect.lo
  CC       vdev_indirect_mapping.lo
  CC       vdev_initialize.lo
  CC       vdev_label.lo
  CC       vdev_mirror.lo
  CC       vdev_missing.lo
  CC       vdev_queue.lo
  CC       vdev_raidz.lo
  CC       vdev_raidz_math_aarch64_neon.lo
  CC       vdev_raidz_math_aarch64_neonx2.lo
  CC       vdev_raidz_math_avx2.lo
  CC       vdev_raidz_math_avx512bw.lo
  CC       vdev_raidz_math_avx512f.lo
  CC       vdev_raidz_math.lo
  CC       vdev_raidz_math_scalar.lo
  CC       vdev_raidz_math_sse2.lo
  CC       vdev_raidz_math_ssse3.lo
  CC       vdev_raidz_math_powerpc_altivec.lo
  CC       vdev_rebuild.lo
  CC       vdev_removal.lo
  CC       vdev_root.lo
  CC       vdev_trim.lo
  CC       zap.lo
  CC       zap_leaf.lo
  CC       zap_micro.lo
  CC       zcp.lo
  CC       zcp_get.lo
  CC       zcp_global.lo
  CC       zcp_iter.lo
  CC       zcp_set.lo
  CC       zcp_synctask.lo
  CC       zfeature.lo
  CC       zfs_byteswap.lo
  CC       zfs_debug.lo
  CC       zfs_fm.lo
  CC       zfs_fuid.lo
  CC       zfs_racct.lo
  CC       zfs_sa.lo
  CC       zfs_znode.lo
  CC       zfs_ratelimit.lo
  CC       zfs_rlock.lo
  CC       zil.lo
  CC       zio.lo
  CC       zio_checksum.lo
  CC       zio_compress.lo
  CC       zio_crypt.lo
  CC       zio_inject.lo
  CC       zle.lo
  CC       zrlock.lo
  CC       zthr.lo
  CC       lapi.lo
  CC       lauxlib.lo
  CC       lbaselib.lo
  CC       lcode.lo
  CC       lcompat.lo
  CC       lcorolib.lo
  CC       lctype.lo
  CC       ldebug.lo
  CC       ldo.lo
  CC       lfunc.lo
  CC       lgc.lo
  CC       llex.lo
  CC       lmem.lo
  CC       lobject.lo
  CC       lopcodes.lo
  CC       lparser.lo
  CC       lstate.lo
  CC       lstring.lo
  CC       lstrlib.lo
  CC       ltable.lo
  CC       ltablib.lo
  CC       ltm.lo
  CC       lvm.lo
../../libtool: fork: Invalid argument
make[3]: *** [Makefile:1325: lvm.lo] Error 254
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:708: all-recursive] Error 1
make[1]: *** [Makefile:911: all-recursive] Error 1
make: *** [Makefile:772: all] Error 2

 ./configure --host aarch64-none-linux-gnu
checking for gawk... gawk
checking metadata... git describe
checking build system type... x86_64-pc-linux-gnu
checking host system type... aarch64-none-linux-gnu
checking target system type... aarch64-none-linux-gnu
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether make supports nested variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for aarch64-none-linux-gnu-strip... no
checking for strip... strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking whether make sets $(MAKE)... yes
checking how to print strings... printf
checking whether make supports the include directive... yes (GNU style)
checking for aarch64-none-linux-gnu-gcc... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... no
checking for aarch64-none-linux-gnu-dumpbin... no
checking for aarch64-none-linux-gnu-link... no
checking for dumpbin... no
checking for link... link -dump
checking the name lister (nm) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to aarch64-none-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for aarch64-none-linux-gnu-objdump... no
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for aarch64-none-linux-gnu-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for aarch64-none-linux-gnu-ar... no
checking for ar... ar
checking for archiver @FILE support... @
checking for aarch64-none-linux-gnu-strip... strip
checking for aarch64-none-linux-gnu-ranlib... no
checking for ranlib... ranlib
checking command to parse nm output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for aarch64-none-linux-gnu-mt... no
checking for mt... mt
checking if mt is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for aarch64-none-linux-gnu-gcc... gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking whether ln -s works... yes
checking for aarch64-none-linux-gnu-pkg-config... no
checking for pkg-config... no
checking dependency style of gcc... gcc3
checking whether to build with code coverage support... no
checking how to create a pax tar archive... gnutar
checking zfs author... OpenZFS
checking zfs license... CDDL
checking whether NLS is requested... yes
checking for msgfmt... /usr/bin/msgfmt
checking for gmsgfmt... /usr/bin/msgfmt
checking for xgettext... /usr/bin/xgettext
checking for msgmerge... /usr/bin/msgmerge
checking for ld... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for shared library run path origin... done
checking 32-bit host C ABI... yes
checking for the common suffixes of directories in the library search path... lib,lib
checking zfs config... all
checking the number of available CPUs... 16
checking whether gcc supports -Wno-unused-but-set-variable... yes
checking whether gcc supports -Wno-bool-compare... yes
checking whether gcc supports -Wframe-larger-than=<size>... yes
checking whether gcc supports -Wno-format-truncation... yes
checking whether gcc supports -Wno-format-zero-length... yes
checking whether gcc supports -fno-omit-frame-pointer... yes
checking whether gcc supports -fno-ipa-sra... yes
checking whether to build with -fsanitize=address support... no
checking for system type (linux-gnu)... Linux
checking for python3... python3
checking for python version... 3.8
checking for python platform... linux
checking for python script directory... ${prefix}/lib/python3.8/site-packages
checking for python extension module directory... ${exec_prefix}/lib/python3.8/site-packages
checking for python3.8... /usr/bin/python3.8
checking for a version of Python >= '2.1.0'... yes
checking for a version of Python >= '3.4.0'... yes
checking for the distutils Python package... yes
checking for Python include path... -I/usr/include/python3.8
checking for Python library path... -L/usr/lib -lpython3.8
checking for Python site-packages path... /usr/lib/python3/dist-packages
checking python extra libraries... -lcrypt -lpthread -ldl  -lutil -lm -lm
checking python extra linking flags... -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
checking consistency of all components of python development environment... yes
checking for python3.8 module: setuptools... yes
checking for python3.8 module: cffi... yes
checking whether to enable pyzfs: ... yes
checking for sed --in-place... --in-place
checking for cppcheck... no
checking for shellcheck... no
checking for checkbashisms... no
checking for CFPreferencesCopyAppValue... no
checking for CFLocaleCopyCurrent... no
checking for CFLocaleCopyPreferredLanguages... no
checking for GNU gettext in libc... yes
checking whether to use NLS... yes
checking where the gettext function comes from... libc
checking for dracut directory... /usr/lib/dracut
checking for zlib... no
checking for zlib.h... yes
checking for compress2 in -lz... yes
checking for uncompress in -lz... yes
checking for crc32 in -lz... yes
checking for udev directories... /lib/udev;/lib/udev/rules.d
checking for systemd support... yes
checking for uuid... no
checking for uuid/uuid.h... yes
checking for uuid_generate in -luuid... yes
checking for uuid_is_null in -luuid... yes
checking for blkid... no
checking for blkid/blkid.h... yes
checking for main in -lblkid... yes
checking for library containing xdrmem_create... none required
checking for libudev... no
checking for libudev.h... yes
checking for main in -ludev... yes
checking for udev_device_get_is_initialized... yes
checking for libcrypto... no
checking for openssl/evp.h... yes
checking for PKCS5_PBKDF2_HMAC_SHA1 in -lcrypto... yes
checking for libaio.h... yes
checking for main in -laio... yes
checking whether -latomic is present... yes
checking for libfetch... none
checking for clock_gettime... yes
checking security/pam_modules.h usability... no
checking security/pam_modules.h presence... no
checking for security/pam_modules.h... no
checking makedev() is declared in sys/sysmacros.h... yes
checking makedev() is declared in sys/mkdev.h... no
checking for issetugid... no
checking for mlockall... yes
checking for strlcat... no
checking for strlcpy... no
checking kernel source directory... /usr/src/linux-headers-5.4.0-74-generic
checking kernel build directory... /usr/src/linux-headers-5.4.0-74-generic
checking kernel source version... 5.4.0-74-generic
checking kernel file name for module symbols... Module.symvers
checking whether modules can be built... yes
checking for kernel config option compatibility... done
checking whether kernel was built with 16K or larger stacks... yes
checking whether mutex_lock() is GPL-only... no
checking whether CONFIG_TRIM_UNUSED_KSYM is disabled... yes
checking whether CONFIG_ZLIB_INFLATE is defined... yes
checking whether CONFIG_ZLIB_DEFLATE is defined... yes
checking whether fpu headers are available... asm/fpu/api.h
checking whether objtool header is available... linux/frame.h
checking whether wait_queue_entry_t exists... yes
checking whether /dev/zfs minor is available... 249
checking whether DECLARE_EVENT_CLASS() is available... no
checking for available kernel interfaces... done
checking whether access_ok() has 'type' parameter... no
checking whether global_node_page_state() exists... yes
checking whether global_zone_page_state() exists... yes
checking whether enum node_stat_item contains NR_FILE_PAGES... yes
checking whether enum node_stat_item contains NR_INACTIVE_ANON... yes
checking whether enum node_stat_item contains NR_INACTIVE_FILE... yes
checking whether enum zone_stat_item contains NR_FILE_PAGES... no
checking whether enum zone_stat_item contains NR_INACTIVE_ANON... no
checking whether enum zone_stat_item contains NR_INACTIVE_FILE... no
checking whether global_page_state enums are sane... yes
checking whether compile-time stack validation (objtool) is available... yes
checking whether STACK_FRAME_NON_STANDARD is defined... yes
checking whether PDE_DATA() is available... yes
checking whether fops->fallocate() exists... yes
checking whether zlib_deflate_workspacesize() wants 2 args... yes
checking whether struct rw_semaphore has member activity... no
checking whether struct rw_semaphore has atomic_long_t member count... yes
checking whether header linux/sched/rt.h exists... yes
checking whether header linux/sched/signal.h exists... yes
checking whether io_schedule_timeout() is available... yes
checking whether usleep_range() is available... yes
checking whether kmem_cache_create_usercopy() exists... yes
checking whether kvmalloc(ptr, flags) is available... yes
checking whether __vmalloc(ptr, flags, pageflags) is available... yes
checking whether wait_on_bit() takes an action... no
checking whether wq_head->head and wq_entry->entry exist... yes
checking whether timestamp_truncate() exists... yes
checking whether inode->i_*time's are timespec64... yes
checking whether inode_lock_shared() exists... yes
checking whether group_info->gid exists... yes
checking whether kernel_write() takes loff_t pointer... yes
checking whether kernel_read() takes loff_t pointer... yes
checking whether timer_setup() is available... yes
checking whether timer function expects timer_list... yes
checking whether struct timer_list has flags... yes
checking whether super_block->s_user_ns exists... yes
checking whether proc_ops structure exists... no
checking whether bops->check_events() exists... yes
checking whether bops->release() is void... yes
checking whether bops->revalidate_disk() exists... yes
checking whether REQ_FAILFAST_MASK is defined... yes
checking whether REQ_DISCARD is defined... no
checking whether REQ_FLUSH is defined... no
checking whether REQ_PREFLUSH is defined... yes
checking whether REQ_OP_DISCARD is defined... yes
checking whether REQ_OP_SECURE_ERASE is defined... yes
checking whether REQ_OP_FLUSH is defined... yes
checking whether bio->bi_opf is defined... yes
checking whether bio_set_op_attrs is available... yes
checking whether bio_set_dev() is available... yes
checking whether bio_set_dev() is GPL-only... yes
checking whether bio_end_io_t wants 1 arg... yes
checking whether bio->bi_status exists... yes
checking whether bio has bi_iter... yes
checking whether submit_bio() wants 1 arg... yes
checking whether current->bio_list exists... yes
checking whether blkg_tryget() is available... yes
checking whether blkg_tryget() is GPL-only... no
checking whether bio->bi_bdev->bd_disk exists... no
checking whether blkdev_get_by_path() exists... yes
checking whether blkdev_put() exists... yes
checking whether blkdev_reread_part() exists... yes
checking whether invalidate_bdev() exists... yes
checking whether lookup_bdev() wants dev_t arg... no
checking whether lookup_bdev() wants 1 arg... yes
checking whether bdev_logical_block_size() is available... yes
checking whether bdev_physical_block_size() is available... yes
checking whether check_disk_change() exists... yes
checking whether bdev_disk_changed() exists... no
checking whether bdev_whole() is available... no
checking whether struct blk_plug is available... yes
checking whether blk_queue bdi is dynamic... yes
checking whether blk_queue_discard() is available... yes
checking whether blk_queue_secure_erase() is available... yes
checking whether blk_queue_flag_set() exists... yes
checking whether blk_queue_flag_clear() exists... yes
checking whether blk_queue_flush() is available... no
checking whether blk_queue_write_cache() exists... yes
checking whether blk_queue_write_cache() is GPL-only... yes
checking whether blk_queue_max_hw_sectors() is available... yes
checking whether blk_queue_max_segments() is available... yes
checking whether revalidate_disk_size() is available... no
checking whether revalidate_disk() is available... yes
checking whether get_disk_ro() is available... yes
checking whether generic_readlink is global... no
checking whether ql->discard_granularity is available... yes
checking whether inode_owner_or_capable() exists... yes
checking whether super_block uses const struct xattr_handler... yes
checking whether xattr_handler has name... yes
checking whether xattr_handler->get() wants dentry and inode... yes
checking whether xattr_handler->set() wants dentry, inode, and user_namespace... no
checking whether xattr_handler->set() wants dentry and inode... yes
checking whether xattr_handler->list() wants simple... yes
checking whether posix_acl_from_xattr() needs user_ns... yes
checking whether generic_setxattr() exists... no
checking whether posix_acl_release() is available... yes
checking whether posix_acl_release() is GPL-only... yes
checking whether set_cached_acl() is usable... yes
checking whether __posix_acl_chmod exists... yes
checking whether posix_acl_equiv_mode() wants umode_t... yes
checking whether posix_acl_valid() wants user namespace... yes
checking whether iops->get_acl() exists... yes
checking whether iops->set_acl() exists... yes
checking whether uncached_acl_sentinel() exists... yes
checking whether posix_acl has refcount_t... yes
checking whether iops->getattr() takes user_namespace... no
checking whether iops->getattr() takes a path... yes
checking whether inode_set_flags() exists... yes
checking whether inode_set_iversion() exists... yes
checking whether sops->show_options() wants dentry... yes
checking whether file_inode() is available... yes
checking whether file_dentry() is available... yes
checking whether fops->fsync() wants no dentry... no
checking whether fops->fsync() wants range... range
checking whether fops->aio_fsync() exists... no
checking whether sops->evict_inode() exists... yes
checking whether sops->dirty_inode() wants flags... yes
checking whether super_block has s_shrink... yes
checking whether shrink_control has nid... yes
checking whether new 2-argument shrinker exists... no
checking whether ->count_objects callback exists... yes
checking whether struct shrink_control exists... yes
checking whether iops->mkdir() takes struct user_namespace*... checking whether iops->mkdir() takes umode_t... yes
checking whether iops->lookup() passes flags... yes
checking whether iops->create() takes struct user_namespace*... no
checking whether iops->create() passes flags... yes
checking whether iops->get_link() passes delayed... yes
checking whether i_op->tmpfile() exists... yes
checking whether dops->d_automount() exists... yes
checking whether eops->encode_fh() wants inode... yes
checking whether eops->commit_metadata() exists... yes
checking whether clear_inode() is available... yes
checking whether setattr_prepare() is available and accepts struct user_namespace*... no
checking whether setattr_prepare() is available, doesn't accept user_namespace... yes
checking whether insert_inode_locked() is available... yes
checking whether d_make_root() is available... yes
checking whether d_obtain_alias() is available... yes
checking whether d_prune_aliases() is available... yes
checking whether d_set_d_op() is available... yes
checking whether dops->d_revalidate() takes struct nameidata... no
checking whether dentry uses const struct dentry_operations... yes
checking whether super_block has s_d_op... yes
checking whether truncate_setsize() is available... yes
checking whether security_inode_init_security wants callback... yes
checking whether fst->mount() exists... yes
checking whether super_setup_bdi_name() exists... yes
checking whether set_nlink() is available... yes
checking whether sget() wants 5 args... yes
checking whether lseek_execute() is available... no
checking whether vfs_getattr() wants 4 args... yes
checking whether vfs_getattr() wants 2 args... no
checking whether vfs_getattr() wants 3 args... no
checking whether vfs_fsync() wants 2 args... yes
checking whether fops->iterate_shared() is available... yes
checking whether aops->direct_IO() uses iov_iter... yes
checking whether fops->read/write_iter() are available... yes
checking whether new_sync_read/write() are available... no
checking whether generic_write_checks() takes kiocb... yes
checking whether iov_iter types are available... yes
checking whether iov_iter_advance() is available... yes
checking whether iov_iter_revert() is available... yes
checking whether iov_iter_fault_in_readable() is available... yes
checking whether iov_iter_count() is available... yes
checking whether copy_to_iter() is available... yes
checking whether copy_from_iter() is available... yes
checking whether kmap_atomic wants 1 args... yes
checking whether follow_down_one() is available... yes
checking whether submit_bio is member of struct block_device_operations... no
checking whether blk_alloc_queue() expects request function... no
checking whether make_request_fn() returns void... no
checking whether make_request_fn() returns blk_qc_t... yes
checking whether generic disk_*_io_acct() are available... no
checking whether generic bio_*_io_acct() are available... no
checking whether generic_*_io_acct wants 4 args... yes
checking whether kernel fpu is available... internal
checking whether kernel defines fmode_t... yes
checking whether kuid_t/kgid_t is available... yes
checking whether i_(uid|gid)_(read|write) exist... yes
checking whether module_param_call() is hardened... yes
checking whether iops->rename() takes struct user_namespace*... no
checking whether iop->rename() wants flags... yes
checking whether current_time() exists... yes
checking whether ns_capable exists... yes
checking whether has_capability() is available... yes
checking whether cred_t->user_ns exists... yes
checking whether kuid_has_mapping/kgid_has_mapping exist... yes
checking whether in_compat_syscall() is available... yes
checking whether ktime_get_coarse_real_ts64() exists... yes
checking whether ktime_get_raw_ts64() exists... yes
checking whether totalram_pages() exists... yes
checking whether totalhigh_pages() exists... yes
checking whether kstrtoul() exists... yes
checking whether percpu_counter_init() wants gfp_t... yes
checking whether percpu_counter_add_batch() is defined... yes
checking whether is inside percpu_ref.data... no
checking whether CPU hotplug APIs exist... yes
checking whether generic_fillattr requires struct user_namespace*... no
checking whether iops->mknod() takes struct user_namespace*... no
checking whether iops->symlink() takes struct user_namespace*... no
checking whether bio_max_segs() exists... no
checking whether signal_stop() exists... yes
checking whether kernel_siginfo_t tyepedef exists... yes
checking whether set_special_state() exists... yes
checking os distribution... ubuntu
checking default package type... deb
checking default init directory... ${prefix}/etc/init.d
checking default init script type and shell... lsb:/bin/sh
checking default nfs server init script... nfs
checking default init config directory... /etc/default
checking whether initramfs-tools is available... yes
checking whether rpm is available... yes (4.14.2.1)
checking whether rpmbuild is available... yes (4.14.2.1)
checking whether spec files are available... yes (rpm/generic/*.spec.in)
checking whether dpkg is available... yes (1.19.7)
checking whether dpkg-buildpackage is available... yes (1.19.7.)
checking whether alien is available... yes (8.95)
checking whether assertion support will be enabled... no
checking whether debuginfo support will be forced... no
checking whether basic kmem accounting is enabled... no
checking whether detailed kmem tracking is enabled... no
checking whether FreeBSD kernel INVARIANTS checks are enabled... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating cmd/Makefile
config.status: creating cmd/arc_summary/Makefile
config.status: creating cmd/arcstat/Makefile
config.status: creating cmd/dbufstat/Makefile
config.status: creating cmd/fsck_zfs/Makefile
config.status: creating cmd/mount_zfs/Makefile
config.status: creating cmd/raidz_test/Makefile
config.status: creating cmd/vdev_id/Makefile
config.status: creating cmd/zdb/Makefile
config.status: creating cmd/zed/Makefile
config.status: creating cmd/zed/zed.d/Makefile
config.status: creating cmd/zfs/Makefile
config.status: creating cmd/zfs_ids_to_path/Makefile
config.status: creating cmd/zgenhostid/Makefile
config.status: creating cmd/zhack/Makefile
config.status: creating cmd/zinject/Makefile
config.status: creating cmd/zpool/Makefile
config.status: creating cmd/zstream/Makefile
config.status: creating cmd/ztest/Makefile
config.status: creating cmd/zvol_id/Makefile
config.status: creating cmd/zvol_wait/Makefile
config.status: creating cmd/zpool_influxdb/Makefile
config.status: creating contrib/Makefile
config.status: creating contrib/bash_completion.d/Makefile
config.status: creating contrib/bpftrace/Makefile
config.status: creating contrib/dracut/02zfsexpandknowledge/Makefile
config.status: creating contrib/dracut/90zfs/Makefile
config.status: creating contrib/dracut/Makefile
config.status: creating contrib/initramfs/Makefile
config.status: creating contrib/initramfs/conf.d/Makefile
config.status: creating contrib/initramfs/conf-hooks.d/Makefile
config.status: creating contrib/initramfs/hooks/Makefile
config.status: creating contrib/initramfs/scripts/Makefile
config.status: creating contrib/initramfs/scripts/local-top/Makefile
config.status: creating contrib/pam_zfs_key/Makefile
config.status: creating contrib/pyzfs/Makefile
config.status: creating contrib/pyzfs/setup.py
config.status: creating contrib/zcp/Makefile
config.status: creating etc/Makefile
config.status: creating etc/default/Makefile
config.status: creating etc/init.d/Makefile
config.status: creating etc/modules-load.d/Makefile
config.status: creating etc/sudoers.d/Makefile
config.status: creating etc/systemd/Makefile
config.status: creating etc/systemd/system-generators/Makefile
config.status: creating etc/systemd/system/Makefile
config.status: creating etc/zfs/Makefile
config.status: creating include/Makefile
config.status: creating include/os/Makefile
config.status: creating include/os/freebsd/Makefile
config.status: creating include/os/freebsd/linux/Makefile
config.status: creating include/os/freebsd/spl/Makefile
config.status: creating include/os/freebsd/spl/acl/Makefile
config.status: creating include/os/freebsd/spl/rpc/Makefile
config.status: creating include/os/freebsd/spl/sys/Makefile
config.status: creating include/os/freebsd/zfs/Makefile
config.status: creating include/os/freebsd/zfs/sys/Makefile
config.status: creating include/os/linux/Makefile
config.status: creating include/os/linux/kernel/Makefile
config.status: creating include/os/linux/kernel/linux/Makefile
config.status: creating include/os/linux/spl/Makefile
config.status: creating include/os/linux/spl/rpc/Makefile
config.status: creating include/os/linux/spl/sys/Makefile
config.status: creating include/os/linux/zfs/Makefile
config.status: creating include/os/linux/zfs/sys/Makefile
config.status: creating include/sys/Makefile
config.status: creating include/sys/crypto/Makefile
config.status: creating include/sys/fm/Makefile
config.status: creating include/sys/fm/fs/Makefile
config.status: creating include/sys/fs/Makefile
config.status: creating include/sys/lua/Makefile
config.status: creating include/sys/sysevent/Makefile
config.status: creating include/sys/zstd/Makefile
config.status: creating lib/Makefile
config.status: creating lib/libavl/Makefile
config.status: creating lib/libefi/Makefile
config.status: creating lib/libicp/Makefile
config.status: creating lib/libnvpair/Makefile
config.status: creating lib/libshare/Makefile
config.status: creating lib/libspl/Makefile
config.status: creating lib/libspl/include/Makefile
config.status: creating lib/libspl/include/ia32/Makefile
config.status: creating lib/libspl/include/ia32/sys/Makefile
config.status: creating lib/libspl/include/os/Makefile
config.status: creating lib/libspl/include/os/freebsd/Makefile
config.status: creating lib/libspl/include/os/freebsd/sys/Makefile
config.status: creating lib/libspl/include/os/linux/Makefile
config.status: creating lib/libspl/include/os/linux/sys/Makefile
config.status: creating lib/libspl/include/rpc/Makefile
config.status: creating lib/libspl/include/sys/Makefile
config.status: creating lib/libspl/include/sys/dktp/Makefile
config.status: creating lib/libspl/include/util/Makefile
config.status: creating lib/libtpool/Makefile
config.status: creating lib/libunicode/Makefile
config.status: creating lib/libuutil/Makefile
config.status: creating lib/libzfs/Makefile
config.status: creating lib/libzfs/libzfs.pc
config.status: creating lib/libzfsbootenv/Makefile
config.status: creating lib/libzfsbootenv/libzfsbootenv.pc
config.status: creating lib/libzfs_core/Makefile
config.status: creating lib/libzfs_core/libzfs_core.pc
config.status: creating lib/libzpool/Makefile
config.status: creating lib/libzstd/Makefile
config.status: creating lib/libzutil/Makefile
config.status: creating man/Makefile
config.status: creating man/man1/Makefile
config.status: creating man/man5/Makefile
config.status: creating man/man8/Makefile
config.status: creating module/Kbuild
config.status: creating module/Makefile
config.status: creating module/avl/Makefile
config.status: creating module/icp/Makefile
config.status: creating module/lua/Makefile
config.status: creating module/nvpair/Makefile
config.status: creating module/os/linux/spl/Makefile
config.status: creating module/os/linux/zfs/Makefile
config.status: creating module/spl/Makefile
config.status: creating module/unicode/Makefile
config.status: creating module/zcommon/Makefile
config.status: creating module/zfs/Makefile
config.status: creating module/zstd/Makefile
config.status: creating rpm/Makefile
config.status: creating rpm/generic/Makefile
config.status: creating rpm/generic/zfs-dkms.spec
config.status: creating rpm/generic/zfs-kmod.spec
config.status: creating rpm/generic/zfs.spec
config.status: creating rpm/redhat/Makefile
config.status: creating rpm/redhat/zfs-dkms.spec
config.status: creating rpm/redhat/zfs-kmod.spec
config.status: creating rpm/redhat/zfs.spec
config.status: creating scripts/Makefile
config.status: creating tests/Makefile
config.status: creating tests/runfiles/Makefile
config.status: creating tests/test-runner/Makefile
config.status: creating tests/test-runner/bin/Makefile
config.status: creating tests/test-runner/include/Makefile
config.status: creating tests/test-runner/man/Makefile
config.status: creating tests/zfs-tests/Makefile
config.status: creating tests/zfs-tests/callbacks/Makefile
config.status: creating tests/zfs-tests/cmd/Makefile
config.status: creating tests/zfs-tests/cmd/badsend/Makefile
config.status: creating tests/zfs-tests/cmd/btree_test/Makefile
config.status: creating tests/zfs-tests/cmd/chg_usr_exec/Makefile
config.status: creating tests/zfs-tests/cmd/devname2devid/Makefile
config.status: creating tests/zfs-tests/cmd/draid/Makefile
config.status: creating tests/zfs-tests/cmd/dir_rd_update/Makefile
config.status: creating tests/zfs-tests/cmd/file_check/Makefile
config.status: creating tests/zfs-tests/cmd/file_trunc/Makefile
config.status: creating tests/zfs-tests/cmd/file_write/Makefile
config.status: creating tests/zfs-tests/cmd/get_diff/Makefile
config.status: creating tests/zfs-tests/cmd/largest_file/Makefile
config.status: creating tests/zfs-tests/cmd/libzfs_input_check/Makefile
config.status: creating tests/zfs-tests/cmd/mkbusy/Makefile
config.status: creating tests/zfs-tests/cmd/mkfile/Makefile
config.status: creating tests/zfs-tests/cmd/mkfiles/Makefile
config.status: creating tests/zfs-tests/cmd/mktree/Makefile
config.status: creating tests/zfs-tests/cmd/mmap_exec/Makefile
config.status: creating tests/zfs-tests/cmd/mmap_libaio/Makefile
config.status: creating tests/zfs-tests/cmd/mmapwrite/Makefile
config.status: creating tests/zfs-tests/cmd/nvlist_to_lua/Makefile
config.status: creating tests/zfs-tests/cmd/randfree_file/Makefile
config.status: creating tests/zfs-tests/cmd/randwritecomp/Makefile
config.status: creating tests/zfs-tests/cmd/readmmap/Makefile
config.status: creating tests/zfs-tests/cmd/rename_dir/Makefile
config.status: creating tests/zfs-tests/cmd/rm_lnkcnt_zero_file/Makefile
config.status: creating tests/zfs-tests/cmd/send_doall/Makefile
config.status: creating tests/zfs-tests/cmd/stride_dd/Makefile
config.status: creating tests/zfs-tests/cmd/threadsappend/Makefile
config.status: creating tests/zfs-tests/cmd/user_ns_exec/Makefile
config.status: creating tests/zfs-tests/cmd/xattrtest/Makefile
config.status: creating tests/zfs-tests/include/Makefile
config.status: creating tests/zfs-tests/tests/Makefile
config.status: creating tests/zfs-tests/tests/functional/Makefile
config.status: creating tests/zfs-tests/tests/functional/acl/Makefile
config.status: creating tests/zfs-tests/tests/functional/acl/off/Makefile
config.status: creating tests/zfs-tests/tests/functional/acl/posix/Makefile
config.status: creating tests/zfs-tests/tests/functional/acl/posix-sa/Makefile
config.status: creating tests/zfs-tests/tests/functional/alloc_class/Makefile
config.status: creating tests/zfs-tests/tests/functional/arc/Makefile
config.status: creating tests/zfs-tests/tests/functional/atime/Makefile
config.status: creating tests/zfs-tests/tests/functional/bootfs/Makefile
config.status: creating tests/zfs-tests/tests/functional/btree/Makefile
config.status: creating tests/zfs-tests/tests/functional/cache/Makefile
config.status: creating tests/zfs-tests/tests/functional/cachefile/Makefile
config.status: creating tests/zfs-tests/tests/functional/casenorm/Makefile
config.status: creating tests/zfs-tests/tests/functional/channel_program/Makefile
config.status: creating tests/zfs-tests/tests/functional/channel_program/lua_core/Makefile
config.status: creating tests/zfs-tests/tests/functional/channel_program/synctask_core/Makefile
config.status: creating tests/zfs-tests/tests/functional/chattr/Makefile
config.status: creating tests/zfs-tests/tests/functional/checksum/Makefile
config.status: creating tests/zfs-tests/tests/functional/clean_mirror/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zdb/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_bookmark/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_change-key/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_clone/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_copies/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_create/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_destroy/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_diff/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_get/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_ids_to_path/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_inherit/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_jail/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_load-key/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_mount/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_program/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_promote/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_property/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_receive/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_rename/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_reservation/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_rollback/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_send/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_set/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_share/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_snapshot/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_sysfs/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_unload-key/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_unmount/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_unshare/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_upgrade/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zfs_wait/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_add/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_attach/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_clear/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_create/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_destroy/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_detach/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_events/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_expand/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_export/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_get/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_history/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_import/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_import/blockfiles/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_initialize/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_labelclear/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_offline/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_online/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_remove/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_reopen/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_replace/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_resilver/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_scrub/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_set/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_split/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_status/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_sync/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_trim/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_upgrade/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_upgrade/blockfiles/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_wait/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_root/zpool_wait/scan/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_user/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_user/misc/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_user/zfs_list/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_user/zpool_iostat/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_user/zpool_list/Makefile
config.status: creating tests/zfs-tests/tests/functional/cli_user/zpool_status/Makefile
config.status: creating tests/zfs-tests/tests/functional/compression/Makefile
config.status: creating tests/zfs-tests/tests/functional/cp_files/Makefile
config.status: creating tests/zfs-tests/tests/functional/ctime/Makefile
config.status: creating tests/zfs-tests/tests/functional/deadman/Makefile
config.status: creating tests/zfs-tests/tests/functional/delegate/Makefile
config.status: creating tests/zfs-tests/tests/functional/devices/Makefile
config.status: creating tests/zfs-tests/tests/functional/events/Makefile
config.status: creating tests/zfs-tests/tests/functional/exec/Makefile
config.status: creating tests/zfs-tests/tests/functional/fallocate/Makefile
config.status: creating tests/zfs-tests/tests/functional/fault/Makefile
config.status: creating tests/zfs-tests/tests/functional/features/Makefile
config.status: creating tests/zfs-tests/tests/functional/features/async_destroy/Makefile
config.status: creating tests/zfs-tests/tests/functional/features/large_dnode/Makefile
config.status: creating tests/zfs-tests/tests/functional/grow/Makefile
config.status: creating tests/zfs-tests/tests/functional/history/Makefile
config.status: creating tests/zfs-tests/tests/functional/hkdf/Makefile
config.status: creating tests/zfs-tests/tests/functional/inheritance/Makefile
config.status: creating tests/zfs-tests/tests/functional/inuse/Makefile
config.status: creating tests/zfs-tests/tests/functional/io/Makefile
config.status: creating tests/zfs-tests/tests/functional/l2arc/Makefile
config.status: creating tests/zfs-tests/tests/functional/large_files/Makefile
config.status: creating tests/zfs-tests/tests/functional/largest_pool/Makefile
config.status: creating tests/zfs-tests/tests/functional/libzfs/Makefile
config.status: creating tests/zfs-tests/tests/functional/limits/Makefile
config.status: creating tests/zfs-tests/tests/functional/link_count/Makefile
config.status: creating tests/zfs-tests/tests/functional/log_spacemap/Makefile
config.status: creating tests/zfs-tests/tests/functional/migration/Makefile
config.status: creating tests/zfs-tests/tests/functional/mmap/Makefile
config.status: creating tests/zfs-tests/tests/functional/mmp/Makefile
config.status: creating tests/zfs-tests/tests/functional/mount/Makefile
config.status: creating tests/zfs-tests/tests/functional/mv_files/Makefile
config.status: creating tests/zfs-tests/tests/functional/nestedfs/Makefile
config.status: creating tests/zfs-tests/tests/functional/no_space/Makefile
config.status: creating tests/zfs-tests/tests/functional/nopwrite/Makefile
config.status: creating tests/zfs-tests/tests/functional/online_offline/Makefile
config.status: creating tests/zfs-tests/tests/functional/pam/Makefile
config.status: creating tests/zfs-tests/tests/functional/pool_checkpoint/Makefile
config.status: creating tests/zfs-tests/tests/functional/pool_names/Makefile
config.status: creating tests/zfs-tests/tests/functional/poolversion/Makefile
config.status: creating tests/zfs-tests/tests/functional/privilege/Makefile
config.status: creating tests/zfs-tests/tests/functional/procfs/Makefile
config.status: creating tests/zfs-tests/tests/functional/projectquota/Makefile
config.status: creating tests/zfs-tests/tests/functional/pyzfs/Makefile
config.status: creating tests/zfs-tests/tests/functional/quota/Makefile
config.status: creating tests/zfs-tests/tests/functional/raidz/Makefile
config.status: creating tests/zfs-tests/tests/functional/redacted_send/Makefile
config.status: creating tests/zfs-tests/tests/functional/redundancy/Makefile
config.status: creating tests/zfs-tests/tests/functional/refquota/Makefile
config.status: creating tests/zfs-tests/tests/functional/refreserv/Makefile
config.status: creating tests/zfs-tests/tests/functional/removal/Makefile
config.status: creating tests/zfs-tests/tests/functional/rename_dirs/Makefile
config.status: creating tests/zfs-tests/tests/functional/replacement/Makefile
config.status: creating tests/zfs-tests/tests/functional/reservation/Makefile
config.status: creating tests/zfs-tests/tests/functional/rootpool/Makefile
config.status: creating tests/zfs-tests/tests/functional/rsend/Makefile
config.status: creating tests/zfs-tests/tests/functional/scrub_mirror/Makefile
config.status: creating tests/zfs-tests/tests/functional/slog/Makefile
config.status: creating tests/zfs-tests/tests/functional/snapshot/Makefile
config.status: creating tests/zfs-tests/tests/functional/snapused/Makefile
config.status: creating tests/zfs-tests/tests/functional/sparse/Makefile
config.status: creating tests/zfs-tests/tests/functional/suid/Makefile
config.status: creating tests/zfs-tests/tests/functional/threadsappend/Makefile
config.status: creating tests/zfs-tests/tests/functional/tmpfile/Makefile
config.status: creating tests/zfs-tests/tests/functional/trim/Makefile
config.status: creating tests/zfs-tests/tests/functional/truncate/Makefile
config.status: creating tests/zfs-tests/tests/functional/upgrade/Makefile
config.status: creating tests/zfs-tests/tests/functional/user_namespace/Makefile
config.status: creating tests/zfs-tests/tests/functional/userquota/Makefile
config.status: creating tests/zfs-tests/tests/functional/vdev_zaps/Makefile
config.status: creating tests/zfs-tests/tests/functional/write_dirs/Makefile
config.status: creating tests/zfs-tests/tests/functional/xattr/Makefile
config.status: creating tests/zfs-tests/tests/functional/zpool_influxdb/Makefile
config.status: creating tests/zfs-tests/tests/functional/zvol/Makefile
config.status: creating tests/zfs-tests/tests/functional/zvol/zvol_ENOSPC/Makefile
config.status: creating tests/zfs-tests/tests/functional/zvol/zvol_cli/Makefile
config.status: creating tests/zfs-tests/tests/functional/zvol/zvol_misc/Makefile
config.status: creating tests/zfs-tests/tests/functional/zvol/zvol_swap/Makefile
config.status: creating tests/zfs-tests/tests/perf/Makefile
config.status: creating tests/zfs-tests/tests/perf/fio/Makefile
config.status: creating tests/zfs-tests/tests/perf/regression/Makefile
config.status: creating tests/zfs-tests/tests/perf/scripts/Makefile
config.status: creating tests/zfs-tests/tests/stress/Makefile
config.status: creating udev/Makefile
config.status: creating udev/rules.d/Makefile
config.status: creating zfs.release
config.status: creating zfs_config.h
config.status: executing depfiles commands
config.status: executing libtool commands
config.status: executing po-directories commands

[AttilaFueloep]

Sorry, can't help with the cross compiling issues since I never did that, but I'd look at the --with-sysroot, --with-linux and --with-linux-obj configure options.

Of interest, I'm seeing files like

  CC       algs/aes/aes_impl_aesni.lo
  CC       algs/aes/aes_impl_x86-64.lo

being compiled, which really seem out of place in an ARM build...

Those files are bracketed with #if defined(__x86_64) so no code will be generated.

[bghira]

for raspberry pi architecture, cross compiling can be done via this guide: https://medium.com/@au42/the-useful-raspberrypi-cross-compile-guide-ea56054de187

[albertofustinoni]

@bghira That guide is not really relevant to this project: to start with zfs does not use CMake (unfortunately)

I actually made some inroads in getting zfs to cross compile on my system.

First of all one needs to install the gnu compilers for ARM64 like so: sudo apt install crossbuild-essential-arm64

Then, and this is the bit that was stumping me, one needs to install the ARM specific versions of ZFS's dependencies.

First, one needs to enable multiarch like so: sudo dpkg --add-architecture arm64

Then, since Ubuntu's repositories list seems to be broken by default, one should follow these steps

And finally (note the :arm64 after each library name) sudo apt install libblkid-dev:arm64 uuid-dev:arm64 libudev-dev:arm64 libssl-dev:arm64 zlib1g-dev:arm64 libaio-dev:arm64 libattr1-dev:arm64 libelf-dev:arm64 libffi-dev:arm64

This at last allows me to run configure --host=aarch64-linux-gnu and then build without errors.

[danieldjewell]

FWIW, I know that cross compilation is possible and is a thing, but I've always gotten mixed results when cross-compiling anything more complex than a couple of C source files to a static binary. (Maybe that's me and/or inexperience talking.) I've had much better luck doing native on-target compiles - yes, compiling on a Raspberry Pi is waaaay slower, but eh.

@albertofustinoni I've had a pretty good experience with github/dockcross/dockcross using Docker - perhaps it might be useful here, too.

Also, while on the topic of ARM Crypto -- quite a few ARM64 chips have SHA extensions as well (the ARM SHA extensions were introduced earlier than Intel's SHA extensions, I think?) ... see: https://github.com/openssl/openssl/blob/e59bfbaa2dbd680f77e1121e382502bd522a466c/crypto/sha/asm/sha512-armv8.pl

The speedup is pretty big... On my phone with a Qualcomm Snapdragon 855 (SM8150) single thread SHA256 performance is ~1.3GB/sec versus ~290MB/sec on a Xeon E5-2620 v4 @ 2.1GHz. Multithread performance is even crazier - using 8 cores on my phone I get 6.5-6.6GB/sec! (The phone CPU is 8 cores but it's like "BIG_big.little ... 1x 2.84GHz "BIG", 3x "big" at like 2.4GHz and then 4x "little" at maybe like 1.8GHz.) Compared to 32 threads on the Xeon (16 cores + hyperthreading) which only gives ~4.5GB/sec. (Command: openssl speed -multi 8 sha256)

Even the ARM AES extensions are impressive: 8 threads on my phone (openssl speed -multi 8 -evp aes-128-cbc) tops out at 10+GB/sec! The Xeon does beat it but using 32 threads it's only peaking at ~12GB/sec. Pretty amazing. (The one place where you do see the difference is testing longer key lengths/more complex modes like aes-256-gcm ... the Xeon w/32 threads hits 35GB/sec while the phone is only 6.2GB/sec.) The Xeon has AES/AVX/AVX2 but not SHA.

Long story short: Using the ARM crypto extensions is a big deal for performance.

And also, the OpenSSL assembly code for sha256/512 on x86_64 which includes support for Intel SHA instructions. (Though from a benefit vs. time approach, only the newest x86_64 processors have the WP:Intel SHA instructions and the benefit of implementing on ARM first is probably greater...)

[rincebrain]

The specialized implementations are a BFD for performance, yes.

But as remarked in the source for it, "Reason for undertaken effort is that there is at least one popular SoC based on Cortex-A53 that doesn't have crypto extensions." (Of course, the 4 doesn't either, womp fucking womp.)

So you'd almost certainly want to loot those too, not just the crypto extension implementations, where possible.

[Stannieman]

Does that mean they are intentionally not using the extensions because there are important SOCs out there that don't support them? Why can they not check at runtime if the extensions are enabled and use them if they are?

[rincebrain]

That particular implementation is specifically for SoCs that don't implement them. I believe it was written because they already had an implementation that used the crypto extensions, but it obviously was not useful on platforms which did not implement them.

[Stannieman]

What is still needed to get this working? I could help but unfortunately I don't really know where to start. I assume ready made patches to apply when building ZFS from source don't exist yet?

[rincebrain]

As far as I know, all the overview from this comment is still needed - I don't know of anyone who has done anything for it already beyond trying to setup a cross-compile env.

[Stannieman]

Hmm yep I was afraid of that. I'm totally comfortable with some regular C++ and C coding but unfortunately this is above my skill level 😕 Let's have a look at LUKS then...

[lundman]

https://github.com/openzfs/zfs/discussions/14645#discussioncomment-5363584

[farlongsignal]

This would be great to have, especially for devices such as RockPro64 which can use SAS HBAs in it's PCIe x4 port. I've moved my pool from power hungry amd64 PC to such system, but sadly the performance is severly degraded without extensions.

[shivabohemian]

Nowadays, the arm64 architecture is becoming increasingly common, and I am also using a Rockchip 3568 CPU. When I use ZFS native encryption, I have noticed that its read and write speeds are significantly slower compared to not using encryption. May I ask when we can solve this issue?

[rincebrain]

atm you'd want to solve #14555 first.

[shivabohemian]

atm you'd want to solve #14555 first. It appears that this issue first appeared in kernel 6.2. I can build successfully in kernel 5.10 with arm64.

[Golui]

Let me preface this first. I am absolutely willing to contribute, but my journey with ZFS started less than two weeks ago, and I have little to no experience in writing linux kernel modules. I doubt I will be able to contribute anything meaningful any time soon without some guidance.

I also would like to apologise in advance if I come off as condescending; I am simply frustrated. I do not mean to criticise all of your efforts. My hope is to at the very least get some information on this issue, and hopefully get things moving again.

---

It's been over a year, and it seems that there have not been any developments. Efforts by lundman also appear to have stalled. At least, a quick cursory perusal of related issues and discussions indicates that last updates after from March 2023. I hope this is just me being bad at research and there being actual developments that I just missed.

If this is not the case, and these efforts have really stalled: What realistic solution does this issue have? The current solution to #14555 was to simply disable the acceleration, which is great band-aid to get zfs to work at all on AARCH64 on Linux 6.2+ but in my view it is hardly an acceptable long term solution.

[w0xel]

I got this working and posted a very WIP PR here: https://github.com/openzfs/zfs/pull/16601

I've basically followed @AttilaFueloep 's traces and made similar modifications to the armv8 OpenSSL codepath as they did for the x86 code to integrate in ZFS/ICP. I've then completely reused the x86 avx glue-code for the armv8 crypto variant.

Also after seeing @rincebrain 's comment regarding #14555 I've implemented a naive kfpu_begin similar to the x86 code that creates a strict critical section and stores NEON state between. This should be ok for most cases, but is probably unacceptable in RT environments.

I've tested this on a RPi5 based NAS with a very similar setup to the one described in @geerlingguy 's blog here: https://www.jeffgeerling.com/blog/2024/radxas-sata-hat-makes-compact-pi-5-nas

To develop this I've set up a cross-compile environment using NixOS & ccache to have a somewhat fast development cycle.

I'd be very thankful for some review, especially on the ASM parts from @AttilaFueloep and @rincebrain . My assembly needs armv8.1-a+crypto because it is using ldeoral for atomic toggle, if someone knows how to replace that we can probably get it to armv8-a+crypto and drop the .1.

I'd also need some guidance on how to make this merge-ready and integrate into upstream ZFS, especially the current abstraction for chunk-based AES does not seem like the perfect solution yet (but maybe perfect is the enemy of good here).

Here are some results, let me know if you want to see other benchmarks:

Benchmarks

Summary:

Mode	AES armv8 crypto	AES generic	Plain
Read 80 GiB	557MiB/s	54.3MiB/s	834MiB/s
Write 8 GiB	398MiB/s	67.4MiB/s	517MiB/s

So for the RPi5 we can expect about x10 improvements for read, and x6 improvements for write speed.

(write is only 8 GiB because I got impatient with the generic AES, but I've tested plain/crypto with 80 GiB with similar results)

fio logs below:

Plain Read (no encryption, baseline)

$ fio --rw=read --bs=1M --direct=1 --ioengine=libaio --size=10G  --group_reporting --filename=/zraid/tmp/testdat2 --name=job1 --offset=0G  --name=job2 --offset=10G --name=job3 --offset=20G --name=job4 --offset=30G  --name=job5 --offset=40G --name=job6 --offset=50G --name=job7 --offset=60G  --name=job8 --offset=70G
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job2: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job3: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job4: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job5: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job6: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job7: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job8: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.37
Starting 8 processes
job1: Laying out IO file (1 file / 10240MiB)
job2: Laying out IO file (1 file / 20480MiB)
job3: Laying out IO file (1 file / 30720MiB)
job4: Laying out IO file (1 file / 40960MiB)
job5: Laying out IO file (1 file / 51200MiB)
job6: Laying out IO file (1 file / 61440MiB)
job7: Laying out IO file (1 file / 71680MiB)
job8: Laying out IO file (1 file / 81920MiB)
Jobs: 8 (f=8): [R(8)][100.0%][r=899MiB/s][r=899 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=8): err= 0: pid=3612: Thu Oct  3 16:05:22 2024
  read: IOPS=833, BW=834MiB/s (874MB/s)(80.0GiB/98284msec)
    slat (usec): min=410, max=85000, avg=9553.04, stdev=1237.58
    clat (nsec): min=1537, max=4543.2k, avg=6602.17, stdev=20570.48
     lat (usec): min=413, max=85017, avg=9559.65, stdev=1239.09
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    8], 95.00th=[   10],
     | 99.00th=[   19], 99.50th=[   23], 99.90th=[   37], 99.95th=[   57],
     | 99.99th=[  174]
   bw (  KiB/s): min=169654, max=925696, per=100.00%, avg=853852.99, stdev=9753.30, samples=1568
   iops        : min=  160, max=  904, avg=833.79, stdev= 9.57, samples=1568
  lat (usec)   : 2=0.01%, 4=1.41%, 10=94.34%, 20=3.51%, 50=0.67%
  lat (usec)   : 100=0.04%, 250=0.01%, 500=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%
  cpu          : usr=0.12%, sys=1.86%, ctx=98054, majf=20, minf=2313
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=834MiB/s (874MB/s), 834MiB/s-834MiB/s (874MB/s-874MB/s), io=80.0GiB (85.9GB), run=98284-98284msec

Generic (unaccelerated) Encrypted Read (before patch)

$ fio --rw=read --bs=1M --direct=1 --ioengine=libaio --size=10G  --group_reporting --filename=/zraid/encrypted/testdat --name=job1 --offset=0G  --name=job2 --offset=10G --name=job3 --offset=20G --name=job4 --offset=30G  --name=job5 --offset=40G --name=job6 --offset=50G --name=job7 --offset=60G  --name=job8 --offset=70G
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job2: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job3: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job4: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job5: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job6: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job7: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job8: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.37
Starting 8 processes
Jobs: 3 (f=3): [_(4),R(3),_(1)][99.9%][r=50.0MiB/s][r=50 IOPS][eta 00m:01s]               
job1: (groupid=0, jobs=8): err= 0: pid=3521: Thu Oct  3 15:51:21 2024
  read: IOPS=54, BW=54.3MiB/s (57.0MB/s)(80.0GiB/1508225msec)
    slat (msec): min=24, max=445, avg=147.12, stdev=23.26
    clat (usec): min=4, max=158, avg=14.03, stdev= 8.39
     lat (msec): min=24, max=445, avg=147.14, stdev=23.26
    clat percentiles (nsec):
     |  1.00th=[ 6240],  5.00th=[ 7200], 10.00th=[ 7712], 20.00th=[ 8384],
     | 30.00th=[ 8896], 40.00th=[ 9408], 50.00th=[10048], 60.00th=[11072],
     | 70.00th=[14016], 80.00th=[20608], 90.00th=[27520], 95.00th=[32384],
     | 99.00th=[39680], 99.50th=[42752], 99.90th=[55552], 99.95th=[63232],
     | 99.99th=[79360]
   bw (  KiB/s): min=32759, max=118784, per=100.00%, avg=55684.85, stdev=1132.92, samples=24107
   iops        : min=   31, max=  116, avg=54.37, stdev= 1.11, samples=24107
  lat (usec)   : 10=48.49%, 20=30.65%, 50=20.69%, 100=0.17%, 250=0.01%
  cpu          : usr=0.02%, sys=0.19%, ctx=82693, majf=0, minf=2118
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=54.3MiB/s (57.0MB/s), 54.3MiB/s-54.3MiB/s (57.0MB/s-57.0MB/s), io=80.0GiB (85.9GB), run=1508225-1508225msec

Accelerated Encrypted Read

$ fio --rw=read --bs=1M --direct=1 --ioengine=libaio --size=10G  --group_reporting --filename=/zraid/encrypted/testdat --name=job1 --offset=0G  --name=job2 --offset=10G --name=job3 --offset=20G --name=job4 --offset=30G  --name=job5 --offset=40G --name=job6 --offset=50G --name=job7 --offset=60G  --name=job8 --offset=70G
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job2: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job3: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job4: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job5: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job6: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job7: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job8: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.37
Starting 8 processes
Jobs: 8 (f=8): [R(8)][99.3%][r=534MiB/s][r=533 IOPS][eta 00m:01s]
job1: (groupid=0, jobs=8): err= 0: pid=3456: Thu Oct  3 15:22:04 2024
  read: IOPS=557, BW=557MiB/s (584MB/s)(80.0GiB/147055msec)
    slat (usec): min=1661, max=77212, avg=14250.65, stdev=2095.00
    clat (usec): min=4, max=4823, avg=52.55, stdev=71.88
     lat (usec): min=1668, max=77256, avg=14303.20, stdev=2096.81
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   13], 10.00th=[   16], 20.00th=[   26],
     | 30.00th=[   34], 40.00th=[   38], 50.00th=[   41], 60.00th=[   45],
     | 70.00th=[   51], 80.00th=[   71], 90.00th=[  102], 95.00th=[  128],
     | 99.00th=[  182], 99.50th=[  212], 99.90th=[ 1123], 99.95th=[ 1614],
     | 99.99th=[ 3097]
   bw (  KiB/s): min=476229, max=727040, per=100.00%, avg=571065.45, stdev=4373.86, samples=2336
   iops        : min=  459, max=  710, avg=553.58, stdev= 4.33, samples=2336
  lat (usec)   : 10=0.93%, 20=13.45%, 50=55.22%, 100=19.96%, 250=10.11%
  lat (usec)   : 500=0.20%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.08%, 4=0.02%, 10=0.01%
  cpu          : usr=0.46%, sys=5.82%, ctx=91608, majf=0, minf=2125
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=557MiB/s (584MB/s), 557MiB/s-557MiB/s (584MB/s-584MB/s), io=80.0GiB (85.9GB), run=147055-147055msec

Plain Write (no encryption, baseline)

$ fio --rw=write --bs=1M --direct=1 --ioengine=libaio --size=1G  --group_reporting --filename=/zraid/tmp/testdat2 --name=job1 --offset=0G  --name=job2 --offset=1G --name=job3 --offset=2G --name=job4 --offset=3G  --name=job5 --offset=4G --name=job6 --offset=5G --name=job7 --offset=6G  --name=job8 --offset=7G
job1: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job2: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job3: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job4: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job5: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job6: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job7: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.37
Starting 8 processes
Jobs: 7 (f=7): [_(1),W(7)][94.1%][w=520MiB/s][w=519 IOPS][eta 00m:01s]
job1: (groupid=0, jobs=8): err= 0: pid=3633: Thu Oct  3 16:07:02 2024
  write: IOPS=516, BW=517MiB/s (542MB/s)(8192MiB/15851msec); 0 zone resets
    slat (usec): min=3692, max=62985, avg=14872.87, stdev=5948.60
    clat (usec): min=3, max=12671, avg=25.48, stdev=223.88
     lat (usec): min=3696, max=63667, avg=14898.35, stdev=5959.87
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   11], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   18], 90.00th=[   25], 95.00th=[   33],
     | 99.00th=[  239], 99.50th=[  635], 99.90th=[ 1450], 99.95th=[ 2540],
     | 99.99th=[12649]
   bw (  KiB/s): min=227183, max=798538, per=100.00%, avg=535003.97, stdev=14315.39, samples=243
   iops        : min=  220, max=  778, avg=519.48, stdev=13.96, samples=243
  lat (usec)   : 4=0.12%, 10=37.63%, 20=46.74%, 50=12.39%, 100=1.23%
  lat (usec)   : 250=0.92%, 500=0.40%, 750=0.21%, 1000=0.16%
  lat (msec)   : 2=0.11%, 4=0.05%, 10=0.01%, 20=0.02%
  cpu          : usr=0.44%, sys=28.71%, ctx=86792, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=8192MiB (8590MB), run=15851-15851msec

Generic (unaccelerated) Encrypted Write (before patch)

$ fio --rw=write --bs=1M --direct=1 --ioengine=libaio --size=1G  --group_reporting --filename=/zraid/encrypted/testdat2 --name=job1 --offset=0G  --name=job2 --offset=1G --name=job3 --offset=2G --name=job4 --offset=3G  --name=job5 --offset=4G --name=job6 --offset=5G --name=job7 --offset=6G  --name=job8 --offset=7G
job1: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job2: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job3: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job4: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job5: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job6: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job7: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.37
Starting 8 processes
job1: Laying out IO file (1 file / 1024MiB)
job2: Laying out IO file (1 file / 2048MiB)
job3: Laying out IO file (1 file / 3072MiB)
job4: Laying out IO file (1 file / 4096MiB)
job5: Laying out IO file (1 file / 5120MiB)
job6: Laying out IO file (1 file / 6144MiB)
job7: Laying out IO file (1 file / 7168MiB)
job8: Laying out IO file (1 file / 8192MiB)
Jobs: 2 (f=2): [_(3),W(1),_(3),W(1)][99.2%][w=43.0MiB/s][w=43 IOPS][eta 00m:01s]          
job1: (groupid=0, jobs=8): err= 0: pid=3652: Thu Oct  3 16:09:59 2024
  write: IOPS=67, BW=67.4MiB/s (70.6MB/s)(8192MiB/121595msec); 0 zone resets
    slat (msec): min=40, max=394, avg=116.14, stdev=71.13
    clat (usec): min=4, max=15124, avg=17.16, stdev=258.23
     lat (msec): min=40, max=394, avg=116.16, stdev=71.12
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    8], 20.00th=[    8],
     | 30.00th=[    9], 40.00th=[    9], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   11], 80.00th=[   11], 90.00th=[   12], 95.00th=[   15],
     | 99.00th=[   26], 99.50th=[   34], 99.90th=[  603], 99.95th=[ 8029],
     | 99.99th=[15139]
   bw (  KiB/s): min=16180, max=169474, per=100.00%, avg=70515.47, stdev=3996.05, samples=1885
   iops        : min=    8, max=  161, avg=62.69, stdev= 3.98, samples=1885
  lat (usec)   : 10=64.87%, 20=33.03%, 50=1.77%, 100=0.20%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 10=0.07%, 20=0.01%
  cpu          : usr=0.05%, sys=41.02%, ctx=108151, majf=0, minf=76
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=67.4MiB/s (70.6MB/s), 67.4MiB/s-67.4MiB/s (70.6MB/s-70.6MB/s), io=8192MiB (8590MB), run=121595-121595msec

Accelerated Encrypted Write

$ fio --rw=write --bs=1M --direct=1 --ioengine=libaio --size=1G  --group_reporting --filename=/zraid/encrypted/testdat2 --name=job1 --offset=0G  --name=job2 --offset=1G --name=job3 --offset=2G --name=job4 --offset=3G  --name=job5 --offset=4G --name=job6 --offset=5G --name=job7 --offset=6G  --name=job8 --offset=7G
job1: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job2: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job3: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job4: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job5: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job6: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job7: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
job8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.37
Starting 8 processes
Jobs: 3 (f=3): [_(1),W(1),_(1),W(1),_(2),W(1),_(1)][91.3%][w=416MiB/s][w=416 IOPS][eta 00m:02s]
job1: (groupid=0, jobs=8): err= 0: pid=3685: Thu Oct  3 16:14:30 2024
  write: IOPS=397, BW=398MiB/s (417MB/s)(8192MiB/20596msec); 0 zone resets
    slat (msec): min=4, max=231, avg=18.73, stdev=13.27
    clat (usec): min=3, max=6340, avg=23.80, stdev=98.87
     lat (msec): min=4, max=231, avg=18.75, stdev=13.27
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   14], 60.00th=[   16],
     | 70.00th=[   18], 80.00th=[   22], 90.00th=[   29], 95.00th=[   38],
     | 99.00th=[  235], 99.50th=[  523], 99.90th=[ 1221], 99.95th=[ 1516],
     | 99.99th=[ 6325]
   bw (  KiB/s): min=44976, max=944657, per=100.00%, avg=427202.16, stdev=23370.50, samples=306
   iops        : min=   41, max=  919, avg=413.75, stdev=22.86, samples=306
  lat (usec)   : 4=0.09%, 10=22.11%, 20=55.30%, 50=19.26%, 100=1.51%
  lat (usec)   : 250=0.79%, 500=0.42%, 750=0.26%, 1000=0.15%
  lat (msec)   : 2=0.11%, 10=0.01%
  cpu          : usr=0.40%, sys=33.30%, ctx=123112, majf=0, minf=72
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=398MiB/s (417MB/s), 398MiB/s-398MiB/s (417MB/s-417MB/s), io=8192MiB (8590MB), run=20596-20596msec

[w0xel]

Another potentially interesting datapoint: using the block-based AES crypto acceleration does not seem to bring a big measurable performance boost:

$ cat /sys/module/zfs/parameters/icp_aes_impl 
cycle fastest generic [armv8_crypto] 
$ cat /sys/module/zfs/parameters/icp_gcm_impl 
cycle fastest hardware [generic]

# ... truncated fio log:
   READ: bw=55.3MiB/s (58.0MB/s), 55.3MiB/s-55.3MiB/s (58.0MB/s-58.0MB/s), io=8192MiB (8590MB), run=148034-148034msec
  WRITE: bw=71.0MiB/s (74.4MB/s), 71.0MiB/s-71.0MiB/s (74.4MB/s-74.4MB/s), io=8192MiB (8590MB), run=115427-115427msec

So this should be using the block based aes_v8_encrypt and a lot of software-based hashing around that. In comparison, the benchmarks of the comment above use the special all-in-one AES+ghash codepath that was written for AVX once that operates on larger chunks.

The values are so similar to the generic implementation, that I'm not sure the block-based code path is correctly used, on the other hand I'm pretty sure it is used. Might be that the state store/restore code just completely eats up any potential benefit here... So I think the faster codepath also used for AVX is definitely the way to go here.

[rincebrain]

Yes, I think anyone who's looked at it could tell you there's a lot of optimization on the floor waiting for someone to do for the encryption codepaths.

...of course, that should probably come after "correctness" is also achieved, but, you know.

[w0xel]

A bit of a braindump, because I'd be willing to do this before integrating my PR:

My idea would be extending the gcm_impl_ops_t:

typedef struct gcm_impl_ops {
    gcm_mul_f mul;
    gcm_will_work_f is_supported;
    char name[GCM_IMPL_NAME_MAX];
    boolean_t has_chunked_ops;         // <- new
    gcm_chunked_impl_ops *chunked_ops; // <- new
} gcm_impl_ops_t;

And a new struct for the chunked implementation, i.e. AVX, armv8 crypto and potentially more in the future (pure neon, unrolled):

typedef struct gcm_chunked_impl_ops {
    gcm_clear_fpu_regs_f clear_fpu_regs;
    gcm_xor_f xor;
    aes_encrypt_block_f encrypt_block;
    gcm_encrypt_chunk_f encrypt_chunk;
    gcm_decrypt_chunk_f decrypt_chunk;
    gcm_hash_f ghash;
    gcm_init_htab_f init_htab;
} gcm_chunked_impl_ops_t;

(I'd vote for removing xor from this as it only operates on a single block anyways and I can't imagine a non-simd version would be that much worse here).

I would keep the non-chunked generic codepath, because making that chunked would just make this code overly-complex, and I think it makes a lot of sense to have the generic codepath as simple as possible (i.e. has_chunked_ops == B_FALSE and chunked_ops == NULL).

For the AVX & armv8 crypto codepaths I would put all the functions into the gcm_<impl>.c files and fill out the ops structs, and have the chunked codepath (currently AVX only) serve all of them.

This should pave the way for more impls in the future, e.g. the pure-neon (non-crypto) implementation for the older raspberry-pis.

One thing where I'm not quite sure yet is how to handle the gcm_avx_can_use_movbe. Maybe we need another gcm_impl_cycle_f in the ops, but I'm also not sure I'm even understanding this code correctly.