Configuring stops with error message: This kernel is unable to compile object files.

wq9578 commented 1 year ago

System information

Type	Version/Name
Distribution Name	Raspbian GNU/Linux 11 (bullseye)
Distribution Version	11
Kernel Version	6.1.14-v8+
Architecture	arm64
OpenZFS Version	(current git clone)

Describe the problem you're observing

Configuring stops on a Raspberry Pi 3 Model B+ with Raspberry Pi OS (64-bit) with the error message: "This kernel is unable to compile object files." This happens despite the fact that CONFIG_MODULES=y is specified in the .config in the linux kernel source tree, and make prepare was run. See also the main thread on Raspberry Pi Forums.

General question: Is OpenZFS ready for use in production? Currently there are 500 open issues with "Type: Defect" for ZFS, compared with 137 bugs total for ext4.

Describe how to reproduce the problem

Install required packages: sudo apt install -y git bc bison flex libssl-dev make libc6-dev libncurses5-dev raspberrypi-kernel-headers crossbuild-essential-arm64 automake libtool uuid-dev libblkid-dev
Download kernel source: git clone --depth=1 --branch rpi-6.2.y https://github.com/raspberrypi/linux
Configure kernel (in kernel dir): KERNEL=kernel8; make -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig

Add CONFIG_ZFS=y to .config (in kernel dir):

cat >>.config <<EOF
#CONFIG_CRYPTO_DEFLATE=y
#CONFIG_ZLIB_DEFLATE=y
#CONFIG_KALLSYMS=y
#CONFIG_EFI_PARTITION=y
CONFIG_ZFS=y
EOF

Prepare kernel (in kernel dir): make -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- prepare
Download ZFS source: git clone https://github.com/openzfs/zfs

Configure ZFS:

cd zfs
sh ./autogen.sh
LINUX_DIR=../linux
./configure --enable-linux-builtin --with-linux=$LINUX_DIR --with-linux-obj=$LINUX_DIR

Instructions Kernel: https://www.raspberrypi.com/documentation/computers/linux_kernel.html Instructions ZFS: https://github.com/openzfs/zfs/issues/10450#issuecomment-643654436

Include any warning/errors/backtraces from the system logs

$ LINUX_DIR=../linux; ./configure --enable-linux-builtin --with-linux=$LINUX_DIR --with-linux-obj=$LINUX_DIR
checking for gawk... no
checking for mawk... mawk
checking metadata... git describe
checking build system type... aarch64-unknown-linux-gnu
[...]
checking kernel source and build directories... done
checking kernel source directory... ../linux
checking kernel build directory... ../linux
checking kernel source version... 6.2.3-v8l-ZFS+
checking for kernel config option compatibility... done
checking whether CONFIG_MODULES is defined... error
configure: error: 
        *** This kernel is unable to compile object files.
        ***
        *** Please make sure you prepared the Linux source tree
        *** by running `make prepare` there.
$ grep CONFIG_MODULES= ../linux/.config                                                                     
CONFIG_MODULES=y

ryao commented 1 year ago

First, you should be installing the kernel headers, not compiling a kernel:

https://www.raspberrypi.com/documentation/computers/linux_kernel.html#kernel-headers

Second, your kernel source version is different from your running kernel version. Unless you install this kernel, you are not going to be able to use modules built against it with your running kernel. If you want to continue trying to use it, follow the instructions under Building the Kernel Locally:

https://www.raspberrypi.com/documentation/computers/linux_kernel.html#building-the-kernel-locally

Do not use the cross compilation instructions, since then you will need to cross compile ZFS too and that is unnecessarily complicated.

That said, if the source versions were the same, hypothetically, you should be able to build compatible kernel modules provided that you built the kernel the exact same way that raspbian did, but in that case, you would want to be using make modules_prepare, not make prepare.

You should not use the instructions from #10450. They are for a very special configuration that builds ZFS into the kernel rather than as a module. For your purposes, you likely want to use ZFS as a kernel module. For building ZFS, you want to follow these instructions:

https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html#

On second thought, ignore everything I just said and do this instead:

sudo apt install raspberrypi-kernel-headers

sudo apt install dkms

sudo apt install build-essential autoconf automake libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev linux-headers-generic python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev

wget https://github.com/openzfs/zfs/releases/download/zfs-2.1.9/zfs-2.1.9.tar.gz

tar -xf zfs-2.1.9.tar.gz

cd zfs-2.1.9

./configure
make -s -j$(nproc)
make deb-dkms deb-utils

Then you should have .deb packages that you can install with sudo dpkg -i .... I am a bit fuzzy on which one to install first, since I do not have the list that is generated. I assume doing sudo dpkg -i *.deb would work.

wq9578 commented 1 year ago

make -s -j$(nproc) worked, but gave one warning:

Making all in module
[...]/linux/zfs-2.1.9/module/zfs/../os/linux/zfs/zfs_vnops_os.c: In function ‘zfs_setattr’:
[...]/linux/zfs-2.1.9/module/zfs/../os/linux/zfs/zfs_vnops_os.c:2205:7: warning: cast between incompatible function types from ‘int (*)(struct znode *, mode_t,  cred_t *)’ {aka ‘int (*)(struct znode *, short unsigned int,  struct cred *)’} to ‘int (*)(void *, int,  cred_t *)’ {aka ‘int (*)(void *, int,  struct cred *)’} [-Wcast-function-type]
 2205 |       (int (*)(void *, int, cred_t *))zfs_zaccess_unix, zp);
      |       ^

ryao commented 1 year ago

The warning merits a bug report of its own, but it should be okay.

wq9578 commented 1 year ago

Result of make deb-dkms deb-utils:

make[2]: Leaving directory '/tmp/zfs-build-pi-bjV6jhp3/BUILD/zfs-2.1.9'
+ find /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/lib/arm-linux-gnueabihf -name *.la -exec rm -f {} ;
+ find /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/bin ( -name arc_summary -or -name arcstat -or -name dbufstat ) -exec /bin/sed -i s|^#!.*|#!/usr/bin/python3| {} ;
+ find /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share ( -name test-runner.py -or -name zts-report.py ) -exec /bin/sed -i s|^#!.*|#!/usr/bin/python3| {} ;
+ RPM_EC=0
+ jobs -p
+ exit 0
Processing files: zfs-2.1.9-1.aarch64
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/sbin/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man1/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man4/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man5/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man7/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man8/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/lib/udev/rules.d/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/systemd/system/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/systemd/system-preset/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/modules-load.d/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/systemd/system-generators/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/zfs/zed.d/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/zfs/zpool.d/*
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/zfs/vdev_id.conf.*.example
error: File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/sudoers.d/*

RPM build errors:
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/sbin/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man1/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man4/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man5/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man7/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/share/man/man8/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/lib/udev/rules.d/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/systemd/system/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/systemd/system-preset/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/modules-load.d/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/usr/lib/systemd/system-generators/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/zfs/zed.d/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/zfs/zpool.d/*
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/zfs/vdev_id.conf.*.example
    File not found: /tmp/zfs-build-pi-bjV6jhp3/BUILDROOT/zfs-2.1.9-1.%{_arch}/etc/sudoers.d/*
make[1]: *** [Makefile:1692: rpm-common] Fehler 1
make[1]: Verzeichnis „[...]/linux/zfs-2.1.9“ wird verlassen
make: *** [Makefile:1651: rpm-utils-initramfs] Fehler 2

rincebrain commented 1 year ago

You know, you say "64-bit", but you're really on a 64-bit kernel and 32-bit userland, I think, from a couple of those errors, and the forum thread.

And that is a very different kettle of fish. I don't think anyone's made any attempts to make sure anything in the build chain and user/kernel interface plays well with a 32-bit userland and 64-bit kernel, and I think there are known cases where that would break (I believe nvlists encode things about the pointer length, for example...).

So at least without a 64-bit chroot for userland to run things in, or a full 64-bit userland, I don't think this is going to fly easily.

I will also note, my Pi 3B and 4B are happily running OpenZFS with a full 64-bit userland and kernel. So it's not a Pi problem.

beren12 commented 1 year ago

Install zfs from backports and you get 2.1.9. I have my 3b using 32bit kernel but it does build fine, same as my pi4 with 64bit.

wq9578 commented 1 year ago

Install zfs from backports and you get 2.1.9. I have my 3b using 32bit kernel but it does build fine, same as my pi4 with 64bit.

Could you please provide the exact commands how to do this? I'm a newbie in this field. Thanks!

Low-power commented 1 year ago

I don't think anyone's made any attempts to make sure anything in the build chain and user/kernel interface plays well with a 32-bit userland and 64-bit kernel, and I think there are known cases where that would break (I believe nvlists encode things about the pointer length, for example...).

The primary operating system I uses on my POWER7-based machine is in fact Debian GNU/Linux 8 powerpc, with 32-bit userland and a 64-bit kernel. It runs latest OpenZFS releases without any userland compatibility problems.

rincebrain commented 1 year ago

I'm quite curious how you're building that, as I would really not expect that to work without a lot of handholding for the compiler flags alone, let alone whether I was correct about there being native bits in the ioctls.

openzfs / zfs