search

RemixVSL / iomemory-vsl

Updated Fusion-io iomemory VSL Linux (version 3.2.16) driver for recent kernels.
150 stars 27 forks source link

[BUG] "Invalid module format" on ProxMox 7. #91

Closed kozmanbalint closed 3 years ago

kozmanbalint commented 3 years ago

Bug description

Kernel module compiles, but cannot be loaded on ProxMox 7 with the error above.

How to reproduce

git clone https://github.com/snuf/iomemory-vsl.git
cd iomemory-vsl
git checkout v5.12.1
make module
insmod root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko
insmod: ERROR: could not insert module root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko: Invalid module format

Environment information

root@hades2# pveversion
pve-manager/7.0-9/228c9caa (running kernel: 5.11.22-1-pve)
root@hades2# uname -a
Linux hades2 5.11.22-1-pve #1 SMP PVE 5.11.22-2 (Fri, 02 Jul 2021 16:22:45 +0200) x86_64 GNU/Linux
root@hades2# gcc --version
gcc (Debian 10.2.1-6) 10.2.1 20210110
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
root@hades2# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Source tag: v5.12.1

root@hades2# fio-status
Found 2 VSL driver packages:
   4.3.7 build 1205 Driver: not loaded
   3.2.16 build 1731 Driver: not loaded

Found 2 ioMemory devices in this system with 1 ioDrive Duo

Adapter: Dual Adapter  (driver 3.2.16)
    HP 1280GB MLC PCIe ioDrive Duo for ProLiant Servers, Product Number:641027-B21, SN:620030
    PCIe Power limit threshold: Disabled
    Connected ioMemory modules:
      0e:00.0:  Product Number:641027-B21, SN:675925
      0d:00.0:  Product Number:641027-B21, SN:675924

0e:00.0 ioDIMM 640, SN:675925
    Located in slot 1 Lower of ioDrive Duo HL SN:620030
    PCI:0e:00.0
    Firmware v7.1.15, rev 110356 Public
    Internal temperature: 42.82 degC, max 44.30 degC

0d:00.0 ioDIMM 640, SN:675924
    Located in slot 0 Upper of ioDrive Duo HL SN:620030
    PCI:0d:00.0
    Firmware v7.1.15, rev 110356 Public
    Internal temperature: 42.82 degC, max 44.30 degC
snuf commented 3 years ago

@kozmanbalint this is probably due to the kernel you are building the module for, and the kernel you're inserting the module into are not the same. You've not provided the compile output, would it be possible to add that, as that would verify the hypothesis.

kozmanbalint commented 3 years ago

Here we go:

root@hades2:~/ioDrive/iomemory-vsl# make module
cd root/usr/src/iomemory-vsl-3.2.16 && \
    make clean
make[1]: Entering directory '/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16'
make \
        -j16 \
    -C /lib/modules/5.11.22-1-pve/build \
    FIO_DRIVER_NAME=iomemory-vsl \
    FUSION_DRIVER_DIR=/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16 \
    M=/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16 \
    EXTRA_CFLAGS+="-I/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/include -DBUILDING_MODULE -DLINUX_IO_SCHED -Wall -Werror" \
    KFIO_LIB=kfio/x86_64_cc102_libkfio.o_shipped \
    clean
make[2]: Entering directory '/usr/src/linux-headers-5.11.22-1-pve'
make[2]: Leaving directory '/usr/src/linux-headers-5.11.22-1-pve'
rm -rf include/fio/port/linux/kfio_config.h kfio_config license.c
make[1]: Leaving directory '/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16'
cd root/usr/src/iomemory-vsl-3.2.16 && \
    make gpl
make[1]: Entering directory '/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16'
sed -i 's/Proprietary/GPL/g' Kbuild
if [ "102" -gt "74" ];then \
    if [ ! -f "kfio/x86_64_cc102_libkfio.o_shipped" ];then \
        cp kfio/x86_64_cc74_libkfio.o_shipped kfio/x86_64_cc102_libkfio.o_shipped; \
    fi \
fi
./kfio_config.sh -a x86_64 -o include/fio/port/linux/kfio_config.h -k /lib/modules/5.11.22-1-pve/build -p -d /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config -l 0 -s /lib/modules/5.11.22-1-pve/source
Detecting Kernel Flags
Config dir         : /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config
Output file        : include/fio/port/linux/kfio_config.h
Kernel output dir  : /lib/modules/5.11.22-1-pve/build
Kernel source dir  : /lib/modules/5.11.22-1-pve/build
Starting tests:
  1626204679.592  KFIOC_X_PROC_CREATE_DATA_WANTS_PROC_OPS...
  1626204679.610  KFIOC_X_TASK_HAS_CPUS_MASK...
  1626204679.626  KFIOC_X_LINUX_HAS_PART_STAT_H...
  1626204679.644  KFIOC_X_BLK_ALLOC_QUEUE_NODE_EXISTS...
  1626204679.663  KFIOC_X_HAS_MAKE_REQUEST_FN...
  1626204679.682  KFIOC_X_GENHD_PART0_IS_A_POINTER...
  1626204679.701  KFIOC_X_BIO_HAS_BI_BDEV...
Started tests, waiting for completions...
  1626204681.744  KFIOC_X_PROC_CREATE_DATA_WANTS_PROC_OPS=1
  1626204681.766  KFIOC_X_TASK_HAS_CPUS_MASK=1
  1626204681.791  KFIOC_X_LINUX_HAS_PART_STAT_H=1
  1626204681.815  KFIOC_X_BLK_ALLOC_QUEUE_NODE_EXISTS=0
  1626204681.839  KFIOC_X_HAS_MAKE_REQUEST_FN=0
  1626204681.863  KFIOC_X_GENHD_PART0_IS_A_POINTER=1
  1626204681.886  KFIOC_X_BIO_HAS_BI_BDEV=0
Finished
1626204681.896  Exiting
Preserving configdir due to '-p' option: /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config
grep: license.c: No such file or directory

Adding module version 5.11.22-1-ecc66f1-3.2.16 to source license.c

make \
    -j16 \
-C /lib/modules/5.11.22-1-pve/build \
FIO_DRIVER_NAME=iomemory-vsl \
FUSION_DRIVER_DIR=/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16 \
M=/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16 \
EXTRA_CFLAGS+="-I/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/include -DBUILDING_MODULE -DLINUX_IO_SCHED -Wall -Werror" \
INSTALL_MOD_DIR=extra/fio \
INSTALL_MOD_PATH= \
KFIO_LIB=kfio/x86_64_cc102_libkfio.o_shipped \
modules
make[2]: Entering directory '/usr/src/linux-headers-5.11.22-1-pve'
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/main.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/license.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/pci.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/sysrq.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/driver_init.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/errno.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/state.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kcache.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfile.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kmem.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_common.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kcpu.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kscatter.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/ktime.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/sched.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/cdev.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kblock.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kcondvar.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kinfo.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kexports.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/khotplug.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kcsr.o
  SHIPPED /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc102_libkfio.o
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/module_param.o
  LD [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.o
  MODPOST /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/Module.symvers
  CC [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.mod.o
  LD [M]  /root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko
make[2]: Leaving directory '/usr/src/linux-headers-5.11.22-1-pve'

Ok: patched version for iomemory-vsl.ko replaced 3.2.16.1731 pinnacles@4870ad45b7ea with 5.11.22-1-ecc66f1-3.2.16.1731

make[1]: Leaving directory '/root/ioDrive/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16'
snuf commented 3 years ago

hmm interesting, Looking at the uname you pasted I do see two different versions in there Linux hades2 5.11.22-1-pve #1 SMP PVE 5.11.22-2 (Fri, 02 Jul 2021 16:22:45 +0200) x86_64 GNU/Linux. Has this machine been upgraded and not rebooted perhaps? The only reason this error comes up is if you try to insert a module that is not meant for that kernel.

kozmanbalint commented 3 years ago

Well, really strange. The machine has actually been rebooted quite a few times after upgrading to proxmox 7, but there must have been something wrong related to the upgrade: after manually reinstalling the pve kernel image and headers, reboot, and recompile the iomemory-vsl module, all errors are gone: apt install --reinstall pve-kernel-5.11.22-1-pve pve-headers-5.11.22-1-pve pve-firmware reboot ...

root@hades2:~/ioDrive/iomemory-vsl# make clean
root@hades2:~/ioDrive/iomemory-vsl# make module
root@hades2:~/ioDrive/iomemory-vsl# insmod root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko
root@hades2:~/ioDrive/iomemory-vsl# fio-status 

Found 2 VSL driver packages:
   4.3.7 build 1205 Driver: not loaded
   3.2.16 build 1731 Driver: loaded

Found 2 ioMemory devices in this system with 1 ioDrive Duo

Adapter: Dual Adapter  (driver 3.2.16)
    HP 1280GB MLC PCIe ioDrive Duo for ProLiant Servers, Product Number:641027-B21, SN:620030
    PCIe Power limit threshold: 24.75W
    Connected ioMemory modules:
      fct0: 0d:00.0,    Product Number:641027-B21, SN:675924
      fct1: 0e:00.0,    Product Number:641027-B21, SN:675925

fct0    Attached
    ioDIMM 640, SN:675924
    Located in slot 0 Upper of ioDrive Duo HL SN:620030
    Last Power Monitor Incident: 88 sec
    PCI:0d:00.0
    Firmware v7.1.15, rev 110356 Public
    640.00 GBytes device size
    Internal temperature: 44.79 degC, max 45.77 degC
    Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
    Contained Virtual Partitions:
      fioa: ID:0, UUID:1437a61f-5fe9-44e6-8e27-7de7ab9f060a

fioa    State: Online, Type: block device, Device: /dev/fioa
    ID:0, UUID:1437a61f-5fe9-44e6-8e27-7de7ab9f060a
    640.00 GBytes device size

fct1    Attached
    ioDIMM 640, SN:675925
    Located in slot 1 Lower of ioDrive Duo HL SN:620030
    Last Power Monitor Incident: 88 sec
    PCI:0e:00.0
    Firmware v7.1.15, rev 110356 Public
    640.00 GBytes device size
    Internal temperature: 45.28 degC, max 46.76 degC
    Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
    Contained Virtual Partitions:
      fiob: ID:0, UUID:93e755d4-341a-6c44-9e28-58f0d96d4dc7

fiob    State: Online, Type: block device, Device: /dev/fiob
    ID:0, UUID:93e755d4-341a-6c44-9e28-58f0d96d4dc7
    640.00 GBytes device size

And the adapter seems to work fine! Thanks for your help!