search

scaleway / kernel-tools

:penguin: Kernels on Scaleway
http://devhub.scaleway.com/#/bootscripts
MIT License
104 stars 36 forks source link

ZFS kernel module failed to install on latest bootscripts #371

Closed caseycs closed 5 years ago

caseycs commented 6 years ago

Hi!

I'm trying to switch to new bootscripts, so I tried both x86_64 mainline 4.14.11 rev1 and x86_64 4.10.8 std #1

However on each of them I could not install zfs kernel module:

deploy@scaleway2:~$ sudo bash -ex zfs.sh
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
++ uname -m
+ arch=x86_64
++ uname -r
+ release=4.14.11-mainline-rev1
+ upstream=4.14.11
+ local=mainline-rev1
+ mkdir -p /usr/src
+ wget -O /usr/src/linux-4.14.11.tar.xz https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.11.tar.xz
--2018-01-05 14:08:00--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.11.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.37.176, 2a04:4e42:9::432
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.37.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100816808 (96M) [application/x-xz]
Saving to: '/usr/src/linux-4.14.11.tar.xz'

/usr/src/linux-4.14.11.tar.xz 100%[================================================>]  96.15M   107MB/s    in 0.9s

2018-01-05 14:08:01 (107 MB/s) - '/usr/src/linux-4.14.11.tar.xz' saved [100816808/100816808]

+ tar xf /usr/src/linux-4.14.11.tar.xz -C /usr/src/
+ ln -fns /usr/src/linux-4.14.11 /usr/src/linux
+ ln -fns /usr/src/linux-4.14.11 /lib/modules/4.14.11-mainline-rev1/build
+ zcat /proc/config.gz
+ printf 'CONFIG_LOCALVERSION="%s"\nCONFIG_CROSS_COMPILE=""\n' -mainline-rev1
+ wget -O /usr/src/linux/Module.symvers http://mirror.scaleway.com/kernel/x86_64/4.14.11-mainline-rev1/Module.symvers
--2018-01-05 14:08:15--  http://mirror.scaleway.com/kernel/x86_64/4.14.11-mainline-rev1/Module.symvers
Resolving mirror.scaleway.com (mirror.scaleway.com)... 51.15.33.64
Connecting to mirror.scaleway.com (mirror.scaleway.com)|51.15.33.64|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 632925 (618K) [application/octet-stream]
Saving to: '/usr/src/linux/Module.symvers'

/usr/src/linux/Module.symvers 100%[================================================>] 618.09K  --.-KB/s    in 0.009s

2018-01-05 14:08:15 (70.8 MB/s) - '/usr/src/linux/Module.symvers' saved [632925/632925]

+ apt-get install -y libssl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libssl-dev is already the newest version (1.0.2g-1ubuntu4.10).
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
+ make -C /usr/src/linux prepare modules_prepare
make: Entering directory '/usr/src/linux-4.14.11'
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
scripts/kconfig/conf  --silentoldconfig Kconfig
.config:4887:warning: override: reassigning to symbol LOCALVERSION
.config:4888:warning: override: reassigning to symbol CROSS_COMPILE
#
# configuration written to .config
#
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
  CHK     include/config/kernel.release
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
  CHK     include/generated/bounds.h
  CHK     include/generated/timeconst.h
  CHK     include/generated/asm-offsets.h
  CALL    scripts/checksyscalls.sh
  CHK     scripts/mod/devicetable-offsets.h
make: Leaving directory '/usr/src/linux-4.14.11'
deploy@scaleway2:~$ KDIR=/lib/modules/$(uname -r)/build/
deploy@scaleway2:~$ apt-get install zfsutils-linux
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?
deploy@scaleway2:~$ sudo apt-get install zfsutils-linux
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  dkms libnvpair1linux libuutil1linux libzfs2linux libzpool2linux spl-dkms zfs-dkms zfs-doc zfs-zed
Suggested packages:
  default-mta | mail-transport-agent samba-common-bin nfs-kernel-server zfs-initramfs
The following NEW packages will be installed:
  dkms libnvpair1linux libuutil1linux libzfs2linux libzpool2linux spl-dkms zfs-dkms zfs-doc zfs-zed zfsutils-linux
0 upgraded, 10 newly installed, 0 to remove and 3 not upgraded.
Need to get 591 kB/2432 kB of archives.
After this operation, 14.5 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 zfs-doc all 0.6.5.6-0ubuntu18 [50.0 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libuutil1linux amd64 0.6.5.6-0ubuntu18 [27.4 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnvpair1linux amd64 0.6.5.6-0ubuntu18 [23.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libzpool2linux amd64 0.6.5.6-0ubuntu18 [384 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libzfs2linux amd64 0.6.5.6-0ubuntu18 [106 kB]
Fetched 591 kB in 0s (7769 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = "en_US.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Selecting previously unselected package dkms.
(Reading database ... 36287 files and directories currently installed.)
Preparing to unpack .../dkms_2.2.0.3-2ubuntu11.5_all.deb ...
Unpacking dkms (2.2.0.3-2ubuntu11.5) ...
Selecting previously unselected package spl-dkms.
Preparing to unpack .../spl-dkms_0.6.5.6-0ubuntu4_amd64.deb ...
Unpacking spl-dkms (0.6.5.6-0ubuntu4) ...
Selecting previously unselected package zfs-doc.
Preparing to unpack .../zfs-doc_0.6.5.6-0ubuntu18_all.deb ...
Unpacking zfs-doc (0.6.5.6-0ubuntu18) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up dkms (2.2.0.3-2ubuntu11.5) ...
Setting up spl-dkms (0.6.5.6-0ubuntu4) ...
Loading new spl-0.6.5.6 DKMS files...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
First Installation: checking all kernels...
dpkg: warning: version '*-*' has bad syntax: version number does not start with digit
It is likely that 4.14.11-mainline-rev1 belongs to a chroot's host
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Building initial module for 4.14.11-mainline-rev1
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
configure: error: unknown
Error! Bad return status for module build on kernel: 4.14.11-mainline-rev1 (x86_64)
Consult /var/lib/dkms/spl/0.6.5.6/build/make.log for more information.
Selecting previously unselected package zfs-dkms.
(Reading database ... 36683 files and directories currently installed.)
Preparing to unpack .../zfs-dkms_0.6.5.6-0ubuntu18_amd64.deb ...
Unpacking zfs-dkms (0.6.5.6-0ubuntu18) ...
Selecting previously unselected package libuutil1linux.
Preparing to unpack .../libuutil1linux_0.6.5.6-0ubuntu18_amd64.deb ...
Unpacking libuutil1linux (0.6.5.6-0ubuntu18) ...
Selecting previously unselected package libnvpair1linux.
Preparing to unpack .../libnvpair1linux_0.6.5.6-0ubuntu18_amd64.deb ...
Unpacking libnvpair1linux (0.6.5.6-0ubuntu18) ...
Selecting previously unselected package libzpool2linux.
Preparing to unpack .../libzpool2linux_0.6.5.6-0ubuntu18_amd64.deb ...
Unpacking libzpool2linux (0.6.5.6-0ubuntu18) ...
Selecting previously unselected package libzfs2linux.
Preparing to unpack .../libzfs2linux_0.6.5.6-0ubuntu18_amd64.deb ...
Unpacking libzfs2linux (0.6.5.6-0ubuntu18) ...
Selecting previously unselected package zfsutils-linux.
Preparing to unpack .../zfsutils-linux_0.6.5.6-0ubuntu18_amd64.deb ...
Unpacking zfsutils-linux (0.6.5.6-0ubuntu18) ...
Selecting previously unselected package zfs-zed.
Preparing to unpack .../zfs-zed_0.6.5.6-0ubuntu18_amd64.deb ...
Unpacking zfs-zed (0.6.5.6-0ubuntu18) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up zfs-doc (0.6.5.6-0ubuntu18) ...
Setting up zfs-dkms (0.6.5.6-0ubuntu18) ...
Loading new zfs-0.6.5.6 DKMS files...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
First Installation: checking all kernels...
dpkg: warning: version '*-*' has bad syntax: version number does not start with digit
It is likely that 4.14.11-mainline-rev1 belongs to a chroot's host
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Building initial module for 4.14.11-mainline-rev1
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
configure: error:
    *** Please make sure the kmod spl devel <kernel> package for your
    *** distribution is installed then try again.  If that fails you
    *** can specify the location of the spl objects with the
    *** '--with-spl-obj=PATH' option.
Error! Bad return status for module build on kernel: 4.14.11-mainline-rev1 (x86_64)
Consult /var/lib/dkms/zfs/0.6.5.6/build/make.log for more information.
Setting up libuutil1linux (0.6.5.6-0ubuntu18) ...
Setting up libnvpair1linux (0.6.5.6-0ubuntu18) ...
Setting up libzpool2linux (0.6.5.6-0ubuntu18) ...
Setting up libzfs2linux (0.6.5.6-0ubuntu18) ...
Setting up zfsutils-linux (0.6.5.6-0ubuntu18) ...
zfs-import-cache.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-mount.service is a disabled or a static unit, not starting it.
A dependency job for zfs.target failed. See 'journalctl -xe' for details.
zfs.target couldn't start.
Setting up zfs-zed (0.6.5.6-0ubuntu18) ...
zed.service is a disabled or a static unit, not starting it.
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Processing triggers for systemd (229-4ubuntu21) ...
deploy@scaleway2:~$
deploy@scaleway2:~$
deploy@scaleway2:~$

Also I can't rollback to the previous bootscript, so currently ma machine got stuck.

I'm sure that same script worked perfectly before, maybe something is broken only for latest bootscripts?

caseycs commented 6 years ago

Same problem still exists with 4.14.12-mainline-rev1

caseycs commented 6 years ago

Turned out to be spl/kernel incompatibility. The only way I could make it work it 4.10.8-std-1 bootscript with Ubuntu 18.

simos commented 6 years ago

A workaround is to KEXEC the Ubuntu Linux kernel. See https://blog.simos.info/how-to-run-the-stock-ubuntu-linux-kernel-on-scaleway-using-kexec-and-server-tags/

I managed to manually compile ZFS from upstream, but it was too fragile to trust it.

michael-bouvy commented 6 years ago

@caseycs Ilia, how did you manage to get it work with this bootscript? SPL module compile fails on my server. Thanks!

caseycs commented 6 years ago

@michael-bouvy to be honest I gave up with zfs on scaleway few months ago, turned out to be too much hassle.

michael-bouvy commented 6 years ago

@caseycs thanks! Did you use LVM or other alternative?

caseycs commented 6 years ago

@michael-bouvy initially I was using zfs for both lxd and docker containers, however I migrated completely to docker, so have no need to lxd anymore. Do not have any performance issues with native built-in docker fs driver.

michael-bouvy commented 6 years ago

@caseycs ok, I'll have to find another way to be able to do snapshots of my LXC volumes

3ruce commented 5 years ago

It seems that the Start range doesn't have these kernel problems and a conventional ZFS install works fine...

simos commented 5 years ago

@3ruce Scaleway have fixed their images and now they use the stock Ubuntu Linux kernel, which has built-in ZFS support. The issue is that you need to launch a new VPS.