apt-get --reinstall install linux-headers-`uname -r` fails

[jojobo]

sudo apt-get --reinstall install linux-headers-uname -r fails, also "linux-image-extra-$(uname -r)" fails. Tried to reinstall kernel according to your docs, first it needed apt-get install gcc and apt-get install libssl-dev , otherwise the process would fail. Then it failed on 'make menuconfig KERNEL=3.18-std' with error No rule to make target `menuconfig'. Scaleway is turning such a headache for simply installing flynn.io which is supported on all other kvm vps.

[moul]

Hi @jojobo,

---

At boot, we are loading our own kernel before loading your disk, see https://www.scaleway.com/docs/bootscript-and-how-to-use-it/

So basically, if you apt-get install linux-headers, aptitude will download the package and write it on your disk, but it won't be taken into account because our bootscript is always loaded before your disk content

We are working on a boot on local kernel bootscript, but for now you still need to use our bootscripts

---

If you need a new module or special kernel configuration, do not hesitate to open an issue here, we are always happy to enhance our provided kernels

---

About the make menuconfig KERNEL=3.18-std, this Makefile is used to build new kernels (this is how we are building the Scaleway kernels), but it won't help you to update it on your server, the only way is to change your bootscript (see https://www.scaleway.com/docs/bootscript-and-how-to-use-it/)

Cheers

[jojobo]

@moul thanks for thorough answer. I'm trying to install flynn (flynn.io) and here's the issue https://github.com/flynn/flynn/issues/2696 . And here's a suggestion from flynn member https://github.com/flynn/flynn/issues/2709#issuecomment-213594407 .
How can I solve this now? I already tried all available bootscripts.

[moul]

Ok nice, can you just try to use the Docker bootscript then ? it is shipped with AUFS support

Else, can you give us instructions to reproduce your flynn installation error, so we can try to deeply investigate the error ?

[jojobo]

I'll try and let you know soon.

[jojobo]

I ran the installation manually without apt-get install -y "linux-headers-$(uname -r)" or "linux-image-extra-$(uname -r)", cause they return Unable to locate package linux-headers-4.5.1-docker-1 . But installation still fails with:

Setting up linux-headers-4.4.0-21-lowlatency (4.4.0-21.37~14.04.1) ...
Examining /etc/kernel/header_postinst.d.
run-parts: executing /etc/kernel/header_postinst.d/dkms 4.4.0-21-lowlatency /boot/vmlinuz-4.4.0-21-lowlatency
configure: error:
        *** Please make sure the kmod spl devel <kernel> package for your
        *** distribution is installed then try again.  If that fails you
        *** can specify the location of the spl objects with the
        *** '--with-spl-obj=PATH' option.
Error! Bad return status for module build on kernel: 4.4.0-21-lowlatency (x86_64)
Consult /var/lib/dkms/zfs/0.6.5.6/build/make.log for more information.

And then it exits with modprobe: FATAL: Module zfs not found.

Tried to install zfs manually in a couple of way, but modprobe zfs still returns module not found.

[moul]

Ok, so I need to get zfs into the kernel

Can you give me instructions about how you manually tried to install flynn so I can reproduce your environment ? it will be a lot easier for me to try my fixes

[jojobo]

To install manually, you need to run https://github.com/flynn/flynn/blob/v20160412.0/script/install-flynn.tmpl on server, WITHOUT apt-get install -y "linux-headers-$(uname -r)" and "linux-image-extra-$(uname -r)", and run apt-key adv --keyserver keyserver.ubuntu.com --recv E871F18B51E0147C77796AC81196BA81F6B0FC61. That's it.

[moul]

Thanks, I will give a try and tell you here how it is going

[josephglanville]

@moul is there a reason a non-vanilla kernel needs to be used? I imagine something in your netboot/remote disk infrastructure needs to be compiled in rather than loaded from initrd? I'm not familar with Scaleway so apologies if this something already discussed elsewhere.

If you were able to use vanilla kernels you could then just use the headers available in the distro packages to build DKMS modules like ZFS.

Failing that you should probably just also package your kernels for the distros so that people are able to install them to the local disk even if they won't be used so that kernel modules not required to boot the server can be built against them and stored on the network disk and modprobed after system startup.

Ofcourse being able to load the kernel from the disk image would be preferable but I would imagine this is somewhat more difficult without reasonably sophisticated boot infrastructure to first boot an initrd to mount the rootfs, parse the local bootloader configuration, get the guest kernel, push this to somewhere that you can get on the next reboot and cycle the machine. Possible but fairly complicated.

[moul]

@josephglanville

is there a reason a non-vanilla kernel needs to be used? I imagine something in your netboot/remote disk infrastructure needs to be compiled in rather than loaded from initrd?

I suggest you to look at this schema: https://github.com/scaleway/initrd/tree/master/Linux#schema-boot-timeline

In short: we are attaching your volumes (disks) using NBD which is not handled by the boot loaders (u-boot for C1, ipxe for VPS and C2*), so we can't get your kernel from your /boot folder and that's why we created bootscript to give you ability to use different kernels.

We are working on solutions to bypass this limitation so you can just boot using your local kernel

If you were able to use vanilla kernels you could then just use the headers available in the distro packages to build DKMS modules like ZFS.

That's exactly what I understood yesterday, I will try to add a script that sync everything needed to build new modules easily using DKMS, so I can add ZFS to fix this issue, but also improve the ease of customisation for every other usages

thanks for confirming the idea

Failing that you should probably just also package your kernels for the distros so that people are able to install them to the local disk even if they won't be used so that kernel modules not required to boot the server can be built against them and stored on the network disk and modprobed after system startup.

I tried to do it using the builtin make deb-pkg, but sadly it will only work for Debian/Ubuntu, so I'm not 100% satisfied and I'm looking for another solution that may work everywhere, if you have any idea

Of course being able to load the kernel from the disk image would be preferable but I would imagine this is somewhat more difficult without reasonably sophisticated boot infrastructure to first boot an initrd to mount the rootfs, parse the local bootloader configuration, get the guest kernel, push this to somewhere that you can get on the next reboot and cycle the machine. Possible but fairly complicated.

I have a proof of concept that first load our kernel, attach your disk using nbd, and then use kmod to switch to the disk kernel, but it's not ideal

We definitely need to add the "local boot" support

[jojobo]

Hi @moul ! Any progress yet on this issue ?

[moul]

@jojobo, no sorry, I still have issues releasing a kernel in a way dkms can work out of the box, as soon as I fix this, it would be a lot easier to build new modules and release zfs for this issue

[jojobo]

@moul thanks for working on this.

[bbraga]

@moul currently using DigitalOcean because I can deploy with Flynn there, will be waiting for this to be supported and to work out of the box so I can leverage Scaleway.

Thank you for taking this on :)

[moul]

@bbraga, ok :)

actually, we have successfuly built zfs module using dkms, but I didn't found a solution to package it with the kernel so basically you can have flynn running on Scaleway right now, but it may take about 1 hour to prepare the host before being able to install zfs :/

[moul]

Here is the $> history output of an host where I successfully built zfs: https://gist.github.com/moul/18e8ec495728e9db3c1e253719bd0d49

[moul]

I added a note about how to build modules using DKMS on the README.md,

I also created a new issue (#275) to try to do it from our kernel builder, so you can have the module directly without waiting for about 10 minutes

[jojobo]

@moul I went through your gist but it fails.

root@scw-1d76da:/usr/src/zfs-0.6.5.6/scripts# cat dkms.con
cat: dkms.con: No such file or directory
root@scw-1d76da:/usr/src/zfs-0.6.5.6/scripts# dkms
root@scw-1d76da:/usr/src/zfs-0.6.5.6/scripts# ^C
root@scw-1d76da:/usr/src/zfs-0.6.5.6/scripts#
root@scw-1d76da:/usr/src/zfs-0.6.5.6/scripts# apt-get install zfs-utils
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package zfs-utils
root@scw-1d76da:/usr/src/zfs-0.6.5.6/scripts# apt-get install zfsutils
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package zfsutils
root@scw-1d76da:/usr/src/zfs-0.6.5.6/scripts# apt-get install zfsutils-linux
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package zfsutils-linux

[moul]

Hi @jojobo, can you try this instead: https://github.com/scaleway/kernel-tools#how-to-build-a-custom-kernel-module ?

[jojobo]

@moul . I tried your suggestion, but install zfsutils-linux fails again.

  HOSTCC  scripts/conmakehash
  HOSTCC  scripts/sortextable
  HOSTCC  scripts/asn1_compiler
  HOSTCC  scripts/extract-cert
make: Leaving directory `/usr/src/linux-4.5.1'
root@scw-413b14:~# apt-get install zfsutils-linux
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package zfsutils-linux

I tried on TYPE VC1S instance, if that makes a difference.

[moul]

How ok, apparently your error is not related with the kernel stuff, but with aptitude

Can you ensure your local cache is update with apt-get update then run the apt-get install zfs-utils-linux ?

If it still doesn't work, can you give me the result of

• apt-cache search zfs
• cat /etc/scw-release
• lsb_release -a

Thanks

[jojobo]

Still not working @moul .

E: Unable to locate package zfsutils-linux
root@scw-451fff:~# apt-cache search zfs
parted - disk partition manipulator
bzflag-server - bzfs - BZFlag game server
zfs-fuse - ZFS on FUSE
root@scw-451fff:~# cat /etc/scw-release
IMAGE_FLAVORS="common docker-based feature-motd upstart"
IMAGE_TOOLS_CHECKOUT="764a58e59811d5f6217a3f71a4e009d6ec3e5138"
IMAGE_ID="Ubuntu Trusty"
IMAGE_RELEASE=2016-04-06
IMAGE_CODENAME=ubuntu
IMAGE_DESCRIPTION="Ubuntu Trusty (14.04 LTS)"
IMAGE_HELP_URL="https://community.scaleway.com"
IMAGE_SOURCE_URL="https://github.com/scaleway/image-ubuntu"
IMAGE_DOC_URL="https://scaleway.com/docs"
root@scw-451fff:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:        14.04
Codename:       trusty
root@scw-451fff:~#

[jgillich]

@jojobo Ubuntu 14.04 doesn't ship with zfs, only 16.04 does.

[moul]

Thanks @jgillich

@jojobo, can you try again on a more recent Ubuntu ?

[jojobo]

@moul, @jgillich , the whole point was to install flynn, and flynn recommended ubuntu 14.04. I don't know if they support ubuntu 16 yet.

[moul]

do you have a contact from the flynn team ? maybe they can help us to fix this the best way

[jojobo]

I think you already seen I raised this issue on flynn/flynn https://github.com/flynn/flynn/issues/2696 https://github.com/flynn/flynn/issues/2709 . The other participants are from flynn team.

[jgillich]

zfsutils-linux is the package name on 16.04, the flynn install script adds a PPA that has ubuntu-zfs (looks like both the userland and the kernel module, so install will probably fail).

[josephglanville]

@jgillich the ubuntu-zfs package builds the kernel modules using DKMS, even if it ships some pre-compiled modules those would be intended for use via DKMS caching mechanism.

Hailing from flynn/flynn here. Unfortunately I don't use Scaleway so it's harder for me understand why DKMS doesn't "just work".

[jojobo]

@moul so, do you think it's impossible to install flynn on scaleway for now?

[jojobo]

Hi @moul. I saw some new baremetal offers at scaleway. Does flynn work on any of those or same kernel issues there as well? Thanks.

[moul]

@jojobo

@moul so, do you think it's impossible to install flynn on scaleway for now?

As I successfully installed ZFS, yes I think we can install flynn

My concern was about the way we are offering the kernels, I would love to propose a pre-build ZFS kernel module to avoid these manual commands + waiting for some minutes for the kernel module build time

---

For now, I suggest you to setup your server to be DKMS-compatible using this guide: https://github.com/scaleway/kernel-tools#how-to-build-a-custom-kernel-module Then, run apt-get install zfsutils-linux And finally install flynn

[jojobo]

@moul . We already discussed this, flynn needs ubuntu 14.04, and your installation guide requires ubuntu 16.04. So we need zfs on 14.04, which fails for now.

[lmars]

This issue was originally opened as @jojobo was trying to install Flynn on Ubuntu 14.04, but Flynn can now be installed on 16.04 (so installs the zfsutils-linux package) so this may no longer be an issue.

[yoyostile]

@lmars Tried that today, unfortunately I failed. As far as I know we need aufs + zfs. aufs is available in the docker bootscript I think, but I wasn't able to build zfsutils there. On the other hand, I was able to build zfsutils for the std kernel - but kernel-extras were missing. Dunno, maybe it's my fault. :)

[josephglanville]

If it's Flynn you are trying to install it no longer needs AUFS, newer versions use OverlayFS instead.

[alessiodionisi]

Hello, I successfully installed Flynn on a C2L instance with Ubuntu 16.04. Build was really fast (10 minutes).

• Install prequisites: apt-get install build-essential libssl-dev
• Run this script: https://github.com/scaleway/kernel-tools#how-to-build-a-custom-kernel-module
• Now install and build zfs: apt-get install zfsutils-linux
• Enjoy: modprobe zfs
• Download the Flynn install script: https://dl.flynn.io/install-flynn
• Comment from line 172 to 179 (where Flynn try to install aufs-tools)
• Run the installer

[valkum]

As the Ubuntu 16.04 repo only contain zfs-0.6.5.6 which does not compile with >linux-4.8.0, this doesn't work well for docker systems, as the only bootscript available with docker support is 4.8.14 as of today.

Is it possible to provide a docker enabled kernel with the same linux version that is available in the xenial repositories as a bootscript?

[alessiodionisi]

I use the standard ubuntu xenial image on Scaleway with flynn

[valkum]

I don't plan to use this as a production system. I have some other services that i need to run there and these need docker support. As the standard scaleway kernel doesn't support docker (last time i checked) this is no alternative.

[alessiodionisi]

For Docker there is a specific image on Scaleway, try it

[valkum]

I'am using the docker image. 4.8.14-docker-2, but at least for 16.04 spl-dkms is failing to build because of various changes in the Linux api (see https://github.com/zfsonlinux/zfs/commit/68b8d22c6e3c30a59b20879efab1d46824686bdf)

I was able to get it working by using the spl-dkms, zfs-doc and zfs-dkms packages from the 16.10 repositories. But thats really dirty.

[xtagon]

I know I'm late to the party but if you all can get Flynn running on Scaleway, please make a blog post, you will be my heroes.

[ticruz38]

I followed these steps without errors

Install prequisites: apt-get install build-essential libssl-dev Run this script: https://github.com/scaleway/kernel-tools#how-to-build-a-custom-kernel-module Now install and build zfs: apt-get install zfsutils-linux Enjoy: modprobe zfs Download the Flynn install script: https://dl.flynn.io/install-flynn Comment from line 172 to 179 (where Flynn try to install aufs-tools) Run the installer

but get that error

modprobe: FATAL: Module zfs not found in directory /lib/modules/4.10.8-docker-1
===> 15:50:11.879 ERROR: failed to run "modprobe zfs", exit status 1

seems like zfs hasn't been installed at all

--Edited--

Could install Flynn by using Ubuntu 14.04 and installing apt-get install ubuntu-zfs Then commenting out the zfs install lines (153 - 170) in the Flynn installation script. running the custom installation script

Probably the easiest way to install flynn on scaleway.

[Jaysojitra13]

When I tried to install any new package in ubuntu, "E: The package software-properties-gtk needs to be reinstalled, but I can't find an archive for it.", this error is occurred.

Please help me to solve this problem. Thanks