This repository contains the code used to build the Ubuntu-based Delphix Appliance, leveraging open-source tools such as Debian's live-build, Docker, Ansible, OpenZFS, and others.
Apache License 2.0
19
stars
41
forks
source link
DLPX-88183 Generate BTF data for ZFS kernel module during git-zfs-make #740
Our Linux kernels have BTF data that allow us to use BTF info to resolve types and use kfunc (similar to fbt from DTrace) types of probes for all symbols+data from vmlinux and other modules from the kernel repo. Unfortunately this is not the case for our ZFS module which is an external out-of-tree module.
This is one of the two patches that I've created for enabling the generation of BTF data. In this patch I make sure to create a symlink for our Delphix VMs so whenever we compile ZFS on them (generally through zfs-make) we also generate BTF info for it which is later installed with zfs-load.
This means that once this patch gets merged and someone compiles ZFS on a brand-new VM the new module will have BTF info (see Future Steps for having BTF info at first boot).
I'll soon be opening another patch in linux-pkg that will make sure that our first compilation of ZFS (in the bootstrap VM through linux-pkg - e.g. not the Delphix VM) generated a module with BTF info. That patch will help us having BTF info for ZFS in customer VMs too.
Alternative designs
The ideal design for this whole process would be to somehow install that symlink when we install the the linux-headers-* package. Unfortunately the vmlinux file that we need that has the debug info is part of the main linux-image-*-dbgsym package which basically installs the whole kernel (not just the headers) which is harder to do in our bootstrap VM which can compile multiple versions and flavors of the linux kernel in a single run. It would also create an awkward dependency between the two packages. For these two reasons I decided to go with the above two patch approach for the appliance-build and linux-pkg.
Details
Our Linux kernels have BTF data that allow us to use BTF info to resolve types and use
kfunc
(similar tofbt
from DTrace) types of probes for all symbols+data fromvmlinux
and other modules from the kernel repo. Unfortunately this is not the case for our ZFS module which is an external out-of-tree module.This is one of the two patches that I've created for enabling the generation of BTF data. In this patch I make sure to create a symlink for our Delphix VMs so whenever we compile ZFS on them (generally through
zfs-make
) we also generate BTF info for it which is later installed withzfs-load
.This means that once this patch gets merged and someone compiles ZFS on a brand-new VM the new module will have BTF info (see Future Steps for having BTF info at first boot).
Testing
ab-pre-push: http://selfservice.jenkins.delphix.com/job/appliance-build-orchestrator-pre-push/7171/ (and with the help of Prakash: http://selfservice.jenkins.delphix.com/job/delphix-build-and-snapshots/job/ami-snapshots/5115/flowGraphTable/)
Stock VM:
Using an internal-dev VM cloned from the group created by the above ab-pre-push link, I verified that the symlink was created:
Then I ran
zfs-make
withdevelop
on that VM and ensured that the BTF info was generated during compilation:I ensured that the BTF info was installed after
zfs-load
and were placed in the appropriate runtime directory under/sys
:Then I verified that we can now use
kfunc
functions and their arguments:Future Steps
I'll soon be opening another patch in
linux-pkg
that will make sure that our first compilation of ZFS (in the bootstrap VM through linux-pkg - e.g. not the Delphix VM) generated a module with BTF info. That patch will help us having BTF info for ZFS in customer VMs too.Alternative designs
The ideal design for this whole process would be to somehow install that symlink when we install the the
linux-headers-*
package. Unfortunately thevmlinux
file that we need that has the debug info is part of the mainlinux-image-*-dbgsym
package which basically installs the whole kernel (not just the headers) which is harder to do in our bootstrap VM which can compile multiple versions and flavors of the linux kernel in a single run. It would also create an awkward dependency between the two packages. For these two reasons I decided to go with the above two patch approach for the appliance-build and linux-pkg.