DLPX-88183 Generate BTF data for ZFS kernel module during git-zfs-make

Details

Our Linux kernels have BTF data that allow us to use BTF info to resolve types and use kfunc (similar to fbt from DTrace) types of probes for all symbols+data from vmlinux and other modules from the kernel repo. Unfortunately this is not the case for our ZFS module which is an external out-of-tree module.

This is one of the two patches that I've created for enabling the generation of BTF data. In this patch I make sure to create a symlink for our Delphix VMs so whenever we compile ZFS on them (generally through zfs-make) we also generate BTF info for it which is later installed with zfs-load.

This means that once this patch gets merged and someone compiles ZFS on a brand-new VM the new module will have BTF info (see Future Steps for having BTF info at first boot).

Testing

ab-pre-push: http://selfservice.jenkins.delphix.com/job/appliance-build-orchestrator-pre-push/7171/ (and with the help of Prakash: http://selfservice.jenkins.delphix.com/job/delphix-build-and-snapshots/job/ami-snapshots/5115/flowGraphTable/)

Stock VM:

sudo bpftrace -e 'kfunc:zfs:spa_sync {printf("synced %d\n", args.txg);}'
stdin:1:1-19: ERROR: kfunc:spa_sync: no BTF data for spa_sync
kfunc:zfs:spa_sync {printf("synced %d\n", args.txg);}

Using an internal-dev VM cloned from the group created by the above ab-pre-push link, I verified that the symlink was created:

delphix@ip-10-110-242-135:~$ ls -lh /usr/src/linux-headers-5.15.0-1045-dx2023092017-974a77aa4-aws/vmlinux
lrwxrwxrwx 1 root root 66 Oct  3 15:49 /usr/src/linux-headers-5.15.0-1045-dx2023092017-974a77aa4-aws/vmlinux -> /usr/lib/debug/boot/vmlinux-5.15.0-1045-dx2023092017-974a77aa4-aws

Then I ran zfs-make with develop on that VM and ensured that the BTF info was generated during compilation:

  MODPOST /export/home/delphix/zfs/module/Module.symvers
  CC [M]  /export/home/delphix/zfs/module/spl.mod.o
  CC [M]  /export/home/delphix/zfs/module/zfs.mod.o
  LD [M]  /export/home/delphix/zfs/module/spl.ko
  LD [M]  /export/home/delphix/zfs/module/zfs.ko
  BTF [M] /export/home/delphix/zfs/module/spl.ko
  BTF [M] /export/home/delphix/zfs/module/zfs.ko

I ensured that the BTF info was installed after zfs-load and were placed in the appropriate runtime directory under /sys:

delphix@ip-10-110-242-135:~$ sudo readelf -S /usr/lib/debug/lib/modules/5.15.0-1045-dx2023092017-974a77aa4-aws/extra/zfs.ko | grep BTF
  [67] .BTF              PROGBITS         0000000000000000  04b6944b

delphix@ip-10-110-242-135:~$ file /sys/kernel/btf/zfs
/sys/kernel/btf/zfs: data

Then I verified that we can now use kfunc functions and their arguments:

delphix@ip-10-110-242-135:~$ sudo bpftrace -e 'kfunc:zfs:spa_sync {printf("synced %d\n", args.txg);}'
Attaching 1 probe...
synced 1974
synced 1975
synced 1976
synced 1977
^C

Future Steps

I'll soon be opening another patch in linux-pkg that will make sure that our first compilation of ZFS (in the bootstrap VM through linux-pkg - e.g. not the Delphix VM) generated a module with BTF info. That patch will help us having BTF info for ZFS in customer VMs too.

Alternative designs

The ideal design for this whole process would be to somehow install that symlink when we install the the linux-headers-* package. Unfortunately the vmlinux file that we need that has the debug info is part of the main linux-image-*-dbgsym package which basically installs the whole kernel (not just the headers) which is harder to do in our bootstrap VM which can compile multiple versions and flavors of the linux kernel in a single run. It would also create an awkward dependency between the two packages. For these two reasons I decided to go with the above two patch approach for the appliance-build and linux-pkg.

delphix / appliance-build