Missing CONFIG_TRACEPOINTS in linux-esx kernels

[deepskyblue86]

I was trying to have Falco running on Photon OS with linux-esx kernel but it can't work because of the esx kernel config.

root@photon-1d270acad7c [ ~ ]# falco-driver-loader
* Running falco-driver-loader for: falco version=0.29.1, driver version=17f5df52a7d9ed6bb12d3b1768460def8439936d
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to load a system falco module, if present
* Looking for a falco module locally (kernel 4.19.198-1.ph3-esx)
* Trying to download a prebuilt falco module from https://download.falco.org/driver/17f5df52a7d9ed6bb12d3b1768460def8439936d/falco_photon_4.19.198-1.ph3-esx_1.ko
curl: (22) The requested URL returned error: 404 Not Found
Unable to find a prebuilt falco module
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
'/tmp/falco-dkms-make'....
cleaning build area...

DKMS: build completed.

falco.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/4.19.198-1.ph3-esx/kernel/extra/

depmod...

DKMS: install completed.
* falco module installed in dkms, trying to insmod
* Unable to insmod falco module
Consider compiling your own falco driver and loading it or getting in touch with the Falco community

So I decided to check the module:

root@photon-1d270acad7c [ ~ ]# insmod /lib/modules/4.19.198-1.ph3-esx/kernel/extra/falco.ko.xz
insmod: ERROR: could not insert module /lib/modules/4.19.198-1.ph3-esx/kernel/extra/falco.ko.xz: Unknown symbol in module
root@photon-1d270acad7c [ ~ ]# cat /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/4.19.198-1.ph3-esx/x86_64/log/make.log 
DKMS make.log for falco-17f5df52a7d9ed6bb12d3b1768460def8439936d for kernel 4.19.198-1.ph3-esx (x86_64)
Wed Aug 11 09:09:40 UTC 2021
'/tmp/falco-dkms-make' -C /lib/modules/4.19.198-1.ph3-esx/build M=/var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build modules
make[1]: Entering directory '/usr/src/linux-headers-4.19.198-1.ph3-esx'
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/main.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/dynamic_params_table.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/fillers_table.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/flags_table.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/ppm_events.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/ppm_fillers.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/event_table.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/syscall_table.o
  CC [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/ppm_cputime.o
  LD [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/falco.o
  Building modules, stage 2.
  MODPOST 1 modules
WARNING: "tracepoint_probe_register" [/var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/falco.ko] undefined!
WARNING: "for_each_kernel_tracepoint" [/var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/falco.ko] undefined!
WARNING: "tracepoint_probe_unregister" [/var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/falco.ko] undefined!
  CC      /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/falco.mod.o
  LD [M]  /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/falco.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.19.198-1.ph3-esx'

I searched for tracepoint_probe_register being undefined and I found this. So I checked the kernel config:

root@photon-1d270acad7c [ ~ ]# grep -e CONFIG_TRACEPOINTS -e CONFIG_HAVE_SYSCALL_TRACEPOINTS /boot/config-4.19.198-1.ph3-esx
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y

root@photon-1d270acad7c [ ~ ]# grep -e CONFIG_TRACEPOINTS -e CONFIG_HAVE_SYSCALL_TRACEPOINTS /boot/config-4.19.132-6.ph3 
CONFIG_TRACEPOINTS=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y

Even with other esx versions, CONFIG_TRACEPOINTS is not there, while it's available in all the other variants. Is it expected? Was it done on purpose?

[srivatsabhat]

Hi @deepskyblue86

linux-esx is Photon's kernel flavor optimized for ESXi, and we want to keep the kernel small (in size) and performant, for fast boot-time and small memory footprint. That’s likely why debug options such as tracepoints are disabled in linux-esx.

Is there a specific reason you need to use the linux-esx kernel? Can you perhaps use the generic kernel flavor (linux) for your usecase?

[balinaveen]

Is there a specific reason you need to use the linux-esx kernel? Can you perhaps use the generic kernel flavor (linux) for your usecase?

We want to expand the environments where falco can run to include linux-esx to enable users of linux-esx kernel to get falco support.

[YustasSwamp]

Photon OS builds and tests falco with generic linux kernel only.