Falco won't install on some Ubuntu 20.04 (linux 5.4.0) cluster nodes with PPM_IOCTL_GET_API_VERSION complaint

[jrabbit]

Describe the bug If you install falco using helm chart v 1.19.1 and falco 0.32.0 some nodes will never be ready

How to reproduce it

helm install falco falcosecurity/falco starts, but certain pods will never stop being in restart loop reporting "Runtime error: Kernel module does not support PPM_IOCTL_GET_API_VERSION. Exiting." after failing to build the module with gcc but success with other versions of gcc.

Expected behavior Ideally falco loads a working driver.

Illustrative screenshot Environment

• Falco version: 0.32.0
• System info: from working pod on working node

  # falco --support | jq .system_info
  Fri Jun 17 16:31:10 2022: Falco version 0.32.0 (driver version 39ae7d40496793cf3d3e7890c9bbdc202263836b)
  Fri Jun 17 16:31:10 2022: Falco initialized with configuration file /etc/falco/falco.yaml
  Fri Jun 17 16:31:10 2022: Loading rules from file /etc/falco/falco_rules.yaml:
  Fri Jun 17 16:31:10 2022: Loading rules from file /etc/falco/falco_rules.local.yaml:
  {
  "machine": "x86_64",
  "nodename": "falco-2xls8",
  "release": "5.4.0-1038-aws",
  "sysname": "Linux",
  "version": "#40-Ubuntu SMP Fri Feb 5 23:50:40 UTC 2021"
  }

• Cloud provider or hardware configuration: AWS
• OS: Ubuntu 20.04

  NAME="Ubuntu"
  VERSION="20.04.2 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.2 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

• Kernel: Linux <cut> 5.4.0-1038-aws #40-Ubuntu SMP Fri Feb 5 23:50:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Installation method: helm install falco falcosecurity/falco

Additional context

Originally from #1941 but doesn't involve minikube

[jasondellaluce]

This usually happens if you are running a recent version of Falco with an older version of the kernel drivers. The way to fix this should be to uninstall you kernel modules and update them.

[jrabbit]

We can't. Falco keeps spawning new binaries even when unscheduled from the cluster. The module is constantly in use and can't be rmmod'ed. Killing falco then immediately rmmoding doesn't even fix the problem, just loads a new(?) broken driver. e: Further problems with that diagnosis is this failed on brand new nodes that never ran Falco before.

[jrabbit]

Found more falco instances running, seems to work right once they were removed