[UMBRELLA] How do you deploy Falco in production?

[leogr]

Motivation

This issue is currently a WIP. It aims to collect feedback on how users deploy Falco in their environment.

By reading this slack thread, I've noticed that not all users prefer using our official Falco distribution. Instead, some may choose to internally build Falco packages/images (for example, see this and this).

As a maintainer, I'd like to know if there's any common need that we are missing. :thinking:
So, I've created this issue as a starting point to collect user feedback.

If you are a Falco production user, please answer a few questions :point_down: and share your experience! Your feedback is precious to us! :pray:

- Do you use:
  - [ ] only officially released version?
  - [ ] customized build from source code

- How do you install Falco in production?
  - [ ] Using an official DEB/RPM/TAR.GZ package
  - [ ] Using an official docker image
  - [ ] Building Falco packages/images by yourself (if yes, explain your need)
  - [ ] Other? (explain your need)

- If you deploy Falco on K8s, what's your installation method?
  - [ ] Using the official Falco [Helm chart](https://github.com/falcosecurity/charts/tree/master/falco)
  - [ ] Custom manifest files based on [official examples](https://github.com/falcosecurity/deploy-kubernetes/tree/main/kubernetes)
  - [ ] Other? (explain your need)

- Is any Falco feature you disabled or didn't want to use in production? If yes, Why?

- Is any specific deployment need currently unmet by the official Falco distribution?

- Briefly describe your deployment scenario, if possible.

/kind documentation

Additional context

TODO

[happy-dude]

• Do you use:

  • [x] customized build from source code

• How do you install Falco in production?

  • [x] Building Falco packages/images by yourself (if yes, explain your need)
  • better supply-chain control from source to installation of Falco; this was mitigation for certain risks assessed in a threat model
  • unattended install experience during deployment of package via config management tool; change made to not prompt for probe selection since we use the bpf-probe (also avoids dependencies that the probe needs), and also avoids making a network connection to see if a pre-built probe is available upstream to download (we deploy a custom kernel, and same supply-chain concerns apply to reaching for upstream probes)
  • better control of dependencies overall (use bundled deps, not depending on dkms while using bpf-probe)
  • creating a non-stripped build with some debug symbols (see also https://github.com/falcosecurity/falco/issues/2268)
  • applying bugfixes or features from upstream that are not in release version
  • investigating fixes to issues discovered internally (before bubbling it upstream)

• Is any Falco feature you disabled or didn't want to use in production? If yes, Why?

  • the driver selection post-install script is unnecessary when we know which probe to use beforehand and have pre-built probes internally

• Is any specific deployment need currently unmet by the official Falco distribution?

  • falco-exporter, falcosidekick are not available in the an apt-repo, requiring publishing of packages internally
  • a lot of Falco-related projects assume k8s/helm distributions

[incertum]

• Do you use:

  • [x] customized build from source code

• How do you install Falco in production?

  • [x] Building Falco packages/images by yourself (if yes, explain your need)
  • Echoing much of what @Happy-Dude shared above.
  • Also creating .tar.gz and .rpm from scratch. What Stanley mentioned "not depending on dkms when using eBPF" is something that perhaps could be fixed upstream.
  • Also have all eBPF probes pre-compiled and published in an internal artifacts store (plus CI automation to trigger new eBPF probes builds when new custom kernel builds are published internally -> applies to old bpf only), no fall-back to build on host (in fact this is something that should not be done in production and also callouts to the internet should be avoided)
  • Custom artifacts verification mechanisms.
  • Have Falco binary running in an infinite loop vs upstream entrypoint suggestion -> this is both for deployment robustness and being able to log binary restarts.
  • Lastly I also have feature gates (custom pre-init script) for minor config changes for a particular subset of clusters
  • Have custom metrics (using bpftool) as well -> native Falco support for performance metrics is already in flight.
  • A subset of new features are more urgent internally than others. For those urgent ones I directly diverge even if it's a very large diff and for the most part I then propose those features to the community through PRs -> once merged upstream I often already have the next feature running internally in preview mode and the cycle repeats, so it's mostly because of speed and higher urgency for some features than for others (one such more urgent example is the currently open PR around container.ip and container.cni.json -> it is a crucial part of incident response runbooks and lookups already).

• If you deploy Falco on K8s, what's your installation method?

  • [x] Other? -> custom ecosystem

• Is any Falco feature you disabled or didn't want to use in production? If yes, Why?

  • Current major pain point is the not yet finalized syscalls activations / selections mechanism through the Falco rules and configs (but we are working on it and it will land soon), so I have the static simple set + sinsp state enforcement disabled and replaced with the solutions in the respective PRs.
  • One past pain point included the static buffer size, but this is now configurable 🎉 and works great!

• Is any specific deployment need currently unmet by the official Falco distribution? N/A since we need everything custom given a particular ecosystem.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[jasondellaluce]

/remove-lifecycle rotten

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Andreagit97]

/remove-lifecycle stale

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Andreagit97]

/remove-lifecycle stale

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Andreagit97]

/remove-lifecycle stale

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

[poiana]

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2376#issuecomment-2439474678): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.