JPLachance
commented
3 years ago Greetings!
In our environment, Falco was reporting a lot of non-actionable weird events. For example:
"rule": "Non sudo setuid",
"output_fields": {
"proc.pname": null,
"proc.cmdline": "<NA>",
"user.uid": 4294967295,
"k8s.pod.name": null,
"evt.time": 1600013430060232000,
"evt.arg.uid": "root",
"container.id": "host",
"k8s.ns.name": null,
"user.name": null,
"container.image.repository": null
}This event was not telling us anything useful.
We started an investigation with @leogr and we rapidly made the connection with dropped system calls. To really see the problem, we adjusted the rate and max_burst of syscall_event_drops events. Falco was dropping thousands of system calls:
"rule": "Falco internal: syscall event drop",
"output_fields": {
"n_evts": "1113206",
"n_drops_bug": "0",
"n_drops_pf": "0",
"ebpf_enabled": "0",
"n_drops": "621240",
"n_drops_buffer": "621240"
}Our average of dropped system calls was over 1,000,000 every minute.
Our first experiment was to get rid of all Falco rules that are in the rules file, but disabled. It did not fix our problem.
Ou second experiment was to get rid of the Kubernetes integration (by removing -K, -k, and -pk options). Falco's performance improved, but it was only temporary. After a few hours, Falco was re-dropping thousands of system calls.
Then we removed a program_output and Falco almost completely stopped dropping system calls. We still see small drops, but nothing like before.
Our program output was a simple bash script calling aws sns publish to publish events to SNS. The program_output is blocking the main Falco threat that processes system calls. Since the aws sns publish call is slow, Falco falls behind, the buffer between the kernel probe and the user space gets full and Falco drops system calls. Dropping system calls generates a Falco internal: syscall event drop event, which calls aws sns publish which slows down Falco even more. We were creating a self feeding loop of Falco internal: syscall event drop events.
We went back to the drawing board and replaced our program_output by an http_output to the Falco Sidekick service. The http_output is still "blocking" on the main Falco thread, but since the Sidekick replies very quickly, the impact is not noticeable and Falco is able to process system calls without having to drop them.
A permanent fix for theses issues is to make all Falco outputs non-blocking. Falco should not block himself while trying to report us security events 😄
For the drops we still have, they might be caused by increases in the number of system calls to process due to activities in our environment. We might need to dig deeper, get statistics on the number of system calls processed by Falco and statistics on peaks, pin point what is generating those peaks.
At least, weird non-actionable events are gone! 🎉
That's all folks!
leogr
Issif
minhvo3
After read more about n_drops_pf, I decided to deploy falco to production. The first cluster is least traffic cluster and i got many n_drops_buffer alerts. So I will give your team some debugging information.
My cluster use GKE v1.16.13-gke.401, OS: Ubuntu 18.04.5 LTS and Container-Optimized OS.
Resource limitation: 1cpu and 2G Ram (in fact never reached this limit)
Falco v0.25.0
Parameter:
I only use /etc/falco/falco_rules.local.yaml(remove all rule, do not add any new rules)
webserver.enabled = false and i test one by one output type(syslog, file, stdout, http)
Only syslog output:
Only file output:
Only stdout output:
Only http output:
=> n_drops_buffer alerts has same characteristics: 18 syscalls drop per 1 minute(devide to 1, 2, or 3 alerts) and continuously
I think it relate to GKE, traffic can't drop so steadily. I will deploy 1 more cluster to get more information and check about this regularity.
I hope this report is useful to you, now i will remove alert for drop syscall event.
Have a nice day!
Updating: I deployed another cluster and got the same result.
Additional information: number of dropped syscalls per minute depend on version of GKE
DingGGu
poiana
leodido
Andreagit97
jasondellaluce
maynardflies
mimor
Why this issue?
When Falco is running, a producer (a.k.a the driver) continuously forwards events to a consumer (the Falco userspace program) with a buffer that sits in the middle. When - for any reason - the consumer is not able to consume the incoming events, then an events drop occurs.
Starting from v0.15.0, Falco introduced a mechanism to detect dropped events and take actions, as explained in the official documentation. However, events drop is still an issue as reported by many users.
Since the problem depends on many factors and can be hard to analyze and understand, this issue aims to give users an overview of that and collect the knowledge acquired until now.
Please, note that this document does not try to directly solve this problem and consider that some assumptions might be wrong (feel free to confute them!).
N.B. At the time of writing, it was not clear the outcome of this issue, so I just chose to label it as "documentation".
Event drop alerts
Currently, when dropped events are detected, Falco will print out some statistics that can give users some information about the kind of drops that happened.
An example of an alert regarding dropped events:
Note that the statistics reported above are relative to the timeframe specified (the last second) and not cumulative. Furthermore, note that an event represents just a single syscall received by the driver, regardless of whether a rule is triggered or not.
So, what do those values mean?
ebpf_enabledindicates whenever the driver is the eBPF probe (=1) or a kernel module (=0)n_dropsis the sum of othersn_drop_*fields (see the section below) and represents the total number of dropped eventsn_evtsis the number of events that the driver should send according to its configuration. It also includesn_dropssincen_dropsis the number of events that the driver should send to userspace but is not able to send due to various reasons.Also note that in extreme cases, drop alerts may be rate-limited, so consider incrementing those values in the configuration file, for example:
Kind of drops
As you can notice, not all drops are the same. Below an explanation for each kind of them (ordered by the less frequent to the most one).
n_drops_bugis the number of dropped events caused by an invalid condition in the kernel instrumentation, something went wrong basically. AFAIK, only the eBPF probe can generate this kind of drop, and luckily there are no reports of this problem.n_drops_pf(wherepfstands for page fault) is the number of dropped events caused by invalid memory access; it happens whenever the memory page (referenced by the syscall) had disappeared before the driver was able to collect information about the event. We noticed that rarely, it sometimes happens on GKE, and it is related to some process that is continuously crashing (see #1309).n_drops_bufferis the number of dropped events caused by a full buffer (the buffer sits between the producer and the consumer). It's the most frequent one, and it's related to performance. We have also different categories of buffer drops to understand which syscall triggered them (e.g.n_drops_buffer_clone_fork_exit,n_drops_buffer_connect_enter, ...)Performance-related drops (n_drops_buffer)
We experience this kind of event dropping when the consumer is blocked for a while (note that the task that consumes events is single-threaded). That is strictly related to performance and can happen for several reasons. We also added a benchmark command in the event-generator to experiment with this problem (see https://github.com/falcosecurity/event-generator/pull/36 for more details).
Possible causes:
Limited CPU resource
The consumer hits the maximum CPU resources allocated for it and gets blocked for a while. For example, the official Helm chart comes with a 200m CPU hard limit that may cause this problem.
Large/complex ruleset (high CPU usage)
The larger and more complex the ruleset, the more CPU will be needed. At some point, either with or without resource limitation, high CPU usage can produce event dropping.
Fetching metadata from external components (I/O blocking)
In some cases, fetching metadata (e.g., container information, k8s metadata) from an external component can be a blocking operation.
For example, the
--disable-cri-asyncflag is quite explanatory about that:Another option that might cause problem is:
Slow responses from the Kubernetes API server could cause this problem too.
Blocking output (I/O blocking)
Falco outputs mechanism can also have an impact and might block the event processing for a while, producing drops.
The buffer size
If you are not able to solve your drop issues you can always increase the syscall buffer size (the shared buffer between userspace and kernel that contains all collected data). You can find more info on how to change its dimension in the Falco config file
Debugging
When debugging, the first time to consider is that multiple causes may occur simultaneously. It is worth excluding every single cause, one by one.
Once
n_drops_bugandn_drops_pfcases are excluded, for the performance-related drops (ie.n_drops_buffer) a handy checklist is:-Aoption, if any--disable-cri-asyncoption, if any-Uoption, if any-k https://$(KUBERNETES_SERVICE_HOST)(instead of-k https://kubernetes.default, see this comment)webserver.enabledto false in the config file and removing any other related configuration-K,-k, and-pkoptions)stdout_output(event drop alerts still show up)Finally, some useful links that could help with debugging:
Interesting issues about drops https://github.com/falcosecurity/falco/issues/669, https://github.com/falcosecurity/falco/issues/669#issuecomment-570824776, https://github.com/falcosecurity/falco/issues/669#issuecomment-642000182, https://github.com/falcosecurity/falco/issues/961, https://github.com/falcosecurity/falco/issues/1382, https://github.com/falcosecurity/falco/issues/615, https://github.com/falcosecurity/falco/issues/1231, https://github.com/falcosecurity/falco/issues/558
Issues related to
n_drops_pfhttps://github.com/falcosecurity/falco/issues/917, https://github.com/falcosecurity/falco/issues/770, https://github.com/falcosecurity/falco/issues/669#issuecomment-641466585Some threads on our Slack channel https://kubernetes.slack.com/archives/CMWH3EH32/p1592904527372800, https://kubernetes.slack.com/archives/CMWH3EH32/p1599741275086000
Drop related to the K8s support (
-K,-k, and-pkoptions) https://github.com/falcosecurity/falco/issues/2129