search

falcosecurity / libs

libsinsp, libscap, the kernel module driver, and the eBPF driver sources
https://falcosecurity.github.io/libs/
Apache License 2.0
212 stars 159 forks source link

Support Linux Security Modules hook for more accurate security tracing #252

Open LucaGuerra opened 2 years ago

LucaGuerra commented 2 years ago

Motivation

Currently, Falco traces every supported system call using kernel tracepoints, both in the kernel module and eBPF probe. This works well, but in some specific cases it can lead to potential security issues related to rule bypasses such as the ones that have been recently mitigated:

Some of these topics have been reported by Rex Guo and Junyuan Zeng in their presentation and addressed by the Falco community.

While it may be possible to fix or mitigate every demonstrated techniques that allow these kind of rule bypass, it could become hard to address all cases since more bypass techniques may be discovered in the future and I believe it's worth it to introduce features to mitigate this entire class of issues.

Feature

In the presentation linked above, researchers have suggested to use Linux Security Modules hooks. In fact, in the kernel there are many security_* functions that can be traced via kprobes or eBPF even without the need to build an entire Linux Security Module. Those are guaranteed to have the right file names, connection endpoints that the kernel processed without the need to access userspace buffers. This would allow us to:

  1. Create a new class of events (security events) that trace these functions, where available. They don't necessarily map 1:1 to system calls, but for some use cases they may be enough to build interesting rules
  2. Use the data coming from security events to enrich syscall tracing in libsinsp, adding symlink resolution and so removing the class of issues mentioned above

Note that such a solution, while straightforward in theory, can be tricky to implement because we need to analyze its impact on Falco's performance and we need to understand which kernel versions we can support, which is why I think we need community discussion around it. On the flip side, it will be an addition to Falco and so Falco will continue to operate whether it's available or not.

Alternatives

It's often possible to work around potential bypasses like we did recently, but I feel that a full solution, properly developed in the open is the way to go.

Additional context

Some of the most relevant hooks to implement:

Andreagit97 commented 2 years ago

Hi @LucaGuerra :hand: I completely agree with you. These TOCTOU attacks are more and more widespread and trying to put patches here and there no longer seems like a viable situation. As you said, LSM hooks could help us not only with these attacks but can also enrich our state in libsinsp with security-related information. I see a lot of value in having this feature! :rocket:

dwindsor commented 2 years ago

Create a new class of events (security events) that trace these functions, where available. They don't necessarily map 1:1 to system calls, but for some use cases they may be enough to build interesting rules

With respect to enriching state, if we're thinking about adding additional tracepoints into the security subsystem, we could even possibly add LSM-specific hooks. For instance, lockdown_write() for the Lockdown LSM, or security_compute_xperms_decision() for SELinux.

loresuso commented 2 years ago

Hi @LucaGuerra, I really support introducing security events coming from Linux Security Modules. As you said, we will need to attach to these hooks via kprobes (when using the kmod), and at the beginning, I was a little bit concerned about it, especially because of performance. Kprobes in the past were implemented via debug instructions (INT3 in Intel architectures) and because of that, they were extremely slow. Then I started investigating more and I discovered that nowadays they are optimized via jump/call instructions, making the kernel instrumentation for these events much lighter and similar to what we have with the tracepoints we already have in terms of performance. This optimization takes place if the kernel is compiled with CONFIG_OPTPROBES, which seems to be the case for most distributions.

I also agree with @dwindsor, once we have instrumented the kernel for security events related to syscall, we can go further and attach to LSM-specific hooks. I also think that there are many other places we can hook in the kernel. Since we will use kprobes, we could possibly attach one to the commit_creds function. When thinking about container escapes or other kinds of attacks that are performed via the kernel, one of the most commonly used payloads is commit_creds(prepara_kernel_creds(0)). If we attach a kprobe on that function, we could possibly detect if a legal call was performed, otherwise, the call originated as a result of an attack to escalate privileges. I've seen that this payload is pretty used, and it would be something useful to catch potential zero-days. Of course, there may be many other interesting places where we can hook, but we can think about that later.

Finally, I just wanted to say that I am up for implementing this, and I already started doing some experiments! Thank you!

dwindsor commented 2 years ago

file_mprotect() would also be useful to add a probe to, for checking whether an attacker is trying to manipulate the heap to possibly make it executable:

SEC("lsm/file_mprotect")
int BPF_PROG(mprotect_audit, struct vm_area_struct *vma, unsigned long reqprot, unsigned long prot, int ret)
{
    int is_heap = (vma->vm_start >= vma->vm_mm->start_brk && vma->vm_end <= vma->vm_mm->brk);
        int is_exe_perms = (reqprot | S_IXUSR) || (reqprot | S_IXGRP) || (reqprot | S_IXOTH);

    /* Making the heap executable is suspicious. */
    if (is_heap && is_exe_perms) {
        /* Do something */
    }
}
loresuso commented 2 years ago

Yeah, I agree with you, that's why I also introduced support for the mprotect syscall and I suggested very similar stuff here not so long ago. However, I think that these kinds of checks must not be embedded in the kernel, we have to have that logic in the rules somehow, in my opinion, but that is something that deserves to be implemented! Also, I wanted to say that for our use case is better to use LSM only for audit and create events, not actually block malicious actions.

dwindsor commented 2 years ago

However, I think that these kinds of checks must not be embedded in the kernel, we have to have that logic in the rules somehow, in my opinion, but that is something that deserves to be implemented!

Agreed, I like the idea of monitoring for memory protection transitions in general, rather than special-casing something like this.

poiana commented 2 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

Andreagit97 commented 2 years ago

/remove-lifecycle stale

poiana commented 1 year ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

Andreagit97 commented 1 year ago

/remove-lifecycle stale

incertum commented 1 year ago

Bump: Has anyone started experimentation towards supporting a new LSM hook interface?

Would it be the right timing to start prioritizing an initial solution? For instance, starting with security_bprm_check and security_file_open that would cover the initially most crucial use cases?

Andreagit97 commented 1 year ago

Ei @incertum unfortunately I had no time to start investigating it but I've talked with @FedeDP about a possible roadmap to reach this, let me know if it seems reasonable to you:

  1. extending the framework test of the modern probe to the old drivers (kmod, old bpf probe)
  2. apply a libbpf approach also to the old probe, maybe we can port also the CO-RE approach :thinking:
  3. introduce instrumentation like kprobe, lsm hooks at least in bpf and modern bpf probes (not sure we want to introduce them also in the kmod, it could be tricky and dangerous since the wide portability of the kmod (kernel versions from 2.36 to 6.0.6 :upside_down_face: )

Let's say the (2) point requires the (1) to avoid disasters in production, and the (3) should require the (2) if we don't want to go mad in reading the elf section manually and create another logic flow to manage kprobes instead of tracepoints that in any case it won't be a clean solution unless we rewrite almost all from scratch...

Let's say we can start some experimentation with LSM hooks in parallel with (1) and (2) just to see where to hook and what we want to collect. Maybe we can do that in the modern probe, where it should be easy to implement it from scratch and we can simply backport the logic in the old one hoping that the (2) point will be mature enough to allow us an easy and smooth backporting :) WDYT? @LucaGuerra @incertum @FedeDP

Andreagit97 commented 1 year ago

BTW as some of you said here https://github.com/falcosecurity/libs/pull/705 I think it's time to try some other form of instrumentation to mitigate these kinds of issues

dwindsor commented 1 year ago

security_bprm_check is not a great starting point because its struct linux_binprm parameter is very difficult to serialize and will have to be flattened.

That said, I went ahead and implemented support for security_file_mprotect (tagging @loresuso since we discussed this earlier in the thread)..This function also takes a complex parameter (struct vm_area_struct), but I was able to sanely flatten it (I think).

I've done this work in #717.

incertum commented 1 year ago

Huh, Christmas is starting early this year ❤️ , you rock @dwindsor, way to go!

  1. extending the framework test of the modern probe to the old drivers (kmod, old bpf probe)

This would be fantastic, all about the next generation falco-fuzzer test framework

  1. apply a libbpf approach also to the old probe, maybe we can port also the CO-RE approach 🤔

Why not?

  1. introduce instrumentation like kprobe, lsm hooks at least in bpf and modern bpf probes (not sure we want to introduce them also in the kmod, it could be tricky and dangerous since the wide portability of the kmod (kernel versions from 2.36 to 6.0.6 🙃 )

Sharing the same sentiment re the dinosaur kernels (< 4.14) and kmod in general. Therefore, would support focusing on bpf for the lsm hooks at first.

Let's say we can start some experimentation with LSM hooks in parallel with (1) and (2) just to see where to hook and what we want to collect. Maybe we can do that in the modern probe, where it should be easy to implement it from scratch and we can simply backport the logic in the old one hoping that the (2) point will be mature enough to allow us an easy and smooth backporting :) WDYT? @LucaGuerra @incertum @FedeDP

This is a very good idea to start experimenting with LSM hooks using the modern_bpf. Shall we open another issue with a list of candidate LSM hooks similar to how you did it for the missing syscalls? Get the first one up as template for others to use? With this approach we can also get the userspace logic in place and see if and how the state engine could be augmented as well.

dwindsor commented 1 year ago

Shall we open another issue with a list of candidate LSM hooks similar to how you did it for the missing syscalls?

A good starting point would be these I think:

Andreagit97 commented 1 year ago

This is a very good idea to start experimenting with LSM hooks using the modern_bpf. Shall we open another issue with a list of candidate LSM hooks similar to how you did it for the missing syscalls? Get the first one up as template for others to use? With this approach we can also get the userspace logic in place and see if and how the state engine could be augmented as well

Yeah, that's exactly the idea, having something working with the modern probe and all the userspace stuff in place so we could easily introduce the bpf code in the old probe when it will be ready. Maybe we can use this issue since the name is exactly what we are looking for :thinking: I will modify the initial description made by @LucaGuerra adding @dwindsor hooks (that you for this list, it seems a reasonable starting point!!)

dwindsor commented 1 year ago

(that you for this list, it seems a reasonable starting point!!)

Sure! fwiw it mirrors the list of Tracee's instrumented LSM hooks. We could wholesale just copy their hooks, but for security_file_mprotect, I noticed that there were additional fields that would be useful (vm_mm->start_code, vm_mm->end_code, vm_mm->start_data, vm_mm->end_data, etc.)

incertum commented 1 year ago

Just bumping: LSM hooks shall be prioritized early next year for modern_bpf at first as discussed, CC @Andreagit97

incertum commented 1 year ago

Great news, things are moving, https://github.com/falcosecurity/libs/pull/937/ marks the start of a refactor that will enable supporting LSM hooks events. The complete refactor will be split across a few PRs and the next PR after 937 will merge ppm_sc_codes and ppm_tp_codes and can be considered the critical PR to look out for to ensure LSM hooks integration and operationalization is possible without requiring another refactor.

Early feedback from everyone is very much appreciated, thank you! Will keep this issue up to date and post each relevant PR here as well.

incertum commented 1 year ago

Hey everyone 👋 this is the next PR to watch out for https://github.com/falcosecurity/libs/pull/963. @FedeDP opened the PR as early WIP. Important would be to decide which approach to take. Federico contrasts two possibilities. Lastly, tests and testing will follow ...

dwindsor commented 1 year ago

thanks for the updates @incertum! I prefer #963 over #889 =).

Either way, once either one of those lands and the location for the source of the kprobes themselves has been determined, we should be able to immediately start in on the above list of LSM hooks. The procedure for adding a new kprobe seems similar enough to the procedure for adding a new syscall that folks should be able to use existing documentation to figure it out.

Andreagit97 commented 1 year ago

The procedure for adding a new kprobe seems similar enough to the procedure for adding a new syscall that folks should be able to use https://github.com/falcosecurity/libs/pull/781to figure it out.

Let's say yes and no :laughing, I thought a little bit about it: in kernels >=5.5 we have fentry/fexit programs, optimized kprobes that can use BTF, so probably in the modern probe I would go for them. The bad news is that these programs are not supported on s390x and arm64 (in arm64 they were introduced in kernel 6.0 IIRC). Another bad news is that if we use plain kprobes we must have this kernel config enabled CONFIG_OPTPROBES otherwise they will kill perf :/ and as far as I know again arm64 and s390x don't support this config :( And moreover, we have to think very well about how to instrument them, I would avoid collecting something like 27 params as in execve filler, I would keep them quite simple with just the fundamental info, and for doing that we need deep research on what we want to collect before doing it! The integration in the modern probe should be quite easy but as I said most of the time the real pain point is how to sync this info with our actual system state derived from syscall :/ BTW we will find a way to address all these concerns!

dwindsor commented 1 year ago

in kernels >=5.5 we have fentry/fexit programs, optimized kprobes that can use BTF, so probably in the modern probe I would go for them. The bad news is that these programs are not supported on s390x and arm64 (in arm64 they were introduced in kernel 6.0 IIRC). Another bad news is that if we use plain kprobes we must have this kernel config enabled CONFIG_OPTPROBES otherwise they will kill perf :/ and as far as I know again arm64 and s390x don't support this config :(

Ahhh very good points. Would it make sense to opportunistically use the optimized kprobes when possible, falling back to plain kprobes if CONFIG_OPTPROBES is enabled?

And moreover, we have to think very well about how to instrument them, I would avoid collecting something like 27 params as in execve filler, I would keep them quite simple with just the fundamental info, and for doing that we need deep research on what we want to collect before doing it!

Agreed, but in the execve filler we're only collecting a few params? Did you mean that new tracepoints/kprobes should be more careful about how many params to collect? Because some of the LSM hooks have massive number of paramaters that could be pushed

Andreagit97 commented 1 year ago

Ahhh very good points. Would it make sense to opportunistically use the optimized kprobes when possible, falling back to plain kprobes if CONFIG_OPTPROBES is enabled?

yeah probably this will be the plan :)

Did you mean that new tracepoints/kprobes should be more careful about how many params to collect?

Yep exactly we should push only real needed params otherwise we will only increase the complexity of the program and requested time for instrumentation... today in some syscall we send parameters that are not so useful to userspace processing and I would avoid this approach here since we are starting from scratch! If we want to capture something we must have in mind what to do with this param and how to take advantage of it in userspace

dwindsor commented 1 year ago

Totally agreed! With these also, IIUC we won't have enter/exit events, which also influenced our choice about which parameters to push.

dwindsor commented 1 year ago

Yeah, I agree with you, that's why I also introduced support for the mprotect syscall and I suggested very similar stuff https://github.com/falcosecurity/libs/pull/174 not so long ago. However, I think that these kinds of checks must not be embedded in the kernel, we have to have that logic in the rules somehow, in my opinion, but that is something that deserves to be implemented! Also, I wanted to say that for our use case is better to use LSM only for audit and create events, not actually block malicious actions.

Wow, it's been a year since we talked about this @loresuso!

And we've all had a year to think about it =). Do you still feel like a userspace engine to make decisions about memory mapping transitions is the best way forward here? When I first was thinking about this, it made sense, but certain types of ops are always going to be suspicious. For instance, attempting to change the protection of a segment from X -> WX will always be suspicious, even on systems with W^X. I feel like we could come up with a list of a few of these transitions that will be suspicious everywhere?

  1. !X -> X
  2. X -> WX
  3. WX -> X

Also, I wanted to say that for our use case is better to use LSM only for audit and create events, not actually block malicious actions.

Heheh, totally fair =). If blocking were to be done, though, it would most effectively be done atomically in LSM. Could you see, for instance, a userspace engine that could supply the libs with a list of pathnames that should never be executed and/or IP addresses to connect to, etc. This seems not performant enough to do in the kernel, but just thinking out loud..

poiana commented 7 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

Andreagit97 commented 7 months ago

/remove-lifecycle stale

poiana commented 4 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

incertum commented 4 months ago

/remove-lifecycle stale

We will get there eventually, I'll bring it up again in the next core maintainer meeting.

poiana commented 1 month ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

Andreagit97 commented 1 month ago

/remove-lifecycle stale