search

falcosecurity / falco

Cloud Native Runtime Security
https://falco.org
Apache License 2.0
7.42k stars 903 forks source link

Container information missing in the alerts generated by the "Privileged Shell Spawned in Container" rule. #3345

Open dmeetreey opened 1 month ago

dmeetreey commented 1 month ago

Hello,

My team and I have recently encountered an issue where alerts triggered by the "Privileged Shell Spawned in Container" rule are missing container-related details.

Example of Alert Output:

Critical Privileged Shell Spawned in Container (user.uid=0 proc.cmdline=sh proc.name=sh proc.pname=<NA> gparent=<NA> ggparent=<NA> gggparent=<NA> proc.pcmdline=<NA> user.name=root user.loginuid=-1 proc.args= container.name=<NA> evt.type=execve evt.res=SUCCESS proc.pid=2121961 proc.cwd= proc.ppid=2121895 proc.sid=1 proc.exepath=/bin/sh user.loginname=<NA> group.gid=0 group.name=root, container.image=<NA>)
container_id=827accfc01b7 
container_image=<NA> 
container_image_tag=<NA> 
container_name=<NA> 
k8s_ns=<NA> 
k8s_pod_name=<NA>

Rule Configuration:

- rule: Privileged shell spawned inside container
  enabled: true
  desc: This rule detects the creation of a shell as root for interaction within a container. If this rule fires, it may be an indication of compromise.
  condition: spawned_process and container and shell_procs and user.uid = 0 and (proc.args = "" or proc.args startswith "-i") and not container_entrypoint and not user_shell_container_exclusions and not user_expected_terminal_shell_in_container_conditions and proc.tty = 0
  exceptions:
  [REDACTED]
  output: Privileged Shell Spawned in Container (user.uid=%user.uid proc.cmdline=%proc.cmdline proc.name=%proc.name proc.pname=%proc.pname gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] proc.pcmdline=%proc.pcmdline user.name=%user.name user.loginuid=%user.loginuid proc.args=%proc.args container.name=%container.name evt.type=%evt.type evt.res=%evt.res proc.pid=%proc.pid proc.cwd=%proc.cwd proc.ppid=%proc.ppid proc.sid=%proc.sid proc.exepath=%proc.exepath user.loginname=%user.loginname group.gid=%group.gid group.name=%group.name, container.image=%container.image.repository)
  priority: CRITICAL
  tags: [container, shell, mitre_container_administration_command, mitre_execution, CIS]

Environment:

Problem: The missing container information makes it difficult to troubleshoot and correlate alerts with the actual containers.

Expected Behavior: We expect the alert output to include detailed container information such as container name, image, image tag, Kubernetes namespace, and pod name.

Additional Context: It’s critical for us to have complete information in alerts for effective incident response and troubleshooting.

Request: Could you please provide guidance on how to resolve this issue or if it’s a known issue of Falco? Any suggestions or workarounds would be greatly appreciated.

Thank you for your support!

dmeetreey commented 1 month ago

/kind bug

UgniusV commented 1 month ago

Hey @poiana any updates for this?

leogr commented 1 month ago

Hey @dmeetreey

Can you share more details about the running environment (OS, Kernel version, etc..) and the setup (installation method, config, etc)?

Thanks

doublez13 commented 1 month ago

I'm also seeing fields missing from certain alerts. Some have them populated while others don't.

For instance, popping a shell into a terminal triggers the following alert, but the fields are usually null (but not always).

{
  "hostname": "REDACTED",
  "output": "18:44:49.495227176: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=www-data user_uid=33 user_loginuid=-1 process=bash proc_exepath=/usr/bin/bash parent=systemd command=bash terminal=34816 exe_flags=EXE_LOWER_LAYER container_id=d4758580aec2 container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)",
  "output_fields": {
    "container.id": "d4758580aec2",
    "container.image.repository": null,
    "container.image.tag": null,
    "container.name": null,
    "evt.arg.flags": "EXE_LOWER_LAYER",
    "evt.time": 1729190689495227176,
    "evt.type": "execve",
    "k8s.ns.name": null,
    "k8s.pod.name": null,
    "proc.cmdline": "bash",
    "proc.exepath": "/usr/bin/bash",
    "proc.name": "bash",
    "proc.pname": "systemd",
    "proc.tty": 34816,
    "user.loginuid": -1,
    "user.name": "www-data",
    "user.uid": 33
  },
  "priority": "Notice",
  "rule": "Terminal shell in container",
  "source": "syscall",
  "tags": [
    "T1059",
    "container",
    "maturity_stable",
    "mitre_execution",
    "shell"
  ],
  "time": "2024-10-17T18:44:49.495227176Z"
}

I've got falco deployed via helm with the following settings...

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

collectors:
  enabled: true
  containerd:
    enabled: true
    socket: /run/containerd/containerd.sock
  docker:
    enabled: false
  crio:
    enabled: false

#As mentioned in the docs, I've disabled async, but am still seeing null values
extra:
  args:
    - '--disable-cri-async'

#I've got a few custom rules not included
Falco version: 0.39.1 (x86_64)
System info: Linux version 6.1.0-25-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26)
UgniusV commented 1 month ago

Can you share more details about the running environment (OS, Kernel version, etc..) and the setup (installation method, config, etc)?

@leogr

sure:

OS Info

root@falcosecurity-falco-sandbox1-lpjn6:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel version:

root@falcosecurity-falco-sandbox1-lpjn6:/# uname -a
Linux falcosecurity-falco-sandbox1-lpjn6 6.1.94-1.el8.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 20 08:31:35 UTC 2024 x86_64 GNU/Linux

config:

base_syscalls:
  custom_set: []
  repair: false
buffered_outputs: true
config_files:
- /etc/falco/config.d
engine:
  kind: modern_ebpf
  modern_ebpf:
    buf_size_preset: 8
    cpus_for_each_buffer: 2
    drop_failed_exit: false
falco_libs:
  thread_table_size: 262144
file_output:
  enabled: false
  filename: ./events.txt
  keep_alive: false
grpc:
  bind_address: unix:///run/falco/falco.sock
  enabled: true
  threadiness: 0
grpc_output:
  enabled: true
http_output:
  ca_bundle: ""
  ca_cert: ""
  ca_path: /etc/falco/certs/
  client_cert: /etc/falco/certs/client/client.crt
  client_key: /etc/falco/certs/client/client.key
  compress_uploads: false
  echo: false
  enabled: true
  insecure: false
  keep_alive: false
  mtls: false
  url: http://falcosecurity-falcosidekick:80/
  user_agent: falcosecurity/falco
json_include_output_property: true
json_include_tags_property: true
json_output: true
libs_logger:
  enabled: false
  severity: debug
load_plugins:
- k8saudit
- json
log_level: notice
log_stderr: true
log_syslog: true
metrics:
  convert_memory_to_mb: true
  enabled: true
  include_empty_values: false
  interval: 1h
  kernel_event_counters_enabled: true
  libbpf_stats_enabled: true
  output_rule: true
  resource_utilization_enabled: true
  rules_counters_enabled: true
  state_counters_enabled: true
output_timeout: 2000
outputs:
  max_burst: 1000
  rate: 0
outputs_queue:
  capacity: 0
plugins:
- init_config:
    maxEventSize: "125829120"
    webhookMaxBatchSize: "125829120"
  library_path: libk8saudit.so
  name: k8saudit
  open_params: http://:9765/k8s-audit
- init_config: ""
  library_path: libjson.so
  name: json
priority: debug
program_output:
  enabled: false
  keep_alive: false
  program: 'jq ''{text: .output}'' | curl -d @- -X POST https://hooks.slack.com/services/XXX'
rule_matching: first
rules_files:
- /etc/falco/common-rules.yaml
stdout_output:
  enabled: true
syscall_event_drops:
  actions:
  - log
  - alert
  max_burst: 1
  rate: 0.03333
  simulate_drops: false
  threshold: 0.1
syscall_event_timeouts:
  max_consecutives: 1000
syslog_output:
  enabled: true
time_format_iso_8601: false
watch_config_files: true
webserver:
  enabled: true
  k8s_healthz_endpoint: /healthz
  listen_port: 8765
  prometheus_metrics_enabled: true
  ssl_certificate: /etc/falco/falco.pem
  ssl_enabled: false

Installation method: modern eBPF

leogr commented 1 month ago

These are obvious questions, but they may still help:

@doublez13 have you checked the socket filepath exits?

collectors:
  enabled: true
  containerd:
    enabled: true
    socket: /run/containerd/containerd.sock

@alacuku the unixsocket may exist, still may not be accessible for some reason since we are using leastPrivileged: true? :thinking: 

driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

@doublez13 it would be helpful to know if leastPrivileged: false solved the issue. If you can try, let us know :pray:

leogr commented 1 month ago

@UgniusV

doublez13 commented 1 month ago

@doublez13 have you checked the socket filepath exits?

The socket file on the host is at /run/containerd/containerd.sock, which then appears to get mounted in the falco container at /host/run/containerd/containerd.sock. Let me know If I need to change the helm value.

@doublez13 it would be helpful to know if leastPrivileged: false solved the issue. If you can try, let us know 🙏

Ah setting leastPrivileged: false got all the pod information! I've reverted the setting to leastPrivileged: true and confirmed that the values are null again, so that setting seems to be the culprit.

leogr commented 1 month ago

leastPrivileged: true

Thank you so much! I hope we are close to finding a solution.

@doublez13 May I ask you for another try? I suspect that we need another capability to make Falco read the container socket properly.

The missing capability is CAP_DAC_READ_SEARCH. You can add it using the containerSecurityContext option in the helm chart. So, if possible for you, give it a try with:

containerSecurityContext:
  securityContext:
    capabilities:
      add:
        - BPF
        - SYS_RESOURCE
        - PERFMON
        - SYS_PTRACE
        - CAP_DAC_READ_SEARCH
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

Thanks

dmeetreey commented 1 month ago

Hi, @leogr!

@UgniusV

  • have you used Helm? Or have you deployed Falco directly on a host using the package (deb/rpm/tar) installation?
  • I can't see the container_engines in your configuration. So, it looks like Falco isn't configured to collect container metadata.
  1. Yes, we are using Helm.
  2. Updated the config with container_engines, will check if this solves the issue!
leogr commented 1 month ago

Hey @dmeetreey

Thank you!

Just one doubt:

  1. Yes, we are using Helm.
  2. Updated the config with container_engines, will check if this solves the issue!

The helm chart should correctly set the container_engines configuration when collectors are enabled. So I don't understand why you must manually update container_engines. Am I missing something? :thinking:

doublez13 commented 1 month ago

@leogr

The missing capability is CAP_DAC_READ_SEARCH. You can add it using the containerSecurityContext option in the helm chart. So, if possible for you, give it a try with:

Unfortunately I'm not seeing any difference between the following two configs.



containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
#      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true



containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

Both have the container fields displayed as follows: (container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)

dmeetreey commented 1 month ago

Hey @dmeetreey

Thank you!

Just one doubt:

  1. Yes, we are using Helm.
  2. Updated the config with container_engines, will check if this solves the issue!

The helm chart should correctly set the container_engines configuration when collectors are enabled. So I don't understand why you must manually update container_engines. Am I missing something? 🤔

@leogr I guess I misunderstood what you meant. If I understand correctly now, it is enough to set necessary collectors to true instead of container_engines to start logging the container metadata? Couldn't find it in the docs, so just making sure.

leogr commented 1 month ago

If I understand correctly now, it is enough to set necessary collectors to true instead of container_engines to start logging the container metadata?

If you are deploying Falco via Helm, you should only use collectors setting in the helm values. The chart will configure Falco's config for you. On the other hand, if you're not using Helm, you should configure container_engines by yourself.

So, if you haven't enabled collectors in the Helm chart, please try to enable them (note that they are enabled by default; also, please check that the container unix socket path matches your environment).

If you have collectors enabled already, I would have expected to see the container configuration translated to the container_engines setting in Falco config.

I'm unsure what is happening in your case since I did not see container_engines in Falco's config you posted above.

Hoping my explanation makes sense now :innocent:

leogr commented 1 month ago

@leogr

The missing capability is CAP_DAC_READ_SEARCH. You can add it using the containerSecurityContext option in the helm chart. So, if possible for you, give it a try with:

Unfortunately I'm not seeing any difference between the following two configs.

containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
#      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true
containerSecurityContext:
  capabilities:
    add:
      - BPF
      - SYS_RESOURCE
      - PERFMON
      - SYS_PTRACE
      - CAP_DAC_READ_SEARCH

tty: true
driver:
  kind: modern_ebpf
  modernEbpf:
    leastPrivileged: true

Both have the container fields displayed as follows: (container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)

@doublez13

Is AppArmor enabled on your system? If yes, this should be the only possible cause. If this is the case, we don't have a solution at this time, so I recommend you use leastPrivileged: false for now.

To address this in the long term, I will open a specific issue for the least privileged issue with the Helm chart (or feel free to open it by you if you want).

doublez13 commented 1 month ago

Is AppArmor enabled on your system? If yes, this should be the only possible cause. If this is the case, we don't have a solution at this time, so I recommend you use leastPrivileged: false for now.

To address this in the long term, I will open a specific issue for the least privileged issue with the Helm chart (or feel free to open it by you if you want).

@leogr

AppArmor seems to be the cause! If I run the pod with least leastPrivileged: true and manually set the apparmor profile to unconfined the pod fields are filled. Thanks for helping to track this down!

leogr commented 3 weeks ago

Is AppArmor enabled on your system? If yes, this should be the only possible cause. If this is the case, we don't have a solution at this time, so I recommend you use leastPrivileged: false for now. To address this in the long term, I will open a specific issue for the least privileged issue with the Helm chart (or feel free to open it by you if you want).

@leogr

AppArmor seems to be the cause! If I run the pod with least leastPrivileged: true and manually set the apparmor profile to unconfined the pod fields are filled. Thanks for helping to track this down!

Good to know, thank you!

I guess we need both to update the documentation and find a way to force AppArmor profile to unconfined from the pod manifest (not sure if possible). Adding this to my todo list :innocent: /assign

leogr commented 3 weeks ago

Just noticed @doublez13 was already working on the chart, awesome! https://github.com/falcosecurity/charts/pull/769