Falco reports almost no event details in some alerts.

jozseffarago commented 5 years ago

What happened: Falco send alerts where only the alert-triggering detail is usable, most of the other fields are NAs. This only happens in some cases, but seemingly for the same events. Other events are reported correctly.

What you expected to happen: Falco can include all the event specific information to the alert.

How to reproduce it (as minimally and precisely as possible): Unfortunately do not know.

Anything else we need to know?: Falco is running with the kernel module, instead of the EBPF program. Tried to enable EBPF but it was neither able to compile the module, nor download it from s3, due to bad URL.

Environment:

Falco version (use falco --version):

0.17.0

System info

{
"machine": "x86_64",
"nodename": "falcoservice-5dr9j",
"release": "4.14.133-113.112.amzn2.x86_64",
"sysname": "Linux",
"version": "#1 SMP Tue Jul 30 18:29:50 UTC 2019"
}

Cloud provider or hardware configuration:

AWS EKS, on Spotinst ocean.

OS (e.g: cat /etc/os-release):

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Kernel (e.g. uname -a):

4.14.133-113.112.amzn2.x86_64
Install tools (e.g. in kubernetes, rpm, deb, from source):

Helm chart

Others: Falco config:

# Default values for falco.
image:
registry: docker.io
repository: falcosecurity/falco
tag: 0.17.0
pullPolicy: IfNotPresent

docker:
enabled: true
socket: /var/run/docker.sock

cri:
enabled: true
socket: /run/containerd/containerd.sock

resources:
# Although resources needed are subjective on the actual workload we provide
# a sane defaults ones. If you have more questions or concerns, please refer
# to Sysdig Support for more info about it
requests:
  cpu: 100m
  memory: 512Mi
limits:
  cpu: 200m
  memory: 1024Mi

priorityClassName: high-priority

extraArgs: []

rbac:
# Create and use rbac resources
create: true

podSecurityPolicy:
# Create a podSecurityPolicy
create: false

serviceAccount:
# Create and use serviceAccount resources
create: true
# Use this value as serviceAccountName
name:

fakeEventGenerator:
enabled: false
replicas: 1

daemonset:
# Perform rolling updates by default in the DaemonSet agent
# ref: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
updateStrategy:
  # You can also customize maxUnavailable, maxSurge or minReadySeconds if you
  # need it
  type: RollingUpdate

# If is behind a proxy you can set the proxy server
proxy:
httpProxy:
httpsProxy:
noProxy:

# Set daemonset timezone
timezone:

ebpf:
# Enable eBPF support for Falco
enabled: false

settings:
  # Needed to enable eBPF JIT at runtime for performance reasons.
  # Can be skipped if eBPF JIT is enabled from outside the container
  hostNetwork: true
  # Needed to correctly detect the kernel version for the eBPF program
  # Set to false if not running on Google COS
  mountEtcVolume: true

Missing event details examples:

02:26:53.997991631: Error File below a known binary directory opened for writing (user=<NA> command=<NA> file=/usr/sbin/zic parent=<NA> pcmdline=<NA> gparent=<NA> container_id=host image=<NA>) proc.exe=<NA> proc.name=<NA> (proc.args=) proc.exeline=<NA>  (thread.cgroups=<NA>) proc.exepath= (evt.args=fd=3(<f>/usr/sbin/zic) dirfd=-100(AT_FDCWD) name=/usr/sbin/zic flags=4102(O_CREAT|O_WRONLY|O_CLOEXEC) mode=0755 dev=10301 ) evt.res=SUCCESS evt.type=openat k8s.ns=<NA> k8s.pod=<NA> container=host

06:41:02.437336448: Notice Unexpected setuid call by non-sudo, non-root program (user=<NA> cur_uid=4294967295 parent=<NA> command=<NA> uid=root container_id=host image=<NA> proc.exe=<NA> proc.name=<NA> (proc.args=) proc.exeline=<NA>  (thread.cgroups=<NA>) proc.exepath= (evt.args=uid=0(root) ) evt.res=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host

05:34:25.850458382: Notice Known system binary sent/received network traffic (user=<NA> command=cat header.yaml.fragment - connection=192.168.99.108:57678->52.218.192.1:443 container_id=68af675e46b7 image=783721547467.dkr.ecr.us-east-1.amazonaws.com/featureswitch-configmap-updater proc.exe=cat proc.name=cat (proc.args=header.yaml.fragment -) proc.exeline=cat header.yaml.fragment - (thread.cgroups=cpuset=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 cpu=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 cpuacct=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 blkio=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 memory=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 devices=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 freezer=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 net_cls=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 perf_event=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 net_prio=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 hugetlb=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6 pids=/kubepods/burstable/pode4364509-cd41-11e9-a600-1202c166ab34/68af675e46b783edf0ef8ef9fc0834c42a80a617fd01100ee065ece713dadcf6) proc.exepath= (evt.args=res=-115(EINPROGRESS) tuple=192.168.99.108:57678->52.218.192.1:443 ) evt.res=EINPROGRESS) k8s.ns=<NA> k8s.pod=<NA> container=68af675e46b7

jozseffarago commented 5 years ago

One more details which maybe relevant. As we are on Amazon linux, we have auditd enabled by default. Maybe it messes up Falco somehow. (we had this experience when we did some trial with commercial products, which has similar kernel modules)

fntlnz commented 5 years ago

This one is a bit difficult to tackle since it seems to be related to your specific environment. What I would do here is to try to use sysdig directly and see what it reports.

For example, to emulate the rule about setuid call by non-sudo you could do:

sysdig -j evt.type=setuid

That will dump all the setuid events (both directions) in the current machine.

If in the meanwhile you have any setuid the output should look like the following:

{"evt.cpu":3,"evt.dir":">","evt.info":"uid=1000(fntlnz) ","evt.num":52513,"evt.outputtime":1568426611382001092,"evt.type":"setuid","proc.name":"a.out","thread.tid":224713}
{"evt.cpu":3,"evt.dir":"<","evt.info":"res=0 ","evt.num":52514,"evt.outputtime":1568426611382007072,"evt.type":"setuid","proc.name":"a.out","thread.tid":224713}

fntlnz commented 5 years ago

/triage needs-information

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

falcosecurity / falco

Falco reports almost no event details in some alerts. #812