hangrymuppet
commented
9 months ago Updating the policy to have just 1 event still results in all events being shown in the logs.
Closed hangrymuppet closed 9 months ago
hangrymuppet
commented
9 months ago Updating the policy to have just 1 event still results in all events being shown in the logs.
NDStrahilevitz
commented
9 months ago @geyslan net packet is derived and security_socket_connect can be derived, sounds like a possible relation. If we can reproduce I think we should also release a minor with this fixed WDYT?
geyslan
commented
9 months ago If we can reproduce I think we should also release a minor with this fixed WDYT?
@NDStrahilevitz absolutely.
geyslan
commented
9 months ago I wasn't able to reproduce it locally, either in the global or in the container scope - using this single policy.
This
sudo ./dist/tracee --config examples/config/global_config.yaml -s container -e stdio_over_socket,net_packet_http_request
returned only expected:
{"timestamp":325706186936,"threadStartTime":325705730394,"processorId":2,"processId":8,"cgroupId":24903,"threadId":8,"parentProcessId":1,"hostProcessId":504379,"hostThreadId":504379,"hostParentProcessId":503762,"userId":0,"mountNamespace":4026534646,"pidNamespace":4026534649,"processName":"nc","executable":{"path":""},"hostName":"hb0","containerId":"c4908f5d0086bcd349366028811050833066cda685d6e440bf8c6eadaa6f300d","container":{"id":"c4908f5d0086bcd349366028811050833066cda685d6e440bf8c6eadaa6f300d","name":"trusting_pare","image":"alpine:latest","imageDigest":"alpine@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b"},"kubernetes":{},"eventId":"6000","eventName":"stdio_over_socket","matchedPolicies":[""],"argsNum":4,"returnValue":0,"syscall":"dup2","stackAddresses":null,"contextFlags":{"containerStarted":true,"isCompat":false},"threadEntityId":3042714030,"processEntityId":3042714030,"parentEntityId":206480808,"args":[{"name":"IP address","type":"const char *","value":"::1"},{"name":"Port","type":"const char *","value":"4444"},{"name":"File descriptor","type":"int","value":0},{"name":"triggeredBy","type":"unknown","value":{"args":[{"name":"oldfd","type":"int","value":3},{"name":"newfd","type":"int","value":0},{"name":"remote_addr","type":"struct sockaddr*","value":{"sa_family":"AF_INET6","sin6_addr":"::1","sin6_flowinfo":"0","sin6_port":"4444","sin6_scopeid":"0"}}],"id":748,"name":"socket_dup","returnValue":0}}],"metadata":{"Version":"2","Description":"A process has its standard input/output redirected to a socket. This behavior is the base of a Reverse Shell attack, which is when an interactive shell being invoked from a target machine back to the attacker's machine, giving it interactive control over the target. Adversaries may use a Reverse Shell to retain control over a compromised target while bypassing security measures like network firewalls.","Tags":null,"Properties":{"Category":"execution","Kubernetes_Technique":"","Severity":3,"Technique":"Unix Shell","external_id":"T1059.004","id":"attack-pattern--a9d4b653-6915-42af-98b2-5758c4ceee56","signatureID":"TRC-101","signatureName":"Process standard input/output over socket detected"}}}
{"timestamp":325706191314,"threadStartTime":325705730394,"processorId":2,"processId":8,"cgroupId":24903,"threadId":8,"parentProcessId":1,"hostProcessId":504379,"hostThreadId":504379,"hostParentProcessId":503762,"userId":0,"mountNamespace":4026534646,"pidNamespace":4026534649,"processName":"nc","executable":{"path":""},"hostName":"hb0","containerId":"c4908f5d0086bcd349366028811050833066cda685d6e440bf8c6eadaa6f300d","container":{"id":"c4908f5d0086bcd349366028811050833066cda685d6e440bf8c6eadaa6f300d","name":"trusting_pare","image":"alpine:latest","imageDigest":"alpine@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b"},"kubernetes":{},"eventId":"6000","eventName":"stdio_over_socket","matchedPolicies":[""],"argsNum":4,"returnValue":0,"syscall":"dup2","stackAddresses":null,"contextFlags":{"containerStarted":true,"isCompat":false},"threadEntityId":3042714030,"processEntityId":3042714030,"parentEntityId":206480808,"args":[{"name":"IP address","type":"const char *","value":"::1"},{"name":"Port","type":"const char *","value":"4444"},{"name":"File descriptor","type":"int","value":1},{"name":"triggeredBy","type":"unknown","value":{"args":[{"name":"oldfd","type":"int","value":0},{"name":"newfd","type":"int","value":1},{"name":"remote_addr","type":"struct sockaddr*","value":{"sa_family":"AF_INET6","sin6_addr":"::1","sin6_flowinfo":"0","sin6_port":"4444","sin6_scopeid":"0"}}],"id":748,"name":"socket_dup","returnValue":0}}],"metadata":{"Version":"2","Description":"A process has its standard input/output redirected to a socket. This behavior is the base of a Reverse Shell attack, which is when an interactive shell being invoked from a target machine back to the attacker's machine, giving it interactive control over the target. Adversaries may use a Reverse Shell to retain control over a compromised target while bypassing security measures like network firewalls.","Tags":null,"Properties":{"Category":"execution","Kubernetes_Technique":"","Severity":3,"Technique":"Unix Shell","external_id":"T1059.004","id":"attack-pattern--a9d4b653-6915-42af-98b2-5758c4ceee56","signatureID":"TRC-101","signatureName":"Process standard input/output over socket detected"}}}This
sudo ./dist/tracee --config examples/config/global_config.yaml -s container -e stdio_over_socket,net_packet_http_request,stdio_over_socket,k8s_api_connection,aslr_inspection,proc_mem_code_injection,docker_abuse,scheduled_task_mod,ld_preload,cgroup_notify_on_release,default_loader_mod,sudoers_modification,sched_debug_recon,system_request_key_mod,cgroup_release_agent,rcd_modification,core_pattern_modification,proc_kcore_read,proc_mem_access,hidden_file_created,anti_debugging,ptrace_code_injection,process_vm_write_inject,disk_mount,dynamic_code_loading,fileless_execution,illegitimate_shell,kernel_module_loading,k8s_cert_theft,proc_fops_hooking,syscall_hooking,dropped_executable,container_create,container_remove
also returned only what was expected without confusion between host and containers.
I also tested the config and policies above in minikube and all worked ok.
@NDStrahilevitz if you are able to reproduce it, please post the steps.
NDStrahilevitz
commented
9 months ago @geyslan I suspect from his issue that this might be an openshift issue. Note that the kernel is 4.18.0-513.5.1.el8_9.x86_64. I'll test this on such an env for now. @hangrymuppet Am I correct in my guess?
hangrymuppet
commented
9 months ago @NDStrahilevitz this is not specifically an openshift issue; we don't use openshift.
We are running on a supported RHEL kernel and we deploy our own rolled k8s.
However, I can repro this on my local mac dev machine (using docker-desktop version 4.26.1) with a different kernel.
Side note: Newer versions of docker-desktop broke Tracee; some features were removed in the linux vm that is shipped with docker. I plan to open a separate issue with them for this (just mentioning it here to explain why I am on an older docker release)
Linux nanna-tracee-8fblb 6.5.11-linuxkit #1 SMP PREEMPT Wed Dec 6 17:08:31 UTC 2023 aarch64 GNU/Linux
hangrymuppet
commented
9 months ago @NDStrahilevitz please ignore the above posts. I figured out the issue and it had to do with the controller not being able to read the policy file and I guess falling back to all event reporting.
{"L":"DEBUG","T":"2024-02-22T20:29:24.480Z","M":"kubernetes cluster","error":"policies.tracee.aquasec.com is forbidden: User \"system:serviceaccount:nanna-tracee:nanna-tracee\" cannot list resource \"policies\" in API group \"tracee.aquasec.com\" at the cluster scope","origin":"cobra:pkg/cmd/cobra/cobra.go:218","calls":"GetTraceeRunner() < glob..func15() < (*Command).execute() < (*Command).ExecuteC() < (*Command).Execute() < Execute() < main()"}
{"L":"DEBUG","T":"2024-02-22T20:29:24.480Z","M":"using policies from --scope and --events flag","origin":"cobra:pkg/cmd/cobra/cobra.go:227","calls":"GetTraceeRunner() < glob..func15() < (*Command).execute() < (*Command).ExecuteC() < (*Command).Execute() < Execute() < main()"}Thank you for looking into this and apologies for the time sink. I will close the issue.
Description
Tracee is emitting events that are not included in container scope and also events that are not in the rules list in its output. There are thousands of events per second and this is overwhelming logging infrastructure.
There are two examples below; both have empty
matchedPoliciesfields.Event Output
net_packet_http_requestevent not in the policy.json
```json { "timestamp": 1708472596270507125, "threadStartTime": 1707279037876750021, "processorId": 1, "processId": 13, "cgroupId": 10372, "threadId": 15, "parentProcessId": 1, "hostProcessId": 302149, "hostThreadId": 302162, "hostParentProcessId": 302089, "userId": 0, "mountNamespace": 4026533464, "pidNamespace": 4026533465, "processName": "flb-pipeline", "executable": { "path": "" }, "hostName": "fluent-bit-t9hm", "containerId": "XXXX", "container": { "id": "XXXX", "name": "fluent-bit", "image": "XXXX/fluent-bit:v2.2.0", "imageDigest": "XXXX/fluent-bit@sha256:XXXX" }, "kubernetes": { "podName": "fluent-bit-t9hm6", "podNamespace": "logging", "podUID": "b26961b3-17c8-4add-9aa0-79bc67dad116" }, "eventId": "2010", "eventName": "net_packet_http_request", "matchedPolicies": [ "" ], "argsNum": 2, "returnValue": 0, "syscall": "sendto", "stackAddresses": [ 0 ], "contextFlags": { "containerStarted": true, "isCompat": false }, "threadEntityId": 3807349527, "processEntityId": 3197503450, "parentEntityId": 4130927439, "args": [ { "name": "metadata", "type": "trace.PktMeta", "value": { "src_ip": "XXXXX", "dst_ip": "XXXXX", "src_port": 56988, "dst_port": 3100, "protocol": 6, "packet_len": 225, "iface": "any" } }, { "name": "http_request", "type": "trace.ProtoHTTPRequest", "value": { "method": "POST", "protocol": "HTTP/1.1", "host": "XXXX", "uri_path": "/loki/api/v1/push", "headers": { "Connection": [ "keep-alive" ], "Content-Length": [ "1499" ], "Content-Type": [ "application/json" ], "User-Agent": [ "Fluent-Bit" ] }, "content_length": 1499 } } ] } ```security_socket_connectevent not in the policy and is inhostscope whereas the policy specifiescontainer scope.json
```json { "timestamp": 1708472596370235936, "threadStartTime": 1706192218472036490, "processorId": 0, "processId": 1820340, "cgroupId": 1, "threadId": 1820407, "parentProcessId": 1, "hostProcessId": 1820340, "hostThreadId": 1820407, "hostParentProcessId": 1, "userId": 0, "mountNamespace": 4026531840, "pidNamespace": 4026531836, "processName": "kubelet", "executable": { "path": "" }, "hostName": "k8s", "containerId": "", "container": {}, "kubernetes": {}, "eventId": "735", "eventName": "security_socket_connect", "matchedPolicies": [ "" ], "argsNum": 3, "returnValue": 0, "syscall": "connect", "stackAddresses": null, "contextFlags": { "containerStarted": false, "isCompat": false }, "threadEntityId": 1725199941, "processEntityId": 3203671770, "parentEntityId": 1844877023, "args": [ { "name": "sockfd", "type": "int", "value": 639893506 }, { "name": "type", "type": "string", "value": "SOCK_STREAM" }, { "name": "remote_addr", "type": "struct sockaddr*", "value": { "sa_family": "AF_INET", "sin_addr": "169.254.25.10", "sin_port": "9254" } } ] } ```Config file
yaml
```yaml apiVersion: v1 data: config.yaml: |- blobPerfBufferSize: "" cache: size: 512 type: mem cri: - runtime: name: containerd socket: /var/run/containerd/containerd.sock - runtime: name: docker socket: /var/run/docker.sock - runtime: name: cri-o socket: /var/run/crio/crio.sock - runtime: name: podman socket: /var/run/podman/podman.sock healthz: true installPath: "" listenAddr: :3366 log: level: info metrics: true output: json: files: - stdout options: execEnv: true execHash: true parseArguments: true relativeTime: true sortEvents: false stackAddresses: false perfBufferSize: 1024 pprof: false pyroscope: false signaturesDir: "" kind: ConfigMap metadata: creationTimestamp: "2024-02-16T00:08:34Z" labels: app.kubernetes.io/instance: tracee app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: tracee app.kubernetes.io/version: 0.20.0 helm.sh/chart: tracee-0.20.0 name: tracee-config namespace: nanna-tracee resourceVersion: "277500421" uid: be8ee520-3e80-4f53-b8bb-dc3a07660974 ```Policy file
yaml
```yaml apiVersion: tracee.aquasec.com/v1beta1 kind: Policy metadata: annotations: description: traces default events meta.helm.sh/release-name: tracee meta.helm.sh/release-namespace: nanna-tracee creationTimestamp: "2024-02-16T00:08:34Z" generation: 1 labels: app.kubernetes.io/managed-by: Helm name: default-policy resourceVersion: "272161700" uid: e5284373-d8a8-49cc-b008-4bff9843652b spec: rules: - event: stdio_over_socket - event: k8s_api_connection - event: aslr_inspection - event: proc_mem_code_injection - event: docker_abuse - event: scheduled_task_mod - event: ld_preload - event: cgroup_notify_on_release - event: default_loader_mod - event: sudoers_modification - event: sched_debug_recon - event: system_request_key_mod - event: cgroup_release_agent - event: rcd_modification - event: core_pattern_modification - event: proc_kcore_read - event: proc_mem_access - event: hidden_file_created - event: anti_debugging - event: ptrace_code_injection - event: process_vm_write_inject - event: disk_mount - event: dynamic_code_loading - event: fileless_execution - event: illegitimate_shell - event: kernel_module_loading - event: k8s_cert_theft - event: proc_fops_hooking - event: syscall_hooking - event: dropped_executable - event: container_create - event: container_remove scope: - container ```Output of
tracee version:The below command was run from inside the container in k8s
Output of
uname -a:Additional details
It looks like all events (network, syscalls, extra) are being emitted by tracee.