Tracee enriches container event but says runtime is unknown

[rafaeldtinoco]

Description

When running tracee in a microk8s environment (that uses containerd), I use the following config:

apiVersion: v1
data:
  config.yaml: |-
    no-containers: false
    cache:
        type: mem
        size: 512
    perf-buffer-size: 1024
    healthz: true
    metrics: true
    pprof: false
    pyroscope: false
    listen-addr: :3366
    log:
        level: info
    output:
        json:
          files:
            - stdout
        options:
            parse-arguments: true
            stack-addresses: false
            exec-env: false
            relative-time: true
            exec-hash: false
            sort-events: true
kind: ConfigMap

and the following daemonset deployment settings:

      volumes:
      - hostPath:
          path: /tmp/tracee
          type: ""
        name: tmp-tracee
      - hostPath:
          path: /etc/os-release
          type: ""
        name: etc-os-release
      - hostPath:
          path: /var/snap/microk8s/common/run/containerd.sock
          type: ""
        name: containerd-sock
      - configMap:
          defaultMode: 420
          name: tracee-config
        name: tracee-config
...
        volumeMounts:
        - mountPath: /tmp/tracee
          name: tmp-tracee
        - mountPath: /etc/os-release-host
          name: etc-os-release
          readOnly: true
        - mountPath: /var/run/containerd/containerd.sock
          name: containerd-sock
          readOnly: true
        - mountPath: /tracee/config.yaml
          name: tracee-config
          readOnly: true
          subPath: config.yaml

And I'm able to enrich the container events correctly because I'm telling that the "snap" containerd socket should be placed at the "default containerd.sock" location, so tracee knows how to find it.

Problem is that, despite being able to talk to the socket and enriching the event correctly:

{"timestamp":132738596945,"threadStartTime":132738558439,"processorId":2,"processId":58538,"cgroupId":6367,"threadId":58538,"parentProcessId":58536,"hostProcessId":58538,"hostThreadId":58538,"hostParentProcessId":58536,"userId":1000,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"strace","executable":{"path":""},"hostName":"vm01","containerId":"","container":{},"kubernetes":{},"eventId":"6018","eventName":"anti_debugging","matchedPolicies":["default-policy"],"argsNum":1,"returnValue":0,"syscall":"","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":0,"processEntityId":0,"parentEntityId":0,"args":[{"name":"triggeredBy","type":"unknown","value":{"args":[{"name":"request","type":"string","value":"PTRACE_TRACEME"},{"name":"pid","type":"pid_t","value":0},{"name":"addr","type":"void*","value":"0x0"},{"name":"data","type":"void*","value":"0x0"}],"id":101,"name":"ptrace","returnValue":0}}],"metadata":{"Version":"1","Description":"A process used anti-debugging techniques to block a debugger. Malware use anti-debugging to stay invisible and inhibit analysis of their behavior.","Tags":null,"Properties":{"Category":"defense-evasion","Kubernetes_Technique":"","Severity":1,"Technique":"Debugger Evasion","external_id":"T1622","id":"attack-pattern--e4dc8c01-417f-458d-9ee0-bb0617c1b391","signatureID":"TRC-102","signatureName":"Anti-Debugging detected"}}}
{"timestamp":140209766224,"threadStartTime":140189795344,"processorId":3,"processId":58733,"cgroupId":2898,"threadId":58739,"parentProcessId":58723,"hostProcessId":58733,"hostThreadId":58739,"hostParentProcessId":58723,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"runc","executable":{"path":""},"hostName":"vm01","containerId":"","container":{},"kubernetes":{},"eventId":"2014","eventName":"container_create","matchedPolicies":["default-policy"],"argsNum":10,"returnValue":0,"syscall":"mkdirat","stackAddresses":[0],"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":832392698,"processEntityId":1169416937,"parentEntityId":4094784935,"args":[{"name":"runtime","type":"const char*","value":"unknown"},{"name":"container_id","type":"const char*","value":"42af7ebd212b469a7161fdd371f7a61cf1148ec36ea65844b74fb48a864edf7f"},{"name":"ctime","type":"unsigned long","value":1698979865159663000},{"name":"container_image","type":"const char*","value":"registry.k8s.io/pause:3.7"},{"name":"container_image_digest","type":"const char*","value":"registry.k8s.io/pause:3.7"},{"name":"container_name","type":"const char*","value":""},{"name":"pod_name","type":"const char*","value":"fedora"},{"name":"pod_namespace","type":"const char*","value":"default"},{"name":"pod_uid","type":"const char*","value":"c7f0e802-84e5-411b-82e9-6bd6bb659e66"},{"name":"pod_sandbox","type":"bool","value":true}]}
{"timestamp":140287072236,"threadStartTime":140271716888,"processorId":0,"processId":58768,"cgroupId":2898,"threadId":58776,"parentProcessId":58723,"hostProcessId":58768,"hostThreadId":58776,"hostParentProcessId":58723,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"runc","executable":{"path":""},"hostName":"vm01","containerId":"","container":{},"kubernetes":{},"eventId":"2014","eventName":"container_create","matchedPolicies":["default-policy"],"argsNum":10,"returnValue":0,"syscall":"mkdirat","stackAddresses":[0],"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":2252951342,"processEntityId":440895368,"parentEntityId":4094784935,"args":[{"name":"runtime","type":"const char*","value":"unknown"},{"name":"container_id","type":"const char*","value":"974d4a27f442260c82d64ead15e29d46375618829494a62d6093372020ef3e4b"},{"name":"ctime","type":"unsigned long","value":1698979865239662300},{"name":"container_image","type":"const char*","value":"docker.io/library/fedora:38"},{"name":"container_image_digest","type":"const char*","value":"docker.io/library/fedora:38"},{"name":"container_name","type":"const char*","value":"fedora"},{"name":"pod_name","type":"const char*","value":"fedora"},{"name":"pod_namespace","type":"const char*","value":"default"},{"name":"pod_uid","type":"const char*","value":"c7f0e802-84e5-411b-82e9-6bd6bb659e66"},{"name":"pod_sandbox","type":"bool","value":false}]}
{"timestamp":146252331098,"threadStartTime":146243632662,"processorId":1,"processId":58912,"cgroupId":2898,"threadId":58912,"parentProcessId":58723,"hostProcessId":58912,"hostThreadId":58912,"hostParentProcessId":58723,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"runc","executable":{"path":""},"hostName":"vm01","containerId":"","container":{},"kubernetes":{},"eventId":"2015","eventName":"container_remove","matchedPolicies":["default-policy"],"argsNum":2,"returnValue":0,"syscall":"unlinkat","stackAddresses":[0],"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":953470485,"processEntityId":953470485,"parentEntityId":4094784935,"args":[{"name":"runtime","type":"const char*","value":"unknown"},{"name":"container_id","type":"const char*","value":"974d4a27f442260c82d64ead15e29d46375618829494a62d6093372020ef3e4b"}]}
{"timestamp":146320219125,"threadStartTime":146316627881,"processorId":1,"processId":58946,"cgroupId":2898,"threadId":58946,"parentProcessId":58723,"hostProcessId":58946,"hostThreadId":58946,"hostParentProcessId":58723,"userId":0,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"runc","executable":{"path":""},"hostName":"vm01","containerId":"","container":{},"kubernetes":{},"eventId":"2015","eventName":"container_remove","matchedPolicies":["default-policy"],"argsNum":2,"returnValue":0,"syscall":"unlinkat","stackAddresses":[0],"contextFlags":{"containerStarted":false,"isCompat":false},"threadEntityId":3764885266,"processEntityId":3764885266,"parentEntityId":4094784935,"args":[{"name":"runtime","type":"const char*","value":"unknown"},{"name":"container_id","type":"const char*","value":"42af7ebd212b469a7161fdd371f7a61cf1148ec36ea65844b74fb48a864edf7f"}]}

The runtime appears as unknown. Same happens for container_create and container_remove.

I believe we're picking the container runtime name from the cgroup filesystem hierarchy. We could also pick from the default socket place, like this case, we do know its coming from "containerd".

[geyslan]

And I'm able to enrich the container events correctly because I'm telling that the "snap" containerd socket should be placed at the "default containerd.sock" location, so tracee knows how to find it.

So shouldn't the container and kubernetes keys be populated?

[rafaeldtinoco]

"value":"docker.io/library/fedora:38"
"value":"registry.k8s.io/pause:3.7"

THe container image and container name worked. Those came from the runtime. I believe the ones you're seeing empty come from k8s and would have come only if the expected labels were done (and only with docker):

SO containerd enrichment does not support (yet) picking Pod metadata (or it does not even contain the metadata). That is the part where the k8s API would take place (the k8s info would come from the k8s API and not from labels).

What did not work, in this case, was recognizing the runtime from the cgroup hierarchy pattern (there are some regex that checks uuids within cgroup directories, and things alike).

[geyslan]

Could you provide the cgroup_path from cgroup_mkdir so I can update this table for microk8s?

---

EDIT: I get it from https://github.com/aquasecurity/tracee/issues/3003#issuecomment-1788232984 👍🏼

[NDStrahilevitz]

@rafaeldtinoco As you correctly pointed out, the runtime is picked from the path pattern. But that enrichment worked because of the fallback mechanism for unknown runtimes, where the "main" enricher goes through all sub-enrichers. So it also went through the containerd enricher which was working because of the mount you set up for tracee's container (see pkg/containers/service.go):

The pattern is probably the same one described in #3003 (pod<uid>/<container_id>).

[NDStrahilevitz]

@geyslan @rafaeldtinoco About the kubernetes field being empty, this is because the kubernetes data of the container created are in the arguments for container events, not in the event fields. If we had for example a write event inside the newly created pod, the enrichment data would go there. But since these are lifecycle events the data is in the argument. This is useful for example if we have a container created in a container, you'd have the creating container's data in container and kubernetes fields of the event, and the actual container created described in the arguments.

The 2nd event from the issue: ...{"name":"pod_name","type":"const char*","value":"fedora"},{"name":"pod_namespace","type":"const char*","value":"default"},{"name":"pod_uid","type":"const char*","value":"c7f0e802-84e5-411b-82e9-6bd6bb659e66"},{"name":"pod_sandbox","type":"bool","value":true}]} (The sandbox container created) The 3rd: ...{"name":"pod_name","type":"const char*","value":"fedora"},{"name":"pod_namespace","type":"const char*","value":"default"},{"name":"pod_uid","type":"const char*","value":"c7f0e802-84e5-411b-82e9-6bd6bb659e66"},{"name":"pod_sandbox","type":"bool","value":false}]} (The actual workload)

So kubernetes enrichment did work here, the only problem is the runtime.

[josedonizetti]

Considering @NDStrahilevitz comment above, this seems to be a duplicated of https://github.com/aquasecurity/tracee/issues/3003? So I'm closing it in favor of the other issue that @NDStrahilevitz is already investigating. Feel free to reopen if I misunderstood.