Kubearmor Alerts/Telemetry sends different container name on each logs

[seswarrajan]

Logs/Alerts throw different container names

General Information

• Environment : Azure aks cluster
• Kernel version (run uname -a)
• Orchestration system version in use (e.g. kubectl version, ...)
• Link to relevant artifacts (policies, deployments scripts, ...)
• Target containers/pods USe wordpress-mysql deployment for testing

To Reproduce

1. Install kubearmor latest using karmor
2. Once kubearmor is deployed, run karmor logs --logFilter all
3. Then deploy wordpress-mysql microservice
4. Observe container name for all logs

You will notice all logs have different container name for the first few mins.

Expected behavior The container name should be same on any case even during restart of any deployments/pods.

A description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

[daemon1024]

{"Timestamp":1691567368,"UpdatedTime":"2023-08-09T07:49:28.977327Z","ClusterName":"default","HostName":"minikube","NamespaceName":"wordpress-mysql","PodName":"wordpress-69b6c47cf8-2w5wg","ContainerID":"cdf7ab3769d8578e8913980d527f79f4a2ac0733d7c7e3317bcb51f7c6597b9c","ContainerName":"k8s_wordpress_wordpress-69b6c47cf8-2w5wg_wordpress-mysql_7c49573e-8a55-42a7-a721-4df6373d430a_0","ParentProcessName":"/bin/bash","ProcessName":"/bin/tar","HostPPID":812344,"HostPID":812359,"PPID":1,"PID":9,"Type":"ContainerLog","Source":"/bin/tar xf -","Operation":"File","Resource":"/var/www/html/wp-includes/fonts/dashicons.eot","Data":"syscall=SYS_OPENAT fd=-100 flags=O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC","Result":"Passed"}
{"Timestamp":1691567368,"UpdatedTime":"2023-08-09T07:49:28.977195Z","ClusterName":"default","HostName":"minikube","NamespaceName":"wordpress-mysql","PodName":"wordpress-69b6c47cf8-2w5wg","ContainerID":"cdf7ab3769d8578e8913980d527f79f4a2ac0733d7c7e3317bcb51f7c6597b9c","ContainerName":"k8s_wordpress_wordpress-69b6c47cf8-2w5wg_wordpress-mysql_7c49573e-8a55-42a7-a721-4df6373d430a_0","ParentProcessName":"/bin/bash","ProcessName":"/bin/tar","HostPPID":812344,"HostPID":812359,"PPID":1,"PID":9,"Type":"ContainerLog","Source":"/bin/tar xf -","Operation":"File","Resource":"./wp-includes/fonts","Data":"syscall=SYS_FCHOWNAT userid=33 group=33 mode=256","Result":"Passed"}
{"Timestamp":1691567369,"UpdatedTime":"2023-08-09T07:49:29.122982Z","ClusterName":"default","HostName":"minikube","NamespaceName":"wordpress-mysql","PodName":"wordpress-65f4f68f69-46gzj","Labels":"app=wordpress","ContainerID":"2279bc0883e8a8a28ab7694bc8d8e7f4b252063ff8ebaba55eb7331d263e762f","ContainerName":"wordpress","ContainerImage":"wordpress:4.8-apache@sha256:6216f64ab88fc51d311e38c7f69ca3f9aaba621492b4f1fa93ddf63093768845","ParentProcessName":"/usr/bin/containerd-shim-runc-v2","ProcessName":"/usr/sbin/apache2","HostPPID":811125,"HostPID":811148,"PPID":811125,"PID":1,"Type":"ContainerLog","Source":"/usr/sbin/apache2","Operation":"Syscall","Resource":"(\u00100\u0012","Data":"syscall=SYS_UNLINK","Result":"Passed"}

Reproduced the issue successfully. Need to check why is the container name/id changing.

[DelusionalOptimist]

@daemon1024 @rajaSahil Some points that might help you folks. Back when I was investigating it, I remember these logs generating for some time when new containers are added. If that's still the case: We have two sources of container metadata: container runtime and the K8s API. The runtime monitor generally detects new containers first and adds it to the global container map. In case of docker, the names that the runtime APIs gives are of the format "/k8s_*" and we trim the preceeding '/'. So we get this in the initial telemetry logs. Code ref - https://github.com/kubearmor/KubeArmor/blob/e58e72215c240786d153af680ac6e080a9c5b6f0/KubeArmor/core/dockerHandler.go#L102 At some point later, the map is updated with metadata from K8s APIs and we get the new name. I'm not entirely sure but the case with containerd might be similar as well.

Possible fix: Containers that are created by Kubernetes have the io.kubernetes.container.name label when queried through runtime APIs. This can be used to get the actual container name. In non-k8s env, we can continue to use the current logic as the container names are the actual container names.

[EraKin575]

does this issue still need to be worked upon?