search

kubeshark / tracer

The kernel tracer that attaches eBPF probes to containers for capturing TLS traffic
GNU General Public License v3.0
12 stars 6 forks source link

Tracer incompatible with `kind` #38

Closed alongir closed 2 months ago

alongir commented 4 months ago

Regarding this https://github.com/kubeshark/kubeshark/issues/1493|https://github.com/kubeshark/kubeshark/issues/1493

  1. pf_ring is not supported yet for new kernels used in kind (btw all the issue is about kind, not minikube)
  2. when I try to run kubeshark on mac m1 kind (which is actually running in linux qemu vm), tracer fails with 
    
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 303, [0::/../../../../../system.slice/containerd.service ]" pid=303
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 310, [0::/../../kubelet-kubepods-burstable-pod8cbf9887da25b380bf1858a4d3b399d8.slice/cri-containerd-195a91a1b8102bc1b937235cabf043c8fb6b2b2bd9b77d8dd51c8cc89831a8ea.scope ]" pid=310
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 311, [0::/../../../../../system.slice/containerd.service ]" pid=311
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 326, [0::/../../kubelet-kubepods-burstable-pod584bd4e38ab0df294af27207ce1a29b1.slice/cri-containerd-123b744f4d48f7464a6cb227b16c11849d0f01e4c744adeeed751abc0389df10.scope ]" pid=326
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 367, [0::/../../kubelet-kubepods-burstable-pode44a910c33eb66c4d3b4b617ed29c23e.slice/cri-containerd-a87815ecec535258596da80d6a49016fd00f498f43df2659b4870591a995530e.scope ]" pid=367
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 379, [0::/../../kubelet-kubepods-burstable-pode98a496a5002e4d3842f58e2bb4420dd.slice/cri-containerd-d45d800cc899cf374a9522a225a7190386660c24810b1d9f97575757faa6dfcc.scope ]" pid=379
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 497, [0::/../../kubelet-kubepods-burstable-pod8cbf9887da25b380bf1858a4d3b399d8.slice/cri-containerd-ada2550592573f8a9296d1fb5605b3c35232713bc00319695a6a0d8aa2c08c29.scope ]" pid=497
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 507, [0::/../../kubelet-kubepods-burstable-pode98a496a5002e4d3842f58e2bb4420dd.slice/cri-containerd-9f2ef536b7a4a924eea35993e54aa7f6566c67d788269dae40f1c6485992383d.scope ]" pid=507
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 515, [0::/../../kubelet-kubepods-burstable-pode44a910c33eb66c4d3b4b617ed29c23e.slice/cri-containerd-0b36aec7be58dc122954c9ce73adbe86f40e4ae26b2821fee79c4f0b260782c4.scope ]" pid=515
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 603, [0::/../../kubelet-kubepods-burstable-pod584bd4e38ab0df294af27207ce1a29b1.slice/cri-containerd-07448cee64114b817548f7ba79d70be3010a2c954fde456ddbfa3bed8d5bf705.scope ]" pid=603
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 655, [0::/../../../../kubelet.service ]" pid=655
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 723, [0::/../../../../../system.slice/containerd.service ]" pid=723
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 751, [0::/../../../../../system.slice/containerd.service ]" pid=751
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 767, [0::/../../../kubelet-kubepods-pod0b029d7c_9870_48ef_a6d5_1e6959bfb310.slice/cri-containerd-2990b76b9e90ee6c9e7fc01d641fdcd860d8fce18ead116d0c00e41494d73b17.scope ]" pid=767
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 777, [0::/../../../kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-podbb419ebd_484c_4547_9e99_e87fb73d299c.slice/cri-containerd-10120d72758aecb70439b9cefc1d56576dbd69992cc2f526882b27292ffe1719.scope ]" pid=777
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 822, [0::/../../../kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-podbb419ebd_484c_4547_9e99_e87fb73d299c.slice/cri-containerd-d8c07db59ccd2f3a6fa45c9b3cc6ad16f4f8e817c0a9a026e739a2ec04f227b6.scope ]" pid=822
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 86, [0::/../../../../../system.slice/systemd-journald.service ]" pid=86
2024-02-15T14:35:34Z WRN tracer/tls_process_discoverer.go:69 > Couldn't get the cgroup of process. error="Cgroup path not found for 992, [0::/../../../kubelet-kubepods-pod0b029d7c_9870_48ef_a6d5_1e6959bfb310.slice/cri-containerd-66bcbf0f81d2a239216c661540fd24403a9b987ee7071da1fca2f30c9b85e17c.scope ]" pid=992
2024-02-15T14:35:34Z INF tracer/tls_process_discoverer.go:30 > pids=[]
2024-02-15T14:35:34Z FTL tracer/main.go:68 > Couldn't initialize the tracer: error="field GoCryptoTlsAbi0Read: program go_crypto_tls_abi0_read: apply CO-RE relocations: can't read types: type id 4132: unknown kind: Unknown (19)"```

Slack Message

iluxa commented 4 months ago

@alongir Can you please provide version information?

the issue might be already fixed in https://github.com/kubeshark/tracer/pull/32

corest commented 4 months ago

@iluxa I've tried latest and it is indeed fixed. But now I run into this error

2024-02-18T17:44:25Z FTL: tracer/tracer.go:45 > Failed to set rlimit error="SYS_RESOURCE is required to change rlimits for eBPF: unexpected error detecting memory cgroup accounting: function not implemented"

I've changed par this part to only log error to see what happens next and in that case it fails with:

2024-02-18T17:44:25Z ERR tracer/tracer.go:45 > Failed to set rlimit error="SYS_RESOURCE is required to change rlimits for eBPF: unexpected error detecting memory cgroup accounting: function not implemented"
2024-02-18T17:44:25Z INF tracer/tracer.go:54 > Detected Linux kernel version: 6.7.4-200.fc39.aarch64
2024-02-18T17:44:25Z FTL tracer/main.go:67 > Couldn't initialize the tracer: error="field GoCryptoTlsAbi0Read: program go_crypto_tls_abi0_read: map pids_map: map create: function not implemented (without BTF k/v)"
corest commented 4 months ago

Ok, the error from above was related to running podman machine in rootless mode. With rootful mode I have this

2024-02-18T19:05:07Z INF tracer/tls_process_discoverer.go:67 > Starting TLS auto discoverer: pids=103 procfs=/hostproc
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=init pid=1
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=101
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=1135
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=1149
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=1173
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=2d115f272f04bfb9185c907122fb5f9d76187357aba881c7bdd81fd73b3b2b77 pid=1205
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=736b998475042b9a8f3a0cfa8b22d276fee6cceeca76c59a1bc016d73c61c5eb pid=1213
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=01bfefe9463006731f7a7db385c1ed14ed31eba24a47f7688593d726b729bb71 pid=1221
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=1684
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=1697
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=13b1bbe73811e917e2545f26d7d33306c155ac95ac444de6e52ed155d620dec2 pid=1726
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=c6c4ba007fb5ae15929fa952bf7c758c5c56888c9f3c38be688c08b0af2edcd3 pid=1734
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=07b57f807e44e4ea72a80518640379c9978a61f5a3d9fb12b718843ea225513f pid=1913
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=07b57f807e44e4ea72a80518640379c9978a61f5a3d9fb12b718843ea225513f pid=1940
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=07b57f807e44e4ea72a80518640379c9978a61f5a3d9fb12b718843ea225513f pid=1941
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=07b57f807e44e4ea72a80518640379c9978a61f5a3d9fb12b718843ea225513f pid=1942
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=07b57f807e44e4ea72a80518640379c9978a61f5a3d9fb12b718843ea225513f pid=1943
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=ac2fdef039c0d772e6762c315cc4cacbe2e5470c3ac53bade2006f8fcc980ca3 pid=1985
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=255
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=256
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=257
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=3125
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=d5168e8e55b3e18b9313aeddcdc6d9aaa695376b7b0d4b74a4ec726d71cf30da pid=3145
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=63d2e6b201473f0a387a31446e13139d1ac873785385555897ecccd82c0f08d3 pid=315
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=7d1331b5b77d3973fc5aca4a64a878ee0e13cf3ce42009ff96f9e1f5c889ee93 pid=322
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=59f84dd314a42f6ce9542ce755895615f061af52fab1125cdee47343552dcaee pid=3239
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=f601e3d2124ad3e31283544d6ef2d849879e18860ca349b915d1082861db1f51 pid=329
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:80 > Couldn't get the cgroup of process. error="Cgroup path not found for 3385, [0::/ ]" pid=3385
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=365
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=05df8788270abe1e985d736eb230b838b1a231b30e7351c79dcae335e97aeea9 pid=385
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=kubelet pid=652
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=719
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=containerd pid=746
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=1a7ee71fdf4b661c0645997f88d78c250ceca955d71b767ab664c58fd2142624 pid=768
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=f5fd44a83220ac549b91ccb966d9ecab9fd9eabe49141add578f751d78e216af pid=775
2024-02-18T19:05:07Z WRN tracer/tls_process_discoverer.go:86 > Couldn't find the pod for the given cgroup of pid. cgroup=journald pid=86
2024-02-18T19:05:07Z INF tracer/tls_process_discoverer.go:41 > pids=[{},{},{},{},{},{},{},{},{}]
2024-02-18T19:05:07Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/1427/root/coredns with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:07Z INF tracer/tracer.go:237 > PID skipped not a Go binary or symbol table is stripped (pid: 1427) /hostproc/1427/root/coredns
2024-02-18T19:05:07Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/1434/root/coredns with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:07Z INF tracer/tracer.go:237 > PID skipped not a Go binary or symbol table is stripped (pid: 1434) /hostproc/1434/root/coredns
2024-02-18T19:05:07Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/596/root/usr/local/bin/etcd with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:07Z INF tracer/packet_unix_socket.go:135 > Accepted unix socket: Address=@
2024-02-18T19:05:07Z INF tracer/tracer.go:241 > Targeting TLS (pid: 596) (Go: /hostproc/596/root/usr/local/bin/etcd)
2024-02-18T19:05:07Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/821/root/usr/local/bin/kube-proxy with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:07Z INF tracer/tracer.go:237 > PID skipped not a Go binary or symbol table is stripped (pid: 821) /hostproc/821/root/usr/local/bin/kube-proxy
2024-02-18T19:05:07Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/990/root/bin/kindnetd with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:08Z INF tracer/tracer.go:241 > Targeting TLS (pid: 990) (Go: /hostproc/990/root/bin/kindnetd)
2024-02-18T19:05:08Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/1468/root/usr/local/bin/local-path-provisioner with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:08Z INF tracer/tracer.go:237 > PID skipped not a Go binary or symbol table is stripped (pid: 1468) /hostproc/1468/root/usr/local/bin/local-path-provisioner
2024-02-18T19:05:08Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/480/root/usr/local/bin/kube-controller-manager with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:08Z INF tracer/tracer.go:237 > PID skipped not a Go binary or symbol table is stripped (pid: 480) /hostproc/480/root/usr/local/bin/kube-controller-manager
2024-02-18T19:05:08Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/506/root/usr/local/bin/kube-scheduler with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:08Z INF tracer/tracer.go:237 > PID skipped not a Go binary or symbol table is stripped (pid: 506) /hostproc/506/root/usr/local/bin/kube-scheduler
2024-02-18T19:05:08Z INF tracer/go_offsets.go:230 > Disassembling /hostproc/515/root/usr/local/bin/kube-apiserver with Capstone 5.0 (arch: 1, mode: 0)
2024-02-18T19:05:08Z INF tracer/tracer.go:237 > PID skipped not a Go binary or symbol table is stripped (pid: 515) /hostproc/515/root/usr/local/bin/kube-apiserver

Can this be considered as working?

corest commented 4 months ago

So, latest canary worker doesn't work. But worker with latest tracer seems to work ok. Even service-map image

iluxa commented 4 months ago

...

Can this be considered as working?

Yes, it works based on logs

corest commented 4 months ago

@iluxa when do you plan then to release canary worker with latest tracer changes?

iluxa commented 4 months ago

I am not a right person who makes worker releases. I suppose release can be make with latest tracer changes.

alongir commented 2 months ago

Fixed with: https://github.com/kubeshark/kubeshark/releases/tag/v52.3.0