k8spacket / k8spacket

k8spacket - collects TCP traffic and TLS connection metadata in the Kubernetes cluster using eBPF and visualizes in Grafana
Apache License 2.0
1.02k stars 52 forks source link

Pod-k8spacket always restart until Crashloopback #25

Closed durant0819 closed 1 year ago

durant0819 commented 1 year ago

Dear, I met a problem,my k8spacket pod never got the two *.so ,i didn't know whether it is related to the previous issue about $(path), which lead to this problem :

nodegraph-x86_64.so

tls-parser-x86_64.so

So I tried to download the two .so , when the pod is ok, I use kubectl cp ,copy the two .so into the pod's /home/k8spacket/plugins,the grafana is normal

after a while , the pod k8spacket begun to restart, and the /home/k8spacket/plugins/*.so are missing ,of course.

I found the reason,from the log: `panic: interface conversion: gopacket.TransportLayer is nil, not *layers.TCP

goroutine 13 [running]: github.com/k8spacket/k8spacket/tcp.listenInterface({0xc000044106, 0xf}, {0x20d08c4, 0x3}) /home/k8spacket/tcp/tcp.go:52 +0x6e8 created by github.com/k8spacket/k8spacket/tcp.interfacesRefresher /home/k8spacket/tcp/tcp.go:88 +0x1cals`

So what cause this problem and how to slove it , thx

k8spacket commented 1 year ago

Hi @durant0819 Could you share logs of the init-k8spacket container:

k -n k8spacket logs <k8spacket-pod-name> -f -c init-k8spacket 

and whole logs from the k8spacket container as well:

k -n k8spacket logs <k8spacket-pod-name> -f -c k8spacket
durant0819 commented 1 year ago

Dear,

It's my Pod status

[root@k8s-aio k8spacket-helm]# kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE k8spacket k8spacket-9btzh 1/1 Running 13 (4m39s ago) 116m

And Actually I found that: [root@k8s-aio k8spacket-helm]# kubectl -n k8spacket logs k8spacket-9btzh -f -c init-k8spacket Connecting to github.com (20.205.243.166:443) wget: note: TLS certificate validation not implemented wget: short read, have only 0: Connection reset by peer wget: error getting response: Connection reset by peer

The whole log here, [root@k8s-aio k8spacket-helm]# kubectl -n k8spacket logs k8spacket-9btzh -f -c k8spacket 2023/01/18 11:34:37 Serving requests on port 6676 2023/01/18 11:34:37 Refreshing interfaces for capturing... Getting k8s resources 2023/01/18 11:34:37 Starting capture on interface "cali7added6fba2" 2023/01/18 11:34:37 Starting capture on interface "cali1421c7edebd" 2023/01/18 11:34:37 Starting capture on interface "tunl0" 2023/01/18 11:34:37 Starting capture on interface "cali122f70eab85" 2023/01/18 11:34:37 Starting capture on interface "caliee991d03a8a" 2023/01/18 11:34:37 Starting capture on interface "cali3f0b4533ac1" 2023/01/18 11:34:37 reading in packets 2023/01/18 11:34:37 reading in packets 2023/01/18 11:34:37 reading in packets 2023/01/18 11:34:37 reading in packets 2023/01/18 11:34:37 reading in packets 2023/01/18 11:34:37 reading in packets Found 12 pods Found 5 services 2023/01/18 11:34:47 Refreshing interfaces for capturing... 2023/01/18 11:34:57 Refreshing interfaces for capturing... 2023/01/18 11:35:07 Refreshing interfaces for capturing...

and it is a loop (Refreshing interfaces for capturing...) till now~

Thank you very much for you support~~~

k8spacket commented 1 year ago

@durant0819

Logs from the k8spacket container look great. Refreshing interfaces for capturing... means that every 10 seconds k8spacket check if there are new network interfaces to listen to. (See: https://github.com/k8spacket/k8spacket-helm-chart/blob/master/charts/k8spacket/values.yaml#L73)

The problem is in the init-k8spacket container. It's hard to say why you got wget: error getting response: Connection reset by peer Is the GitHub page possibly blocked somehow from your Kubernetes cluster area? If yes, You can download plugins, expose them somewhere visible for the K8S cluster, and change links in values for the helm, similar to this: https://github.com/k8spacket/k8spacket/issues/15#issuecomment-1381790083

durant0819 commented 1 year ago

Dear, Maybe I could only say that because some indescribable reason in "天(和谐)朝", github can be used all depends.... whatever... Thanks a lot, I will tried the lower edition....

k8spacket commented 1 year ago

Hi @durant0819

I found that issue. Indeed it wasn't a problem with your environment. The problem was in the busybox image. It is fixed, and you can try it once again with updated helm repo:

helm repo update k8spacket
helm install k8spacket --namespace k8spacket k8spacket/k8spacket --create-namespace