kubeshark / kubeshark

The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters. Inspired by Wireshark, purposely built for Kubernetes
https://kubeshark.co
Apache License 2.0
10.86k stars 460 forks source link

Sniffer Crash & No Data on the UI #1559

Open sahilmarhatta opened 1 month ago

sahilmarhatta commented 1 month ago

Installed kubeshark latest via helm, the sniffer container restarts sending an OOM killed and eventually crashes. Tried giving it enough resources via yaml, still the same.

Also there is no data across the UI available, its empty

Hell me fix it pls

alongir commented 1 month ago

Hi @sahilmarhatta Do you get OOMs across all nodes? Or just one node? I assume you use v52.3.69? Did you change the resource values using helm values? For example:

tap:
  resources:
    sniffer:
      limits:
        cpu: 2750m
        memory: 2Gi
      requests:
        cpu: 50m
        memory: 50Mi
    tracer:
      limits:
        cpu: 2750m
        memory: 2Gi
      requests:
        cpu: 50m
        memory: 50Mi
  storageLimit: 5000Mi # this is also a very important value

Memory consumption is related to the amount of traffic processing. You can start by controlling the amount of processes traffic using pod targeting rules. Read more here: https://docs.kubeshark.co/en/pod_targeting TL;DR - specify certain pods or namespaces to focus on for the purpose of limiting traffic processing. Once you get Kubeshark running and you're able to see traffic, you can increase the list of targeted pods.For example:

tap: 
  namespaces:
  - default # to target pods in the default namespace

or:

tap:
  regex: my-pod-prefix # put any regex or the first few characters of the pod name to target specific pods

It would be very helpful to paste logs form the Sniffer that crashes.

sahilmarhatta commented 1 month ago

Thanks Alon ! I tried all the changes suggested, however the sniffer is still crashing and there is no data being populated across the dashboard (UI). Attached are sniffer logs for ready reference.

2024-07-14T06:56:44.779455832Z 2024-07-14T06:56:44Z INF source/packet_source_manager.go:236 > Setting BPF filter: 2024-07-14T06:56:45.520041579Z 2024/07/14 06:56:45 ERROR: har: cannot parse Content-Type header "": mime: no media type 2024-07-14T06:56:45.520223821Z 2024/07/14 06:56:45 ERROR: har: cannot parse Content-Type header "": mime: no media type 2024-07-14T06:56:45.544755908Z 2024/07/14 06:56:45 ERROR: har: cannot parse Content-Type header "": mime: no media type 2024-07-14T06:56:45.549262763Z 2024/07/14 06:56:45 ERROR: har: cannot parse Content-Type header "": mime: no media type

On Sat, Jul 13, 2024 at 9:26 AM Alon Girmonsky @.***> wrote:

Hi @sahilmarhatta https://github.com/sahilmarhatta Do you get OOMs across all nodes? Or just one node? I assume you use v52.3.69? Did you change the resource values using helm values? For example:

tap: resources: sniffer: limits: cpu: 2750m memory: 2Gi requests: cpu: 50m memory: 50Mi tracer: limits: cpu: 2750m memory: 2Gi requests: cpu: 50m memory: 50Mi storageLimit: 5000Mi # this is also a very important value

Memory consumption is related to the amount of traffic processes. You can start by controlling the amount of processes traffic using pod targeting rules. Read more here: https://docs.kubeshark.co/en/pod_targeting TL;DR - specify certain pods or namespaces to focus on for the purpose of limiting traffic processing. Once you get Kubeshark running and you're able to see traffic, you can increase the list of targeted pods.For example:

tap: namespaces:

  • default # to target pods in the default namespace

or:

tap: regex: my-pod-prefix # put any regex or the first few characters of the pod name to target specific pods

It would be very helpful to paste logs form the Sniffer that crashes.

— Reply to this email directly, view it on GitHub https://github.com/kubeshark/kubeshark/issues/1559#issuecomment-2226759094, or unsubscribe https://github.com/notifications/unsubscribe-auth/BJ2V43WPMM5T27HBQIZR6ILZMCQP3AVCNFSM6AAAAABKZ47SYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRWG42TSMBZGQ . You are receiving this because you were mentioned.Message ID: @.***>