Beyla not able to provide k8 attributes in metrics with warning : Expecting a Pod. Got: *kube.PodInfo

[AbhijithHarness]

Trying to run Beyla auto instrumentation in GCP Kubernetes Cluster. But k8 attributes are missing in the data.

In Beyla DeamonSet, we are seeing the following warning:

"k8s.io/client-go/informers/factory.go:150: watch of *v1.Pod ended with: couldn't queue object: was expecting a Pod. Got: *kube.PodInfo"
 "k8s.io/client-go/informers/factory.go:150: watch of *v1.ReplicaSet ended with: couldn't queue object: was expecting a ReplicaSet. Got: *kube.ReplicaSetInfo"

serverVersion:
  buildDate: "2024-01-04T22:48:32Z"
  compiler: gc
  gitCommit: 6f460c12ad45abb234c18ec4f0ea335a1203c415
  gitTreeState: clean
  gitVersion: v1.27.8-gke.1067004
  goVersion: go1.20.11 X:boringcrypto
  major: "1"
  minor: "27"
  platform: linux/amd64

[mariomac]

Hi @AbhijithHarness ! You can ignore the messages you mentioned. I recommend you to update to Beyla 1.5.1 and you won't see them anymore.

With respect to Kubernetes attributes, if Beyla 1.5.1 does not fixes your issues, could I see your configuration and the debug logs of Beyla?

[grcevski]

Just to follow up on what Mario mentioned, to get debug logs from Beyla, set BEYLA_LOG_LEVEL=debug, the log information is printed on stdout.

[AbhijithHarness]

Now I am seeing high restart counts in pods.

Logs: downloaded-logs-20240423-023608.csv

Configuration:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: beyla
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: beyla
rules:
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: beyla
subjects:
  - kind: ServiceAccount
    name: beyla
    namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: beyla

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: monitoring
  name: beyla-config
data:
  beyla-config.yml: |
    # this is required to enable kubernetes discovery and metadata
    attributes:
      kubernetes:
        enable: true
    # this will provide automatic routes report while minimizing cardinality
    routes:
      unmatched: heuristic
    # let's instrument only the docs server
    discovery:
      services:
        - k8s_deployment_name: "demo"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: monitoring
  name: beyla
spec:
  selector:
    matchLabels:
      instrumentation: beyla
  template:
    metadata:
      labels:
        instrumentation: beyla
    spec:
      serviceAccountName: beyla
      hostPID: true # mandatory!
      containers:
        - name: beyla
          image: grafana/beyla:1.5.1
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true # mandatory!
            readOnlyRootFilesystem: true
          volumeMounts:
            - mountPath: /config
              name: beyla-config
            - mountPath: /var/run/beyla
              name: var-run-beyla
          env:
            - name: BEYLA_CONFIG_PATH
              value: "/config/beyla-config.yml"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector.monitoring.svc.cluster.local:4318"
            - name: BEYLA_KUBE_METADATA_ENABLE
              value: "true"
            - name: BEYLA_LOG_LEVEL
              value: "debug"
      volumes:
        - name: beyla-config
          configMap:
            name: beyla-config
        - name: var-run-beyla
          emptyDir: {}

[grcevski]

Hi @AbhijithHarness, can you please share a bit more about which pods are restarting? Are the applications pods restarting or Beyla? Any indication on why the restarts are happening?

[AbhijithHarness]

In logs I am seeing this:

time=2024-04-22T19:32:44.426Z level=DEBUG msg="kubernetes informers started" component=kube.Metadata
fatal error: concurrent map writes
{}
goroutine 47 [running]:
github.com/grafana/beyla/pkg/internal/transform/kube.(*Database).UpdateNewPodsByIPIndex(...)
/opt/app-root/pkg/internal/transform/kube/db.go:113
github.com/grafana/beyla/pkg/internal/transform/kube.StartDatabase.func1({0x1f14220?, 0xc001b22000})
/opt/app-root/pkg/internal/transform/kube/db.go:52 +0x99

![Uploading Screenshot 2024-04-23 at 11.49.11.png…]()

[mariomac]

Thanks for the information @AbhijithHarness . Providing a fix.

[mariomac]

@AbhijithHarness could you try Beyla 1.5.2 and tell us if your problems got fixed?