search

vexxhost / atmosphere

Simple & easy private cloud platform featuring VMs, Kubernetes & bare-metal
87 stars 21 forks source link

bug: some logs are missing in loki #1630

Closed okozachenko1203 closed 2 weeks ago

okozachenko1203 commented 1 month ago

typically, logs of oauth2-proxy container of alertmanager pod and haproxy of percona-xtradb-haproxy replicas are missing.

loki outputs this kinda error.

level=error ts=2024-07-18T13:54:15.969797889Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="entry for stream '{container_image=\"docker.io/percona/percona-xtradb-cluster-operator:1.13.0-haproxy\", container_name=\"haproxy\", node=\"instance\", pod_ip=\"10.0.0.111\", pod_label_app_kubernetes_io_component=\"haproxy\", pod_label_app_kubernetes_io_instance=\"percona-xtradb\", pod_label_app_kubernetes_io_managed_by=\"percona-xtradb-cluster-operator\", pod_label_app_kubernetes_io_name=\"percona-xtradb-cluster\", pod_label_app_kubernetes_io_part_of=\"percona-xtradb-cluster\", pod_label_apps_kubernetes_io_pod_index=\"0\", pod_label_controller_revision_hash=\"percona-xtradb-haproxy-78b9d6c4ff\", pod_label_statefulset_kubernetes_io_pod_name=\"percona-xtradb-haproxy-0\", pod_name=\"percona-xtradb-haproxy-0\", pod_namespace=\"openstack\", pod_uid=\"7dddbb4b-ec70-464c-a3ce-ab89276748f4\", source_type=\"kubernetes_logs\"}' has 16 label names; limit 15" org_id=fake
level=error ts=2024-07-18T13:54:51.012725148Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="entry for stream '{container_image=\"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0\", container_name=\"oauth2-proxy\", node=\"instance\", pod_ip=\"10.0.0.103\", pod_label_alertmanager=\"kube-prometheus-stack-alertmanager\", pod_label_app_kubernetes_io_instance=\"kube-prometheus-stack-alertmanager\", pod_label_app_kubernetes_io_managed_by=\"prometheus-operator\", pod_label_app_kubernetes_io_name=\"alertmanager\", pod_label_app_kubernetes_io_version=\"0.27.0\", pod_label_apps_kubernetes_io_pod_index=\"0\", pod_label_controller_revision_hash=\"alertmanager-kube-prometheus-stack-alertmanager-845c6f44c5\", pod_label_statefulset_kubernetes_io_pod_name=\"alertmanager-kube-prometheus-stack-alertmanager-0\", pod_name=\"alertmanager-kube-prometheus-stack-alertmanager-0\", pod_namespace=\"monitoring\", pod_uid=\"dbebf18e-cdf4-4508-8782-1619a743fd06\", source_type=\"kubernetes_logs\"}' has 16 label names; limit 15" org_id=fake

Need to drop unnecessary labels by setting relabel_configs properly

okozachenko1203 commented 1 month ago

default label limit is 15 some are 22

{container_image="quay.io/ceph/ceph:v18.2.1",
 container_name="rgw",
 node="instance",
 pod_ip="10.0.0.139",
 pod_label_app="rook-ceph-rgw",
 pod_label_app_kubernetes_io_component="cephobjectstores.ceph.rook.io",
 pod_label_app_kubernetes_io_created_by="rook-ceph-operator",
 pod_label_app_kubernetes_io_instance="ceph",
 pod_label_app_kubernetes_io_managed_by="rook-ceph-operator",
 pod_label_app_kubernetes_io_name="ceph-rgw",
 pod_label_app_kubernetes_io_part_of="ceph",
 pod_label_ceph_daemon_id="ceph",
 pod_label_ceph_daemon_type="rgw",
 pod_label_pod_template_hash="ccdd949f8",
 pod_label_rgw="ceph",
 pod_label_rook_cluster="openstack",
 pod_label_rook_io_operator_namespace="rook-ceph",
 pod_label_rook_object_store="ceph",
 pod_name="rook-ceph-rgw-ceph-a-ccdd949f8-f4j6w",
 pod_namespace="openstack",
 pod_uid="533bc6cd-6fe2-4b88-8719-6c27550f4276",
 source_type="kubernetes_logs"}

some are 16

{container_image="quay.io/oauth2-proxy/oauth2-proxy:v7.6.0",
 container_name="oauth2-proxy",
 node="instance",
 pod_ip="10.0.0.18",
 pod_label_app_kubernetes_io_instance="kube-prometheus-stack-prometheus",
 pod_label_app_kubernetes_io_managed_by="prometheus-operator",
 pod_label_app_kubernetes_io_name="prometheus",
 pod_label_app_kubernetes_io_version="2.51.2",
 pod_label_apps_kubernetes_io_pod_index="0",
 pod_label_controller_revision_hash="prometheus-kube-prometheus-stack-prometheus-6d58ccbf8b",
 pod_label_operator_prometheus_io_name="kube-prometheus-stack-prometheus",
 pod_label_operator_prometheus_io_shard="0",
 pod_label_prometheus="kube-prometheus-stack-prometheus",
 pod_label_statefulset_kubernetes_io_pod_name="prometheus-kube-prometheus-stack-prometheus-0",
 pod_name="prometheus-kube-prometheus-stack-prometheus-0",
 pod_namespace="monitoring",
 pod_uid="30d91b1b-c5b7-4302-9e7e-5cf1ccec15ce",
 source_type="kubernetes_logs"}