stale[bot]
commented
2 years ago Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.
We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.
Stalebots are also emotionless and cruel and can close issues which are still very relevant.
If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.
We regularly sort for closed issues which have a stale label sorted by thumbs up.
We may also:
- Mark issues as
revivableif we think it's a valid issue but isn't something we are likely to prioritize in the future (the issue will still remain closed). - Add a
keepalivelabel to silence the stalebot if the issue is very common/popular/important.
We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.
gnadaban
kapopken
LeegacySystem
korenlev
debMan
Describe the bug The logs are sent to Loki for some minutes after deployment time, but after a short while, the log rotation process fails with error
entry too far behind' for stream. Refer to screenshot below.To Reproduce Steps to reproduce the behavior:
grafana/loki:2.4.0fluent/fluent-bit:1.8.8Expected behavior The stack with Loki, fluent-bit, should work correct out of the box with default configs and logs should be rotated to Loki continually.
Environment:
Linux worker7 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/LinuxconfigMaps and logs
Click to expand configMaps
**fluent-bit** ``` yaml apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: monitoring labels: k8s-app: fluent-bit data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE filter-kubernetes.conf @INCLUDE output-loki.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 512MB Refresh_Interval 60 Ignore_Older 900 filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On # Example: Use pod .metadata.annotations.fluentbit.io/parser: apache for apache webserver pod K8S-Logging.Exclude On # Example: Use pod .metadata.annotations.fluentbit.io/exclude: "true" to exclude some pods output-loki.conf: | [OUTPUT] Name loki Match * host ${FLUENT_LOKI_HOST} port ${FLUENT_LOKI_PORT} labels job=fluentbit auto_kubernetes_labels on Retry_Limit False [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] # http://rubular.com/r/tjUt3Awgg4 Name cri Format regex Regex ^(?Click to expand manifests
**fluent-bit daemonset** ``` yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit namespace: monitoring labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: k8s-app: fluent-bit-logging template: metadata: labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" annotations: prometheus.io/scrape: "true" prometheus.io/port: "2020" prometheus.io/path: /api/v1/metrics/prometheus spec: containers: - name: fluent-bit image: fluent/fluent-bit:1.8.8 imagePullPolicy: Always ports: - containerPort: 2020 env: - name: FLUENT_LOKI_HOST value: "loki.monitoring.svc.cluster.local" - name: FLUENT_LOKI_PORT value: "3100" volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluent-bit-config mountPath: /fluent-bit/etc/ terminationGracePeriodSeconds: 10 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluent-bit-config configMap: name: fluent-bit-config serviceAccountName: fluent-bit tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - operator: Exists effect: NoExecute - operator: Exists effect: NoSchedule ``` **loki statefulset** ``` yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: loki namespace: monitoring labels: app: loki spec: replicas: 1 selector: matchLabels: app: loki serviceName: loki template: metadata: labels: app: loki annotations: prometheus.io/port: loki prometheus.io/scrape: "true" spec: containers: - name: loki image: grafana/loki:2.4.0 imagePullPolicy: IfNotPresent args: - -config.file=/etc/loki/loki.yml volumeMounts: - name: config mountPath: /etc/loki/loki.yml subPath: loki.yml - name: storage mountPath: /loki ports: - name: loki containerPort: 3100 protocol: TCP volumes: - name: config configMap: name: loki-configmap defaultMode: 0644 items: - key: loki.yml path: loki.yml volumeClaimTemplates: - metadata: name: storage spec: accessModes: - ReadWriteOnce storageClassName: standard resources: requests: storage: 20Gi ```Click to expand logs
**loki:** ``` level=info ts=2021-11-08T07:58:04.591683804Z caller=table.go:335 msg="uploading table index_18937" level=info ts=2021-11-08T07:58:04.591725329Z caller=table.go:362 msg="finished uploading table index_18937" level=info ts=2021-11-08T07:58:04.591746668Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18937" level=info ts=2021-11-08T07:58:04.591853861Z caller=table.go:335 msg="uploading table index_18938" level=info ts=2021-11-08T07:58:04.591878254Z caller=table.go:362 msg="finished uploading table index_18938" level=info ts=2021-11-08T07:58:04.591897555Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18938" level=info ts=2021-11-08T07:58:04.591951339Z caller=table_manager.go:207 msg="syncing tables" level=info ts=2021-11-08T07:58:04.5923721Z caller=table.go:432 msg="downloading object from storage with key compactor-1636357989.gz" level=info ts=2021-11-08T07:58:04.602597428Z caller=util.go:109 msg="downloaded file compactor-1636357989.gz from table index_18938" level=warn ts=2021-11-08T07:58:05.06671216Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=518.5µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:10.952917405 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=info ts=2021-11-08T07:58:05.193732292Z caller=checkpoint.go:617 msg="starting checkpoint" level=info ts=2021-11-08T07:58:05.195402519Z caller=checkpoint.go:342 msg="attempting checkpoint for" dir=/loki/wal/checkpoint.000580 level=warn ts=2021-11-08T07:58:06.521557442Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=514.51µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:45.438042792 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:09.948210463Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=143.027µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 00:03:24.914575758 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"cc7ff4cd9\", job=\"fluentbit\", name=\"weave-net\", pod_template_generation=\"1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:10.461808628Z caller=grpc_logging.go:38 duration=192.427µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:47.542087159 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:11.459817508Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=223.127µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:48.014915049 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:11.96483226Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=146.142µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:15.947323971 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:13.461010974Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=241.703µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:48.378003942 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:15.487658545Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=1.259455ms err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:48.813733912 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:16.06807484Z caller=grpc_logging.go:38 duration=780.916µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:17.946487602 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:18.093791067Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=1.354953ms err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:18.945881882 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:21.069031397Z caller=grpc_logging.go:38 duration=971.185µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:19.946430392 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:21.454563859Z caller=grpc_logging.go:38 duration=229.498µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:51.606067797 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:23.064012534Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=463.372µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:20.947347535 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:30.057606556Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=346.645µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:25.94781932 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:30.456170715Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=359.149µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:53.934053461 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:32.457966365Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=342.277µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:54.234289315 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:36.070225801Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=416.028µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:29.949272383 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:36.454696023Z caller=grpc_logging.go:38 duration=191.046µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:57.677493397 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:39.453512515Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=141.017µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:54:59.931919193 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:44.066385432Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=249.337µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:33.947784842 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:48.460168564Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=618.286µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:03.126170952 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:53.062454284Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=271.631µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:38.946548036 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:53.457442151Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=335.362µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:05.65140952 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:57.181229911Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=199.78µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:39.949027228 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:58:57.463552519Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=272.574µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:08.657188651 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:00.462710921Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=268.407µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:09.29720918 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"kube-apiserver\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:01.459415756Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=374.167µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:09.58743336 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=info ts=2021-11-08T07:59:04.590592484Z caller=table_manager.go:171 msg="uploading tables" level=info ts=2021-11-08T07:59:04.590865182Z caller=table.go:335 msg="uploading table index_18937" level=info ts=2021-11-08T07:59:04.590897649Z caller=table.go:362 msg="finished uploading table index_18937" level=info ts=2021-11-08T07:59:04.59092121Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18937" level=info ts=2021-11-08T07:59:04.59102842Z caller=table.go:335 msg="uploading table index_18938" level=info ts=2021-11-08T07:59:04.591071354Z caller=table.go:362 msg="finished uploading table index_18938" level=info ts=2021-11-08T07:59:04.591096869Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18938" level=info ts=2021-11-08T07:59:04.59154821Z caller=table.go:335 msg="uploading table index_18939" level=info ts=2021-11-08T07:59:04.591577465Z caller=table.go:362 msg="finished uploading table index_18939" level=info ts=2021-11-08T07:59:04.591603073Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18939" level=warn ts=2021-11-08T07:59:06.06332854Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=317.12µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:43.947325032 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:11.464381221Z caller=grpc_logging.go:38 duration=302.452µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:12.662852997 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:13.064297645Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=249.101µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:46.949587603 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:14.458798062Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=246.73µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:13.602072931 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:16.066056287Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=211.695µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:48.948461444 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:20.467616378Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=317.145µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:14.656322977 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:25.070239531Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=353.233µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:54.94681042 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:31.521493752Z caller=grpc_logging.go:38 duration=209.421µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:19.119539715 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:32.063163943Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=269.411µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:58.946318944 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:34.124292096Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=246.587µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:30:59.947330431 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:34.455483418Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=224.217µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:20.948708321 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:36.130188078Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=238.979µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:00.948566535 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:37.456537423Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=268.495µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:21.700361483 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:39.135290616Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=2.534623ms err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:01.947427809 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:43.459390927Z caller=grpc_logging.go:38 duration=377.169µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:25.521011172 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:45.456598151Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=421.357µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:26.491297215 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:46.069639284Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=430.301µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:06.949055118 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:49.518136289Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=176.722µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:27.614544927 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:51.067015428Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=200.069µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:08.950135255 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:52.457318994Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=428.939µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:28.650839348 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:53.068671539Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=820.122µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:09.948581761 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:53.954241979Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=352.201µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 00:04:44.931099735 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"cc7ff4cd9\", job=\"fluentbit\", name=\"weave-net\", pod_template_generation=\"1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:56.069902629Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=781.021µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:10.948803716 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T07:59:58.070122688Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=369.341µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:11.947572938 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:00.090669646Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=264.081µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:12.946899103 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:03.065457127Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=253.996µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:13.947763366 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=info ts=2021-11-08T08:00:04.59122197Z caller=table_manager.go:171 msg="uploading tables" level=info ts=2021-11-08T08:00:04.593341525Z caller=table.go:335 msg="uploading table index_18938" level=info ts=2021-11-08T08:00:04.593372579Z caller=table.go:362 msg="finished uploading table index_18938" level=info ts=2021-11-08T08:00:04.593388143Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18938" level=info ts=2021-11-08T08:00:04.593442489Z caller=table.go:335 msg="uploading table index_18939" level=info ts=2021-11-08T08:00:04.593460216Z caller=table.go:362 msg="finished uploading table index_18939" level=info ts=2021-11-08T08:00:04.593475028Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18939" level=info ts=2021-11-08T08:00:04.593520515Z caller=table.go:335 msg="uploading table index_18937" level=info ts=2021-11-08T08:00:04.593535163Z caller=table.go:362 msg="finished uploading table index_18937" level=info ts=2021-11-08T08:00:04.593547001Z caller=table.go:420 msg="cleaning up unwanted dbs from table index_18937" level=warn ts=2021-11-08T08:00:05.063592953Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=207.649µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:14.948261937 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:07.064757046Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=253.858µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:15.94798423 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:10.068168427Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=230.842µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:16.947864386 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:12.027094999Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=259.434µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 00:05:02.940574404 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:12.073752358Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=802.012µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:17.948789497 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:14.06464064Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=272.862µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:18.947236615 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:15.467808525Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=800.938µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:33.108950394 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:16.471273572Z caller=grpc_logging.go:38 duration=431.518µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:32.377664394 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"kube-apiserver\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:17.06825146Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=478.387µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:19.948048005 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:18.521628157Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=305.311µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:32.667948171 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:21.457304608Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=348.514µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:34.663845185 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:22.066823015Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=281.219µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:22.948305496 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:24.468827246Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=333.336µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:36.147998579 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:26.406337479Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=263.583µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:37.133064907 +0000 UTC ignored, reason: 'entry too far behind' for stream: {component=\"etcd\", job=\"fluentbit\", tier=\"control-plane\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:29.070290174Z caller=grpc_logging.go:38 duration=516.982µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-08 03:31:26.948182036 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" level=warn ts=2021-11-08T08:00:31.528954327Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=328.482µs err="rpc error: code = Code(400) desc = entry with timestamp 2021-11-07 17:55:40.72878301 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash=\"67f7f7d9dd\", job=\"fluentbit\", k8s_app=\"fluent-bit-logging\", kubernetes_io_cluster_service=\"true\", pod_template_generation=\"7\", version=\"v1\"},\ntotal ignored: 1 out of 1" msg="gRPC\n" ``` **fluent-bit** ``` [2021/11/08 08:57:30] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:16:40.343542113 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="679796998d", job="fluentbit", name="node-exporter", pod_template_generation="1"}, total ignored: 1 out of 1 [2021/11/08 08:57:32] [ info] [input] tail.0 resume (mem buf overlimit) [2021/11/08 08:57:32] [ warn] [input] tail.0 paused (mem buf overlimit) [2021/11/08 08:57:33] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:16:45.132328335 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:57:39] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:16:58.029408851 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:57:40] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:16:59.995248574 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:57:46] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:16:45.142135689 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="679796998d", job="fluentbit", name="node-exporter", pod_template_generation="1"}, total ignored: 1 out of 1 [2021/11/08 08:57:48] [ info] [input] tail.0 resume (mem buf overlimit) [2021/11/08 08:57:48] [ warn] [input] tail.0 paused (mem buf overlimit) [2021/11/08 08:57:49] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:02.002240917 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:57:54] [ info] [input] tail.0 resume (mem buf overlimit) [2021/11/08 08:57:54] [ warn] [input] tail.0 paused (mem buf overlimit) [2021/11/08 08:57:59] [ info] [input] tail.0 resume (mem buf overlimit) [2021/11/08 08:57:59] [ warn] [input] tail.0 paused (mem buf overlimit) [2021/11/08 08:58:01] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:38.988382679 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:03] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:39.000397162 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:04] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:39.994000246 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:07] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:40.982591975 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:08] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:42.974844655 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:10] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:44.012775829 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:13] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:46.937169884 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:16] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:48.978945976 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:18] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:52.954897473 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:19] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:52.964390333 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:23] [ info] [input] tail.0 resume (mem buf overlimit) [2021/11/08 08:58:23] [ warn] [input] tail.0 paused (mem buf overlimit) [2021/11/08 08:58:24] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:54.965749035 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:26] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:17:58.928269421 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:29] [ info] [input] tail.0 resume (mem buf overlimit) [2021/11/08 08:58:29] [ warn] [input] tail.0 paused (mem buf overlimit) [2021/11/08 08:58:30] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:18:12.004545283 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:31] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:18:20.997447046 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:38] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:18:25.964750439 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 [2021/11/08 08:58:39] [error] [output:loki:loki.0] loki.monitoring.svc.cluster.local:3100, HTTP status=400 Not retrying. entry with timestamp 2021-11-07 13:18:25.972501928 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="67f7f7d9dd", job="fluentbit", k8s_app="fluent-bit-logging", kubernetes_io_cluster_service="true", pod_template_generation="7", version="v1"}, total ignored: 1 out of 1 2021-11-08T08:58:39.598604717Z ```Click to expand grafana screenshot
There are logs only from first moments of deployment 