search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.42k stars 205 forks source link

Agent statefulset rollout causes metric data loss #222

Open LasseHels opened 8 months ago

LasseHels commented 8 months ago

What's wrong?

We run two Grafana agents as a statefulset in Kubernetes:

➜  ~ k get pod | grep grafana-agent-metrics-app
grafana-agent-metrics-app-0                    1/1     Running             0               70m
grafana-agent-metrics-app-1                    1/1     Running             0               75m
➜  ~ k get sts | grep grafana-agent-metrics-app
grafana-agent-metrics-app     2/2     52d

During rollouts of this statefulset, we observe gaps in data written by the agents. Our investigation suggests that this is what is happening:

  1. grafana-agent-metrics-app-0 is terminated.
  2. grafana-agent-metrics-app-0 restarts and starts loading its WAL segments[^1]. At this point, its Kubernetes ready status is 0/1.
  3. grafana-agent-metrics-app-0 finishes loading its WAL segments.
  4. grafana-agent-metrics-app-0 starts replaying its WAL[^2]. At this point, its Kubernetes ready status changes to 1/1.
  5. Because grafana-agent-metrics-app-0 has a ready status of 1/1, Kubernetes terminates grafana-agent-metrics-app-1. Once grafana-agent-metrics-app-1 is terminated, neither agents are writing data.
  6. grafana-agent-metrics-app-1 restarts and starts loading its WAL segments.
  7. grafana-agent-metrics-app-1 finishes loading its WAL segments.
  8. grafana-agent-metrics-app-0 finishes replaying its WAL.
  9. grafana-agent-metrics-app-1 starts replaying its WAL.
  10. grafana-agent-metrics-app-1 finishes replaying its WAL.

Visualisation: Screenshot 2024-03-08 at 09 45 53

We observe that some (but not all) of the missing data is backfilled once the agents resume writing data

The crux of the issue appears to be that an agent's /-/ready endpoints starts returning 200 before the agent is writing data. This causes Kubernetes to terminate the second agent before the first agent is writing data, leading to data gaps/loss.

Our understanding is that an agent can read but not write data while replaying its WAL.

Our agents use these probes:

          readinessProbe:
            failureThreshold: 1
            httpGet:
              port: http-metrics
              path: /-/ready
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          livenessProbe:
            failureThreshold: 1
            httpGet:
              port: http-metrics
              path: /-/healthy
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3

Below are logs from a rollout that caused the issue. Note that in this rollout, grafana-agent-metrics-app-1 was terminated before grafana-agent-metrics-app-0.

grafana-agent-metrics-app-1 ``` 2024-03-08 07:55:38.359 ts=2024-03-08T07:55:38.359762814Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage resharding" from=7 to=11 2024-03-08 07:55:28.359 ts=2024-03-08T07:55:28.359592696Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage resharding" from=12 to=7 2024-03-08 07:55:08.359 ts=2024-03-08T07:55:08.359055237Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage resharding" from=9 to=12 2024-03-08 07:54:18.359 ts=2024-03-08T07:54:18.359396778Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage resharding" from=13 to=9 2024-03-08 07:53:18.359 ts=2024-03-08T07:53:18.359245309Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage resharding" from=9 to=13 2024-03-08 07:53:08.359 ts=2024-03-08T07:53:08.359701932Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage resharding" from=50 to=9 2024-03-08 07:48:28.359 ts=2024-03-08T07:48:28.359046851Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage resharding" from=1 to=50 2024-03-08 07:48:19.488 ts=2024-03-08T07:48:19.488652935Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Done replaying WAL" duration=15m1.129877304s 2024-03-08 07:33:18.358 ts=2024-03-08T07:33:18.358857073Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Replaying WAL" queue=565b90-4ddc16 2024-03-08 07:33:18.358 ts=2024-03-08T07:33:18.358698348Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Starting scraped metadata watcher" 2024-03-08 07:33:18.358 ts=2024-03-08T07:33:18.358671097Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Starting WAL watcher" queue=565b90-4ddc16 2024-03-08 07:33:18.357 ts=2024-03-08T07:33:18.357813351Z caller=kubernetes.go:329 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component="discovery manager" discovery=kubernetes config=integrations/kubernetes/agent msg="Using pod service account via in-cluster config" 2024-03-08 07:33:18.357 ts=2024-03-08T07:33:18.357631942Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18511 maxSegment=18511 2024-03-08 07:33:18.357 ts=2024-03-08T07:33:18.357425395Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18510 maxSegment=18511 2024-03-08 07:33:17.492 ts=2024-03-08T07:33:17.492419828Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18509 maxSegment=18511 2024-03-08 07:33:12.397 ts=2024-03-08T07:33:12.397526137Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18508 maxSegment=18511 2024-03-08 07:33:06.921 ts=2024-03-08T07:33:06.921542091Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18507 maxSegment=18511 2024-03-08 07:33:01.227 ts=2024-03-08T07:33:01.222612917Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18506 maxSegment=18511 2024-03-08 07:32:55.515 ts=2024-03-08T07:32:55.515147455Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18505 maxSegment=18511 2024-03-08 07:32:49.266 ts=2024-03-08T07:32:49.266601374Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18504 maxSegment=18511 2024-03-08 07:32:43.356 ts=2024-03-08T07:32:43.356840796Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18503 maxSegment=18511 2024-03-08 07:32:37.776 ts=2024-03-08T07:32:37.776719186Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18502 maxSegment=18511 2024-03-08 07:32:32.145 ts=2024-03-08T07:32:32.145403053Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18501 maxSegment=18511 2024-03-08 07:32:27.008 ts=2024-03-08T07:32:27.008257941Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18500 maxSegment=18511 2024-03-08 07:32:19.816 ts=2024-03-08T07:32:19.816211653Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18499 maxSegment=18511 2024-03-08 07:32:14.481 ts=2024-03-08T07:32:14.480920927Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18498 maxSegment=18511 2024-03-08 07:32:09.145 ts=2024-03-08T07:32:09.145613439Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18497 maxSegment=18511 2024-03-08 07:32:03.814 ts=2024-03-08T07:32:03.814619273Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18496 maxSegment=18511 2024-03-08 07:31:58.383 ts=2024-03-08T07:31:58.383534257Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18495 maxSegment=18511 2024-03-08 07:31:52.955 ts=2024-03-08T07:31:52.955188433Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18494 maxSegment=18511 2024-03-08 07:31:47.984 ts=2024-03-08T07:31:47.984799465Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18493 maxSegment=18511 2024-03-08 07:31:42.785 ts=2024-03-08T07:31:42.78534433Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18492 maxSegment=18511 2024-03-08 07:31:37.142 ts=2024-03-08T07:31:37.142007611Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18491 maxSegment=18511 2024-03-08 07:31:32.057 ts=2024-03-08T07:31:32.057269021Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18490 maxSegment=18511 2024-03-08 07:31:26.210 ts=2024-03-08T07:31:26.210578994Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18489 maxSegment=18511 2024-03-08 07:31:21.072 ts=2024-03-08T07:31:21.072675215Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18488 maxSegment=18511 2024-03-08 07:31:15.316 ts=2024-03-08T07:31:15.316220999Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18487 maxSegment=18511 2024-03-08 07:31:10.253 ts=2024-03-08T07:31:10.253217566Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18486 maxSegment=18511 2024-03-08 07:31:04.810 ts=2024-03-08T07:31:04.810559417Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18485 maxSegment=18511 2024-03-08 07:30:59.143 ts=2024-03-08T07:30:59.143367837Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18484 maxSegment=18511 2024-03-08 07:30:58.813 ts=2024-03-08T07:30:58.813113788Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18483 maxSegment=18511 2024-03-08 07:30:53.726 ts=2024-03-08T07:30:53.725956137Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18482 maxSegment=18511 2024-03-08 07:30:48.013 ts=2024-03-08T07:30:48.01348954Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18481 maxSegment=18511 2024-03-08 07:30:42.205 ts=2024-03-08T07:30:42.205713424Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18480 maxSegment=18511 2024-03-08 07:30:36.898 ts=2024-03-08T07:30:36.89827613Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18479 maxSegment=18511 2024-03-08 07:30:31.210 ts=2024-03-08T07:30:31.210208443Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18478 maxSegment=18511 2024-03-08 07:30:25.603 ts=2024-03-08T07:30:25.6031352Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18477 maxSegment=18511 2024-03-08 07:30:18.416 ts=2024-03-08T07:30:18.416491949Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18476 maxSegment=18511 2024-03-08 07:30:13.400 ts=2024-03-08T07:30:13.40028769Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18475 maxSegment=18511 2024-03-08 07:30:07.783 ts=2024-03-08T07:30:07.783036049Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18474 maxSegment=18511 2024-03-08 07:30:02.279 ts=2024-03-08T07:30:02.278933103Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18473 maxSegment=18511 2024-03-08 07:29:56.875 ts=2024-03-08T07:29:56.875477536Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18472 maxSegment=18511 2024-03-08 07:29:51.771 ts=2024-03-08T07:29:51.770993019Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18471 maxSegment=18511 2024-03-08 07:29:46.266 ts=2024-03-08T07:29:46.266705945Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18470 maxSegment=18511 2024-03-08 07:29:40.751 ts=2024-03-08T07:29:40.751026063Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18469 maxSegment=18511 2024-03-08 07:29:40.295 ts=2024-03-08T07:29:40.295240979Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18468 maxSegment=18511 2024-03-08 07:29:34.867 ts=2024-03-08T07:29:34.86779146Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18467 maxSegment=18511 2024-03-08 07:29:29.581 ts=2024-03-08T07:29:29.581671554Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18466 maxSegment=18511 2024-03-08 07:29:23.986 ts=2024-03-08T07:29:23.986110035Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18465 maxSegment=18511 2024-03-08 07:29:18.442 ts=2024-03-08T07:29:18.442041496Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18464 maxSegment=18511 2024-03-08 07:29:12.786 ts=2024-03-08T07:29:12.786230548Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18463 maxSegment=18511 2024-03-08 07:29:07.700 ts=2024-03-08T07:29:07.700058339Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18462 maxSegment=18511 2024-03-08 07:29:02.079 ts=2024-03-08T07:29:02.079619415Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18461 maxSegment=18511 2024-03-08 07:28:56.200 ts=2024-03-08T07:28:56.200476698Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18460 maxSegment=18511 2024-03-08 07:28:50.432 ts=2024-03-08T07:28:50.432675629Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18459 maxSegment=18511 2024-03-08 07:28:44.919 ts=2024-03-08T07:28:44.919208384Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18458 maxSegment=18511 2024-03-08 07:28:39.475 ts=2024-03-08T07:28:39.475758002Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18457 maxSegment=18511 2024-03-08 07:28:34.169 ts=2024-03-08T07:28:34.169883019Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18456 maxSegment=18511 2024-03-08 07:28:28.883 ts=2024-03-08T07:28:28.883252135Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18455 maxSegment=18511 2024-03-08 07:28:23.336 ts=2024-03-08T07:28:23.336342861Z caller=wal.go:236 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL checkpoint loaded" 2024-03-08 07:28:09.559 ts=2024-03-08T07:28:09.559252071Z caller=wal.go:211 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="replaying WAL, this may take a while" dir=/data/grafana-agent-wal/565b903835c63f30cc4e6a22792cd869/wal 2024-03-08 07:28:09.556 ts=2024-03-08T07:28:09.556090198Z caller=reporter.go:103 level=info msg="running usage stats reporter" 2024-03-08 07:28:09.553 ts=2024-03-08T07:28:09.553257088Z caller=zapadapter.go:78 level=info component=traces msg="Traces Logger Initialized" 2024-03-08 07:28:09.552 ts=2024-03-08T07:28:09.5519784Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set" 2024-03-08 07:28:09.551 ts=2024-03-08T07:28:09.551915064Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config" 2024-03-08 07:28:09.551 ts=2024-03-08T07:28:09.551614812Z caller=server.go:190 level=info msg="server listening on addresses" http=[::]:12345 grpc=[::]:12346 http_tls_enabled=false grpc_tls_enabled=false 2024-03-08 07:28:09.551 ts=2024-03-08T07:28:09.551435957Z caller=main.go:71 level=info boringcryptoenabled=false 2024-03-08 07:28:04.022 ts=2024-03-08T07:28:04.02204794Z caller=main.go:84 level=info msg="agent exiting" 2024-03-08 07:28:04.022 ts=2024-03-08T07:28:04.021997276Z caller=manager.go:271 level=info agent=prometheus msg="stopped instance" instance=565b903835c63f30cc4e6a22792cd869 2024-03-08 07:28:04.021 ts=2024-03-08T07:28:04.021695039Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage stopped." 2024-03-08 07:27:57.763 ts=2024-03-08T07:27:57.76382651Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Scraped metadata watcher stopped" 2024-03-08 07:27:57.763 ts=2024-03-08T07:27:57.763806212Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Stopping metadata watcher..." 2024-03-08 07:27:57.763 ts=2024-03-08T07:27:57.763769884Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="WAL watcher stopped" queue=565b90-4ddc16 2024-03-08 07:27:56.707 ts=2024-03-08T07:27:56.707356198Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Stopping remote storage..." 2024-03-08 07:27:56.581 ts=2024-03-08T07:27:56.581424331Z caller=instance.go:355 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="scrape manager stopped" 2024-03-08 07:27:56.581 ts=2024-03-08T07:27:56.581318916Z caller=instance.go:376 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="closing storage..." 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275739733Z caller=instance.go:335 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="truncation loop stopped" 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275660446Z caller=instance.go:625 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="discovery manager stopped" 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.27562736Z caller=instance.go:628 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="stopping discovery manager..." 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275596382Z caller=config_watcher.go:104 level=info agent=prometheus component=cluster msg="config watcher run loop exiting" 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275592343Z caller=instance.go:361 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="stopping scrape manager..." 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275580802Z caller=instance.go:339 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="stopping truncation loop..." 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275568339Z caller=node.go:169 level=info agent=prometheus component=cluster msg="node run loop exiting" 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275551457Z caller=node.go:345 level=info agent=prometheus component=cluster msg="node shut down" 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275527241Z caller=node.go:319 level=info agent=prometheus component=cluster msg="shutting down node" 2024-03-08 07:27:56.275 ts=2024-03-08T07:27:56.275121261Z caller=signals.go:62 level=info msg="=== received SIGINT/SIGTERM ===\n*** exiting" ```
grafana-agent-metrics-app-0 ``` 2024-03-08 07:39:59.347 ts=2024-03-08T07:39:59.347362695Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Replaying WAL" queue=565b90-4ddc16 2024-03-08 07:39:59.347 ts=2024-03-08T07:39:59.347163461Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Starting scraped metadata watcher" 2024-03-08 07:39:59.347 ts=2024-03-08T07:39:59.347144906Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Starting WAL watcher" queue=565b90-4ddc16 2024-03-08 07:39:59.346 ts=2024-03-08T07:39:59.346230994Z caller=kubernetes.go:329 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component="discovery manager" discovery=kubernetes config=integrations/kubernetes/agent msg="Using pod service account via in-cluster config" 2024-03-08 07:39:59.346 ts=2024-03-08T07:39:59.346077405Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18463 maxSegment=18463 2024-03-08 07:39:59.346 ts=2024-03-08T07:39:59.345863492Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18462 maxSegment=18463 2024-03-08 07:39:56.041 ts=2024-03-08T07:39:56.041377489Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18461 maxSegment=18463 2024-03-08 07:39:50.575 ts=2024-03-08T07:39:50.575402716Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18460 maxSegment=18463 2024-03-08 07:39:43.448 ts=2024-03-08T07:39:43.448285661Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18459 maxSegment=18463 2024-03-08 07:39:38.025 ts=2024-03-08T07:39:38.025402572Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18458 maxSegment=18463 2024-03-08 07:39:31.837 ts=2024-03-08T07:39:31.837514286Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18457 maxSegment=18463 2024-03-08 07:39:25.668 ts=2024-03-08T07:39:25.668733551Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18456 maxSegment=18463 2024-03-08 07:39:20.116 ts=2024-03-08T07:39:20.116162711Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18455 maxSegment=18463 2024-03-08 07:39:13.494 ts=2024-03-08T07:39:13.494669263Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18454 maxSegment=18463 2024-03-08 07:39:08.213 ts=2024-03-08T07:39:08.213177333Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18453 maxSegment=18463 2024-03-08 07:39:03.091 ts=2024-03-08T07:39:03.090980662Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18452 maxSegment=18463 2024-03-08 07:38:56.470 ts=2024-03-08T07:38:56.46991153Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18451 maxSegment=18463 2024-03-08 07:38:49.833 ts=2024-03-08T07:38:49.833595801Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18450 maxSegment=18463 2024-03-08 07:38:44.537 ts=2024-03-08T07:38:44.537341371Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18449 maxSegment=18463 2024-03-08 07:38:38.935 ts=2024-03-08T07:38:38.935756568Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18448 maxSegment=18463 2024-03-08 07:38:32.677 ts=2024-03-08T07:38:32.677507045Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18447 maxSegment=18463 2024-03-08 07:38:27.294 ts=2024-03-08T07:38:27.294061606Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18446 maxSegment=18463 2024-03-08 07:38:20.834 ts=2024-03-08T07:38:20.834198463Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18445 maxSegment=18463 2024-03-08 07:38:14.101 ts=2024-03-08T07:38:14.101202961Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18444 maxSegment=18463 2024-03-08 07:38:08.405 ts=2024-03-08T07:38:08.405194459Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18443 maxSegment=18463 2024-03-08 07:38:03.271 ts=2024-03-08T07:38:03.271525549Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18442 maxSegment=18463 2024-03-08 07:37:56.062 ts=2024-03-08T07:37:56.062616472Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18441 maxSegment=18463 2024-03-08 07:37:50.778 ts=2024-03-08T07:37:50.777957506Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18440 maxSegment=18463 2024-03-08 07:37:42.650 ts=2024-03-08T07:37:42.649969614Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18439 maxSegment=18463 2024-03-08 07:37:36.543 ts=2024-03-08T07:37:36.543696616Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18438 maxSegment=18463 2024-03-08 07:37:31.365 ts=2024-03-08T07:37:31.365247884Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18437 maxSegment=18463 2024-03-08 07:37:25.157 ts=2024-03-08T07:37:25.157209697Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18436 maxSegment=18463 2024-03-08 07:37:24.195 ts=2024-03-08T07:37:24.195425066Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18435 maxSegment=18463 2024-03-08 07:37:18.869 ts=2024-03-08T07:37:18.869640343Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18434 maxSegment=18463 2024-03-08 07:37:12.976 ts=2024-03-08T07:37:12.976706282Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18433 maxSegment=18463 2024-03-08 07:37:07.640 ts=2024-03-08T07:37:07.63988936Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18432 maxSegment=18463 2024-03-08 07:37:02.244 ts=2024-03-08T07:37:02.244602938Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18431 maxSegment=18463 2024-03-08 07:36:56.851 ts=2024-03-08T07:36:56.851035446Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18430 maxSegment=18463 2024-03-08 07:36:51.222 ts=2024-03-08T07:36:51.222616259Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18429 maxSegment=18463 2024-03-08 07:36:44.361 ts=2024-03-08T07:36:44.361565813Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18428 maxSegment=18463 2024-03-08 07:36:38.669 ts=2024-03-08T07:36:38.668880119Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18427 maxSegment=18463 2024-03-08 07:36:32.853 ts=2024-03-08T07:36:32.853681378Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18426 maxSegment=18463 2024-03-08 07:36:27.583 ts=2024-03-08T07:36:27.582896823Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18425 maxSegment=18463 2024-03-08 07:36:21.299 ts=2024-03-08T07:36:21.298961854Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18424 maxSegment=18463 2024-03-08 07:36:15.072 ts=2024-03-08T07:36:15.072529956Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18423 maxSegment=18463 2024-03-08 07:36:09.107 ts=2024-03-08T07:36:09.107625208Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18422 maxSegment=18463 2024-03-08 07:36:03.631 ts=2024-03-08T07:36:03.630958936Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18421 maxSegment=18463 2024-03-08 07:35:57.328 ts=2024-03-08T07:35:57.328219245Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18420 maxSegment=18463 2024-03-08 07:35:52.105 ts=2024-03-08T07:35:52.105770295Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18419 maxSegment=18463 2024-03-08 07:35:45.777 ts=2024-03-08T07:35:45.777111426Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18418 maxSegment=18463 2024-03-08 07:35:39.029 ts=2024-03-08T07:35:39.029581187Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18417 maxSegment=18463 2024-03-08 07:35:33.056 ts=2024-03-08T07:35:33.05670398Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18416 maxSegment=18463 2024-03-08 07:35:27.940 ts=2024-03-08T07:35:27.94085596Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18415 maxSegment=18463 2024-03-08 07:35:22.306 ts=2024-03-08T07:35:22.306597698Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18414 maxSegment=18463 2024-03-08 07:35:16.433 ts=2024-03-08T07:35:16.43334893Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18413 maxSegment=18463 2024-03-08 07:35:10.288 ts=2024-03-08T07:35:10.288153436Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18412 maxSegment=18463 2024-03-08 07:35:05.103 ts=2024-03-08T07:35:05.103229346Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18411 maxSegment=18463 2024-03-08 07:35:04.358 ts=2024-03-08T07:35:04.358004127Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18410 maxSegment=18463 2024-03-08 07:34:58.089 ts=2024-03-08T07:34:58.089200373Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18409 maxSegment=18463 2024-03-08 07:34:52.289 ts=2024-03-08T07:34:52.289476478Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18408 maxSegment=18463 2024-03-08 07:34:46.340 ts=2024-03-08T07:34:46.340713048Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18407 maxSegment=18463 2024-03-08 07:34:40.753 ts=2024-03-08T07:34:40.753412722Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18406 maxSegment=18463 2024-03-08 07:34:34.452 ts=2024-03-08T07:34:34.452706198Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18405 maxSegment=18463 2024-03-08 07:34:29.369 ts=2024-03-08T07:34:29.368895082Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18404 maxSegment=18463 2024-03-08 07:34:23.545 ts=2024-03-08T07:34:23.544969917Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18403 maxSegment=18463 2024-03-08 07:34:17.027 ts=2024-03-08T07:34:17.026874511Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18402 maxSegment=18463 2024-03-08 07:34:11.470 ts=2024-03-08T07:34:11.470736294Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18401 maxSegment=18463 2024-03-08 07:34:06.215 ts=2024-03-08T07:34:06.215355299Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18400 maxSegment=18463 2024-03-08 07:34:00.640 ts=2024-03-08T07:34:00.639941224Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18399 maxSegment=18463 2024-03-08 07:33:55.615 ts=2024-03-08T07:33:55.615665156Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18398 maxSegment=18463 2024-03-08 07:33:49.809 ts=2024-03-08T07:33:49.8091616Z caller=wal.go:260 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL segment loaded" segment=18397 maxSegment=18463 2024-03-08 07:33:43.681 ts=2024-03-08T07:33:43.681583929Z caller=wal.go:236 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="WAL checkpoint loaded" 2024-03-08 07:33:29.901 ts=2024-03-08T07:33:29.901106236Z caller=wal.go:211 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="replaying WAL, this may take a while" dir=/data/grafana-agent-wal/565b903835c63f30cc4e6a22792cd869/wal 2024-03-08 07:33:29.897 ts=2024-03-08T07:33:29.897740885Z caller=reporter.go:103 level=info msg="running usage stats reporter" 2024-03-08 07:33:29.894 ts=2024-03-08T07:33:29.894841313Z caller=zapadapter.go:78 level=info component=traces msg="Traces Logger Initialized" 2024-03-08 07:33:29.893 ts=2024-03-08T07:33:29.892938485Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set" 2024-03-08 07:33:29.892 ts=2024-03-08T07:33:29.89279048Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config" 2024-03-08 07:33:29.892 ts=2024-03-08T07:33:29.892433017Z caller=server.go:190 level=info msg="server listening on addresses" http=[::]:12345 grpc=[::]:12346 http_tls_enabled=false grpc_tls_enabled=false 2024-03-08 07:33:29.892 ts=2024-03-08T07:33:29.892206861Z caller=main.go:71 level=info boringcryptoenabled=false 2024-03-08 07:33:26.432 ts=2024-03-08T07:33:26.432071623Z caller=main.go:84 level=info msg="agent exiting" 2024-03-08 07:33:26.432 ts=2024-03-08T07:33:26.432033972Z caller=manager.go:271 level=info agent=prometheus msg="stopped instance" instance=565b903835c63f30cc4e6a22792cd869 2024-03-08 07:33:26.431 ts=2024-03-08T07:33:26.431769784Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Remote storage stopped." 2024-03-08 07:33:19.313 ts=2024-03-08T07:33:19.313300944Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Scraped metadata watcher stopped" 2024-03-08 07:33:19.313 ts=2024-03-08T07:33:19.313283401Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Stopping metadata watcher..." 2024-03-08 07:33:19.313 ts=2024-03-08T07:33:19.313242734Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="WAL watcher stopped" queue=565b90-4ddc16 2024-03-08 07:33:19.269 ts=2024-03-08T07:33:19.268928404Z caller=dedupe.go:112 agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 component=remote level=info remote_name=565b90-4ddc16 url=http://metrics-guard.guard.svc:8080/api/v1/push msg="Stopping remote storage..." 2024-03-08 07:33:19.224 ts=2024-03-08T07:33:19.224777861Z caller=instance.go:355 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="scrape manager stopped" 2024-03-08 07:33:19.224 ts=2024-03-08T07:33:19.224705895Z caller=instance.go:376 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="closing storage..." 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.1167802Z caller=instance.go:335 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="truncation loop stopped" 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116682576Z caller=instance.go:628 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="stopping discovery manager..." 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116661186Z caller=instance.go:625 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="discovery manager stopped" 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116657733Z caller=instance.go:361 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="stopping scrape manager..." 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116634356Z caller=config_watcher.go:104 level=info agent=prometheus component=cluster msg="config watcher run loop exiting" 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116642464Z caller=instance.go:339 level=info agent=prometheus instance=565b903835c63f30cc4e6a22792cd869 msg="stopping truncation loop..." 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116614473Z caller=node.go:169 level=info agent=prometheus component=cluster msg="node run loop exiting" 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116593502Z caller=node.go:345 level=info agent=prometheus component=cluster msg="node shut down" 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.116522579Z caller=node.go:319 level=info agent=prometheus component=cluster msg="shutting down node" 2024-03-08 07:33:19.116 ts=2024-03-08T07:33:19.115937939Z caller=signals.go:62 level=info msg="=== received SIGINT/SIGTERM ===\n*** exiting" ```

[^1]: Indicated by the replaying WAL, this may take a while log message. [^2]: Indicated by the Replaying WAL log message.

Steps to reproduce

  1. Create a statefulset with two agent pods.
  2. Do a rollout of the statefulset.

We are able to consistently reproduce the issue by doing a rollout of the statefulset.

System information

amd64 Ubuntu 22.04.3 LTS

Software version

v0.38.1

Configuration

server:
  log_level: info

metrics:
  wal_directory: /data/grafana-agent-wal
  wal_cleanup_age: "6h"
  global:
    scrape_interval: 15s
    remote_write:
      - url: ${METRICS_REMOTE_WRITE_URL}
        oauth2:
          client_id: $${AUTH_CLIENT_ID}
          client_secret: $${AUTH_CLIENT_SECRET}
          token_url: $${AUTH_TOKEN_URL}
          scopes: ["$${AUTH_SCOPE}"]
        queue_config:
          min_backoff: 1s
          max_backoff: 2m
          max_shards: 50
          capacity: 10000
          max_samples_per_send: 2000
        send_exemplars: true
    external_labels:
      k8s_cluster: ${k8s_cluster}
      region: ${region}
      env: ${env}
      product_id: ${product}
      provider: ${provider}
      metrics_ha: ${k8s_cluster};${region};app
      __replica__: $${POD_NAME}
  configs:
    - name: integrations
      wal_truncate_frequency: "30m"
      scrape_configs:
        - job_name: integrations/kubernetes/agent
          kubernetes_sd_configs:
            - role: pod
          metric_relabel_configs:
            - action: drop
              regex: node_dmi_info
              source_labels:
                - __name__
          relabel_configs:
            - source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_scrape
              action: keep
              regex: true
            - source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_path
              action: replace
              target_label: __metrics_path__
              regex: (.+)
            - source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_path
              action: replace
              target_label: __metrics_path__
              regex: (.+)
            - source_labels:
                - __address__
                - __meta_kubernetes_pod_annotation_prometheus_io_port
              action: replace
              regex: (.+):(?:\d+);(\d+)
              replacement: $${1}:$${2}
              target_label: __address__
            - source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
                - __meta_kubernetes_pod_label_app
                - __tmp_controller_name
                - __meta_kubernetes_pod_name
              regex: ^;*([^;]+)(;.*)?$
              action: replace
              target_label: app
            - source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_component
                - __meta_kubernetes_pod_label_component
              regex: ^;*([^;]+)(;.*)?$
              action: replace
              target_label: component
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_node_name
              target_label: node_name
            - action: replace
              source_labels:
                - __meta_kubernetes_namespace
              target_label: namespace
            - action: replace
              replacement: $1
              separator: /
              source_labels:
                - namespace
                - app
              target_label: job
            ### Use component in job for cortex,loki,tempo
            - action: replace
              replacement: $${1}$${2}
              regex: (loki/|cortex/|tempo/|mimir/)(.*)
              separator: /
              source_labels:
                - namespace
                - component
              target_label: job
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_name
              target_label: pod
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_container_name
              target_label: container
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_label_product_id
                - __meta_kubernetes_pod_label_product
              regex: ^;*([^;]+)(;.*)?$
              target_label: product_id

Logs

No response

LasseHels commented 8 months ago
statefulset configuration ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: grafana-agent spec: persistentVolumeClaimRetentionPolicy: whenDeleted: Delete whenScaled: Delete serviceName: grafana-agent volumeClaimTemplates: - metadata: name: storage spec: storageClassName: default resources: requests: storage: 25Gi accessModes: - ReadWriteOnce volumeMode: Filesystem replicas: 2 revisionHistoryLimit: 3 selector: matchLabels: name: grafana-agent template: metadata: labels: name: grafana-agent annotations: prometheus.io/scrape: "true" prometheus.io/port: "12345" spec: nodeSelector: kubernetes.azure.com/scalesetpriority: spot tolerations: - key: "kubernetes.azure.com/scalesetpriority" operator: "Equal" value: "spot" effect: "NoSchedule" affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: name operator: In values: - grafana-agent - key: component operator: In values: - metrics topologyKey: "kubernetes.io/hostname" containers: - args: - -config.file=/etc/agent/agent.yaml - -config.expand-env=true - -server.http.address=0.0.0.0:12345 - -server.grpc.address=0.0.0.0:12346 env: - name: HOSTNAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: AUTH_CLIENT_ID valueFrom: secretKeyRef: name: grafana-agent key: AUTH_CLIENT_ID - name: AUTH_CLIENT_SECRET valueFrom: secretKeyRef: name: grafana-agent key: AUTH_CLIENT_SECRET - name: AUTH_TOKEN_URL valueFrom: secretKeyRef: name: grafana-agent key: AUTH_TOKEN_URL - name: AUTH_SCOPE valueFrom: secretKeyRef: name: grafana-agent key: AUTH_SCOPE image: grafana/agent:v0.38.1 imagePullPolicy: IfNotPresent name: agent-metrics resources: requests: cpu: 1 memory: 5Gi limits: cpu: 3 memory: 10Gi ports: - containerPort: 12345 name: http-metrics readinessProbe: failureThreshold: 1 httpGet: port: http-metrics path: /-/ready scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 livenessProbe: failureThreshold: 1 httpGet: port: http-metrics path: /-/healthy scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 volumeMounts: - mountPath: /etc/agent name: grafana-agent - mountPath: /data name: storage serviceAccount: grafana-agent volumes: - configMap: name: grafana-agent name: grafana-agent ```
LasseHels commented 8 months ago

Any help with this issue would be greatly appreciated.

rfratto commented 7 months ago

Hi there :wave:

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)