Check the resource usage of Alloy

[Rotfuks]

Motivation

We've seen some weird numbers when checking the golem installation on which alloy is already rolled out. It seems that alloy is using significantly more resources than promtail and prometheus-agent combined. This is not good.

Todo

• [ ] Check what the resource usage of alloy is compared to prometheus-agent and promtail on the installations
• [ ] investigate if we can improve that resource usage in any way
  • One such way can be https://github.com/giantswarm/giantswarm/issues/31559 - is this enough?

Outcome

• We are confident that Alloy is not using significantly more resources than prometheus-agent and promtail

[hervenicol]

Actual RAM usage for Alloy on golem

Queries

RAM: sum(container_memory_working_set_bytes{cluster_id="golem", namespace="kube-system", pod=~"alloy-metrics.*", container!="", image!=""}) by (pod)

Series: sum(prometheus_remote_write_wal_storage_active_series{pod=~"alloy-metrics-.*"}) by (pod)

Numbers

Currently:

• 2 alloy-metrics pods
• scraping 350k and 400k metrics
• RAM usage is 9GB and 3.5GB

=> the pod scraping 400k metrics is the one using less RAM. But it's also the youngest one.

Pprof

Extracting data:

• port-forward with ks port-forward alloy-metrics-0 12345
• get heap data with curl localhost:12345/debug/pprof/heap -o heap.pprof

Visualizing data with https://play.grafana.org/a/grafana-pyroscope-app/ad-hoc

• alloy-metrics-0:

  • alloc_objects:
  • alloc_space:
  • inuse_objects:
  • inuse_space:

• alloy-metrics-1:

  • alloc_objects:
  • alloc_space:
  • inuse_objects:
  • inuse_space: