Alloy Test and Investigation for Logs

Rotfuks commented 2 months ago

Motivation

We want to unify all of our agents to use the new opentelemetry agent from grafanalabs: alloy. For this we need to first test out if alloy can deliver exactly the same capability as the promtail/grafana agents when collecting logs.

Todo

[x] Create a feature flag with which we can select which agents are active - for this we still need to deploy both
[x] Check if loki still gets the same logs as we expect
[x] Compare the resource consumption of alloy to the previous agents - is it less, is it more, is it the same?
[ ] Deploy alloy in one test cluster of a test installation (like gazelle) to collect logs

Outcome

We gained experience with alloy and are confident enough to roll it out as agent for collecting logs everywhere

TheoBrigitte commented 2 months ago

How to send logs to Loki using Alloy

Assuming that Loki is already installed

Install alloy

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
kubectl create namespace alloy
kubectl config set-context --current --namespace alloy
helm install --namespace alloy alloy grafana/alloy

Configure Alloy to send logs to Loki


cat <<EOF > podlogs.alloy
loki.source.podlogs "default" {
forward_to = [loki.write.remote.receiver]
namespace_selector {
match_labels = {
  "kubernetes.io/metadata.name" = "alloy",
}
}
selector {
match_labels = {
  "foo" = "bar",
}
}
}

loki.write "remote" { endpoint { url = "https://full-loki.url/loki/api/v1/push" basic_auth { username = "xxxx" password = "xxxx" } } } EOF

cat < values.yaml alloy: configMap: create: true name: alloy-config key: alloy.config EOF

kubectl create configmap alloy-config "--from-file=alloy.config=./podlogs.alloy" helm upgrade -n alloy alloy grafana/alloy -f values.yaml


3. Create a PodLogs (it should target an existing pod producing logs)

```yaml
cat <<EOF > podlogs.yaml
apiVersion: monitoring.grafana.com/v1alpha2
kind: PodLogs
metadata:
  name: some-podlog
  labels:
    foo: bar
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: prometheus-watcher
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana
EOF

Verify that logs are received in Loki

$ k port-forward svc/loki-gateway 8080:80
$ logcli query '{job="alloy/alloy-podlog"}'
2024/06/27 20:07:37 http://127.0.0.1:8080/loki/api/v1/query_range?direction=BACKWARD&end=1719511657441077151&limit=30&query=%7Bjob%3D%22alloy%2Falloy-podlog%22%7D&start=1719508057441077151
2024/06/27 20:07:37 Common labels: {instance="prometheus-watcher/prometheus-watcher-grafana-65c8bc5467-8296j:grafana", job="alloy/alloy-podlog", level="debug", service_name="alloy/alloy-podlog"}
2024-06-27T19:52:14+02:00 {} logger=context userId=1 orgId=1 uname=admin t=2024-06-27T17:52:14.52104332Z level=info msg="Request Completed" method=GET path=/api/live/ws status=400 remote_addr=185.102.95.187 time_ms=1 duration=1.290947ms size=12 referer= handler=/api/live/ws status_source=server
2024-06-27T19:52:09+02:00 {} logger=context userId=1 orgId=1 uname=admin t=2024-06-27T17:52:09.348674096Z level=info msg="Request Completed" method=GET path=/api/live/ws status=400 remote_addr=185.102.95.187 time_ms=1 duration=1.240795ms size=12 referer= handler=/api/live/ws status_source=server
... more log lines ...

TheoBrigitte commented 1 month ago

I added capability to switch between Promtail and Alloy as logging agent into the logging operator, this can be configured via the loggingAgent field in the logging operator helm chart.

Alloy is deploy with the name alloy-logs

Alloy configuration is using the same configuration as Promtail did

Alloy is configured via the logging-operator, this was implemented in https://github.com/giantswarm/logging-operator/pull/164

We need to wait for release > v29.0.0 in order to deploy this on gazelle (https://github.com/giantswarm/releases/pull/1318#pullrequestreview-2187648209)

TheoBrigitte commented 1 month ago

CPU resource usage is slightly higher: Promtail (left) ~ 0.2 and Alloy (right) ~ 0.3

Memory resource usage is slightly higher: Promtail (left) 1.3 GB and Alloy (right) 1.4 GB

Log volume stays the same with Promtail (left) and Alloy (right)

TheoBrigitte commented 1 week ago

v29.1.0 is on its way, once this is release to a CAPA installation we can proceed with our live testing of Alloy as logging agent. Everything should already be in place on gazelle as the flag toggle was already merged here https://github.com/giantswarm/giantswarm-configs/pull/120

QuentinBisson commented 2 days ago

@TheoBrigitte did you also add the grafana-agent config in the alloy-log app via the operator?

giantswarm / roadmap