Closed slapcat closed 2 months ago
*edit find these logs in the grafana-agent logs
Might be related https://github.com/prometheus/prometheus/issues/11589
The linked issue form @IbraAoad suggests the problem is backpressure (good summary comment here). I tried to reproduce this but was unable to get "Sample dropped" errors.
First I tried just using a simple machine charm (ubuntu
) but that didn't cause any issues for the grafana-agent machine charm.
Second I tried grafana-agent-k8s, using flog and avalanche to send large amounts of fake logs/metrics. I artificially created backpressure by continually killed prometheus and loki (while true; do kubectl delete pod prometheus-0; kubectl delete pod loki-0; sleep 1; done
), but still couldn't reproduce the "Sample dropped" errors.
Do you have a minimal reproduction environment I could use?
Also, within your environment, it would be useful to have:
curl -s <grafana-agent-unit>:12345/metrics | grep "prometheus_remote_storage_enqueue_retries_total"
curl -s <grafana-agent-unit>:12345/metrics | grep "prometheus_remote_storage_shards_max"
Which revision of the charm are you using? A related fix upstream was posted in agent 0.35.2, but that version is only in our agent charm 0.36.0 (commit), which I don't think either the machine or k8s versions of the charm use atm. Instead I see them using:
snap info grafana-agent
showing 0.35.4
as latest/stable
ubuntu/grafana-agent:0.35.2-22.04_stable
If you could also include the output of the following at normal operation and at time of failure, that could be interesting.
juju ssh grafana-agent/0 curl -s localhost:12345/metrics | grep -E "wal|agent_inflight_requests|log_messages_total|failed" | grep -v "^# "
juju ssh gragna-agent/0 df -h
Given that upstream thinks this is fixed in grafana-agent 0.36.0, probably we can fix this by bumping our grafana-agent. I don't know if there's a reason why we haven't done that in a while - I'll look into that.
Either way though, I'd really like to be able to reproduce these errors in some sort of test first so we can see bumping the version actually fixes things
I might have been wrong about where this patch landed. the fix commit is tagged >=v0.36.0, but looks like it was also cherry-picked into v0.35.2 and shows up in the v0.35.2 changelog. So probably it is fixed in v0.35.2.
Either way, the next steps are:
I'm working on (2) now
To address this, we've done the following:
Based on those fixes, we're closing this issue as we think it should be addressed. We were unsuccessful in actually reproducing the issue, however, so please reopen if you face this again!
Bug Description
System logs are flooded with the following error leading to high disk utilization:
There should be a way to trim these down or filter them.
To Reproduce
Environment
Grafana-agent is related to machine charms in an openstack cloud.
Relevant log output
Additional context
No response