grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.97k stars 3.46k forks source link

fix: Set a maximum retries for Docker driver to avoid deadlock. #15026

Open jeschkies opened 2 days ago

jeschkies commented 2 days ago

What this PR does / why we need it: There's a long ticket claiming that the Docker driver would deadlock when the configured Loki endpoint becomes unreachable. The root cause is that the Loki client retries forever until it can reach Loki again. This looks like a deadlock.

This issues is documented with a workaround. However, users still struggle. That's why this change proposes to make the workaround the default behavior.

Which issue(s) this PR fixes: Fixes #2361

Special notes for your reviewer:

Checklist