grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.8k stars 3.43k forks source link

Internal load balancer question raised by /fundamentals/architecture/_index.md #4696

Open sandstrom opened 2 years ago

sandstrom commented 2 years ago

The simple scalable deployment mode requires a load balancer in front of Loki, which directs /loki/api/v1/push traffic to the write nodes. All other requests go to the read nodes. Traffic should be sent in a round robin fashion.

  1. Is this really a requirement? I.e. would Promtail or the Docker driver ever send data on other endpoints? (than /loki/api/v1/push)
  2. How about letting Promtail be configured to send to multiple nodes in a round-robin fashion? I.e. if three loki instances are known, send to all three, round robin.
  3. Would be nice if DNS round-robin was supported in Promtail, i.e. if Promtail DNS lookup of the loki hostname returns multiple IP-addresses then it would round-robin among them. This will allow for easy integration with e.g. Consul (https://www.consul.io/docs/discovery/dns#standard-lookup).

What I'm getting at here is that there are several ways to avoid running an internal load balancer in front of Loki, or at least there should be. Internal load balancers are usually a needless bottleneck and point of failure. By architecting around them and leaving the "load balancing" to the client, the architecture will be more resilient.

stale[bot] commented 2 years ago

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

sandstrom commented 2 years ago

not stale

stale[bot] commented 2 years ago

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

sandstrom commented 2 years ago

not stale

stale[bot] commented 2 years ago

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

sandstrom commented 2 years ago

Still relevant

stale[bot] commented 2 years ago

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

GrafanaWriter commented 1 year ago

This is a good question, but it's incorrectly labeled as a docs issue. When the question regarding internal load balancing and architecting around it is answered, we can add to the docs as appropriate.