Write documentation about using the Agent as a proxy for other Agents

[ptodev]

Request

In the last few months the topic of using the Agent as a proxy came up several times. It would be great to have a page in the documentation to which we could refer people on such occasions.

We could cover topics such as:

• Pushing vs pulling from the proxy Agents
• Load balancing (e.g. sharding)
• Replication (sometimes also called "high availability"). Running more than one Agent with the same series and avoiding out of order errors. Note that Agents running with a replication mode usually have extra settings such as cluster and replica labels.
• How much load is acceptable per agent. When should the customer spread the load to more than 1 proxy Agent?
• Which components to use for receiving the data in the proxy (e.g. prometheus.receive_http)

A few thoughts, to get us started:

• If the proxy agents expect data to be pushed to them, it could work via loki.source.api and prometheus.receive_http. However, I'm not sure what would then be used for load balancing.
• If the proxy agents pull the data from the other agents, then they could use normal Prometheus hashmod sharding for load balancing, but I'm not sure what they could use for logs. I don't think Agents can pull logs from other Agents in the way they can pull metrics.
• Is clustering going to work for signals other than metrics?
• The most likely recommended route seems to be a sticky sessions load balancer which redirects remote write requests from non-Proxy Agents to Proxy Agents.
  • Can we test this and show an example load balancer configuration?
  • Which signal types would not be supported? E.g. for profile signals (Phlare) I don't think the Agent supports being pushed to.
• Can we dogfood the recommended approach internally? Is there a good use case?

Use case

There are multiple reasons why a proxy Agent would be needed. Examples include:

• Air gapped networks where the proxy Agent is in a DMZ
• Storing database credentials in only one place - with the proxy Agents

[thampiotr]

Is clustering going to work for signals other than metrics?

It won't without some work, but it seems like a good idea to me to implement it. We could allow any node in the cluster to receive a push of metrics/logs, figure out which node it should be sent to and redirect to the node owning the data.

I like this approach, because with one underlying feature (clustering) we can support multiple use cases.