Load Balancing on client side (Splunk Otel Collector)

[naveen-goswami]

As per the documentation; only single Splunk Platform endpoint is possible to configure. This delegates the responsibility to load balance the telemetry ingestion through third party tools such as Nginx. It will be an extra component to manage in the stack.

splunkPlatform:
  # Required for Splunk Enterprise/Cloud. URL to a Splunk instance to send data
  # to. e.g. "http://X.X.X.X:8088/services/collector". Setting this parameter
  # enables Splunk Platform as a destination.
  endpoint: ""
  # Required for Splunk Enterprise/Cloud (if `endpoint` is specified). Splunk
  # HTTP Event Collector token.
  token: ""

Would it be possible to handle load balancing between different endpoints on the client side itself ? Moreover, it'd be great to have some insights into how Otel collector would handle the DNS load balancing.

[matthewmodestino]

This would be an enhancement and is not possible today.

In general, it is more robust and reliable to use purpose built load balancing technologies that are commonly already found in k8s, or in production networks, like ingesses or true Load Balancers, vs bespoke agent implementations.

Technically if it is easier in your situation, you could leverage a splunk intermediate forwarder and use it's common s2s features, which it sounds like you are describing.

ie. a HF per cluster that receives the single hec input then loadbalances via s2s...tho even at scale a LB in front of HEC ingestors on HF calls for proper load balancers.