open-telemetry / opentelemetry-collector

OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
4.41k stars 1.46k forks source link

multiple otlp receiver network interface binding is not working #10393

Open cforce opened 4 months ago

cforce commented 4 months ago

In https://github.com/grpc/grpc/blob/master/doc/naming.md the documentation says that you can bind several network interfaces (ipv4 and ipv6) as otlp http and grpc listener / receiver: Actually none of the shemas documented i tried and even other have worked out It seems like an array is even not supported at the endpoint key . Either its complaining [] is not allowed at all or if i not formatted as array/list its complaining about to many colons.

Below excerpt from the docs.

_The following schemes are supported by the gRPC C-core implementation, but may not be supported in other languages:

ipv4:address[:port][,address[:port],...] -- IPv4 addresses

Can specify multiple comma-delimited addresses of the form address[:port]: address is the IPv4 address to use. port is the port to use. If not specified, 443 is used. ipv6:address[:port][,address[:port],...] -- IPv6 addresses

Can specify multiple comma-delimited addresses of the form address[:port]: address is the IPv6 address to use. To use with a port the address must enclosed in literal square brackets ([ and ]). Example: ipv6:[2607:f8b0:400e:c00::ef]:443 or ipv6:[::]:1234 port is the port to use. If not specified, 443 is used._


Below some example which works (single interface binding).

receivers:
  otlp:
    protocols:
      http:
        endpoint: 10.203.220.1:4318
      grpc:
        endpoint: 10.203.220.1:4317

Below some examples that i tried which do not work but should. (multiple interface bindings)

Common schema...prefix

receivers:
  otlp:
    protocols:
      http:
        endpoint:
     grpc:
        endpoint:

combinations of endpoints values yaml string

endpoint:  "ipv4:127.0.0.1,10.203.220.1:4317"
endpoint: ipv4:[127.0.0.1,10.203.220.1][4317]
endpoint: ipv4:[127.0.0.1,10.203.220.1]4317
endpoint: ipv4:[127.0.0.1,10.203.220.1]:4317
endpoint: ipv4:127.0.0.1:4317,ipv4:10.203.220.1:4317
endpoint: 127.0.0.1,10.203.220.1[:4317]
endpoint: ipv4:127.0.0.1:4317,10.203.220.1:4317

Combinations of endpoints values yaml string

   endpoint: 
      http:
        endpoint: 10.203.220.1:4318
          - ipv4:127.0.0.1:4318
          - 10.203.220.1:4318

Same error always....

 log info verbose 1 Error: cannot start pipelines: listen tcp: address ipv4:[127.0.0.1,10.203.220.1]:4317: too many colons in address; failed to shutdown pipelines: no existing monitoring routine is running

Looking at the source code it looks like the resolvers from the grpc lib are not used at all, are they? https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/config.go#L25

The validation prevents anything then simple single interface mapping https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/config.go#L28C1-L44C2

It seems the issue is the missing support in Open telemetry collector config management, which does not make use of the underlying go grpc (and http) library which already offers multi interface binding (as well as ipv6 support)

cforce commented 4 months ago

To achieve multiple bindings, you can as workaround configure multiple receiver instances each with a single endpoint. This is not as elegant as a single configuration, but it works within the constraints of the current configuration schema. Also this might introduce additional overhead or on the opposite allow dedicated thread pools for handling ingress

Example Configuration Here’s how you can configure multiple instances of the otlp receiver to bind to different network interfaces:

receivers:
  otlp/http1:
    protocols:
      http:
        endpoint: 10.203.220.1:4318
  otlp/http2:
    protocols:
      http:
        endpoint: 127.0.0.1:4318
  otlp/grpc1:
    protocols:
      grpc:
        endpoint: 10.203.220.1:4317
  otlp/grpc2:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317

service:
  pipelines:
    metrics:
      receivers: [otlp/http1, otlp/http2, otlp/grpc1, otlp/grpc2]
      exporters: [your_exporter]