open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.72k stars 2.16k forks source link

allow to manage multiple opentelemetry collector with the same supervisor #33682

Open cforce opened 1 week ago

cforce commented 1 week ago

Component(s)

No response

Is your feature request related to a problem? Please describe.

The Supervisor should run only once per host group or cluster, managing all connected Collectors. This would reduce overhead for clusters and hosts with multiple VMs by centralizing management to a single Supervisor. This setup would enable one-to-many upgrades by a single Supervisor and synchronous configuration updates for all Collectors.

Note: Remote execution of Collectors on hosts different from the Supervisor is currently not supported (e.g., via SSH). Collectors need to connect to the Supervisor on a well-known host and port. This requires either pre-configured port knowledge on the Supervisor or multiplexing multiple agents on the same port.

Describe the solution you'd like

Dynamic port allocation, which already exists, should be used to serve multiple connected Collectors. This is beneficial for low-resource IoT field devices with dedicated Internet hubs/uplinks and for Kubernetes environments, reducing the need for more Supervisors than Collectors.

In addition to the current automated startup of the Collector triggered by execution on the same host, the initial startup/bootstrapping of the Collector should be handled by the Collector itself. The Collector should know which port to connect to the Supervisor, and the Supervisor should reserve such ports for the Collectors. This requires the Supervisor to listen on preconfigured ports, and the Collector should independently try to connect to this host and port as known ahead.

Alternatively, instead of using dedicated ports per Collector, allowing multiplexing of different clients on the same port could simplify bootstrapping based on a static, well-known host and port for all Collectors managed by the Supervisor. For security, each Collector should be authorized, using something like oauth2clientauthextension on the Collector side and oidcauthextension on the Supervisor side. Integration of OAuth into OpAMP client-server flows is also requested here #32762.

Describe alternatives you've considered

No response

Additional context

No response