search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.12k stars 2.39k forks source link

Simplify Collector Configuration for Users #36116

Open mlunadia opened 3 weeks ago

mlunadia commented 3 weeks ago

Component(s)

No response

Is your feature request related to a problem? Please describe.

Problem Description

The current process of configuring the collector presents a significant barrier to entry for users, especially those who are not familiar with the intricacies of collector components or the external technologies involved. Setting up a proper configuration often requires: Detailed knowledge of multiple collector components and their configuration. A sophisticated understanding of how external technologies translate into collector components. Considerable effort to perform necessary data manipulations which grow in complexity as a single configuration needs to cater for monitoring several technologies eg. applications, databases, infrastructure.

These complexities create friction for users, particularly novices, who may lack the technical background or time to effectively create and maintain configurations. This ultimately hinders the adoption and ease of use of the collector, forcing users to spend an unnecessary amount of time on tasks that should be more straightforward.

Who Is Affected?

Novice Users: Users who are new to the OTel ecosystem or less experienced in technical configurations are most affected. They may find it daunting to understand the specifics of collector components and how they work together. Even if they do understand the collector, now they need the expertise on how data from each external technology should be handled.

Expert Users: While expert users can handle these configurations, they may still face inefficiencies when helping less experienced team members or when managing large-scale deployments where a single configuration needs to cover a variety of technologies.

Impact on User Experience

Time and Effort: Novice users spend excessive time understanding and implementing complex configurations that could be streamlined. Learning Curve: The steep learning curve discourages adoption and creates dependency on expert users for tasks that could be automated or simplified. Potential Errors: Due to the manual and intricate nature of the current configuration process, there is a higher chance of configuration errors, which can lead to system misbehavior or downtime.

Why Is This Important?

By making the configuration process more intuitive, we can:

The end goal is to reduce the complexity of the configuration process, enabling novice users to easily set up and manage collector configurations with minimal input. The configuration should require only basic parameters, such as authentication details and endpoints, while more complex logic is handled behind the scenes.

Example of Current Complexity

A typical configuration for scraping and normalizing metrics from Couchbase might look like the following:

receivers:
  prometheus/couchbase:
    config:
      scrape_configs:
        - job_name: 'couchbase'
          scrape_interval: 5s
          static_configs:
            - targets: ['couchbase:8091']
          basic_auth:
            username: 'otelu'
            password: 'otelpassword'
          metric_relabel_configs:
            # Include only a few key metrics
            - source_labels: [ __name__ ]
              regex: "(kv_ops)|\
                (kv_vb_curr_items)|\
                (kv_num_vbuckets)|\
                (kv_ep_cursor_memory_freed_bytes)|\
                (kv_total_memory_used_bytes)|\
                (kv_ep_num_value_ejects)|\
                (kv_ep_mem_high_wat)|\
                (kv_ep_mem_low_wat)|\
                (kv_ep_tmp_oom_errors)|\
                (kv_ep_oom_errors)"
              action: keep
# ... complex processors and exporters follow

In this example, novice users must navigate complex receiver and processor configurations. A more user-friendly experience might allow a simplified configuration like this:

receivers:
  module/couchbase_metrics:
    endpoint: couchbase:8091
    username: otelu
    password: otelpassword

exporters:
  prometheus:
    endpoint: 0.0.0.0:9123

Describe the solution you'd like

Request for Implementation

Proposed Outcome

Expert users can build and distribute shareable modules of collector configuration that accept a minimal number of parameters. These modules should be natively recognized by the collector, and resolved to “effective” configuration. Novice users can consume these modules to greatly simplify their configuration experience by leveraging industry expertise and best practices. A user would provide values for the minimal set of parameters defined by each module, then the collector would evaluate the values and modules to generate a configuration with best practices out of the box.

This simplification would greatly improve the user experience by:

Describe alternatives you've considered

This upcoming PR solves some of the single technology to be monitored abstraction for receivers and could be a good pattern to follow. However, it only works for Kubernetes and it doesn't solve processors configuration.

Additional context

No response

mlunadia commented 3 weeks ago

@danielgblanco a few months back during the OTel meetup in London I mentioned the need for simplifying configuration for end users, would be great to have your input on this one

VihasMakwana commented 3 weeks ago

Related to https://github.com/open-telemetry/opentelemetry-collector/issues/8372. But AFAIK, the progress is stalled.

Amoolaa commented 3 weeks ago

+1 for that template feature. The other thing I could see being useful here is a configuration wizard