Improved metric gateway pipeline setup

[a-thaler]

Description

With the recent features in the metric gateway, new receivers were added as well as the pipelines were split up into multiple ones being connected via connectors. The current solution has some flaws which needs to be addressed:

• The additional receivers like the k8sclusterreceiver requires the execution of a processor (setting the instrumentation scope). That steps should be executed only for data emmitted by this receiver, not for all data. If someone pushes data from a custom collector with the same data, the logic will be applied, which is wrong
• The pipeline setup is blown up and hard to debug. Instead we should have input pipelines per receiver (not MetricPipeline) which are the routing to exporter pipelines per MetricPipeline, having the custom filter steps included. Of course in between the enrichement should happen which again is general for all.
• The trace gateway should use the same patterns consistently. Currently the enrichment cannot be skipped!!

Proposal: Avoid duplicating definition of logic which is generic for all pipelines and restructure the pipelines in a way that we clearly map inputs to outputs. Instead of having these pipelines for two MetricPipelines called pipe1 and pipe2:

metrics/pipe1-input:
    receivers:
        - otlp
        - singleton_receiver_creator/k8s_cluster
    processors:
        - memory_limiter
    exporters:
        - routing/pipe1
metrics/pipe1-attributes-enrichment:
    receivers:
        - routing/pipe1
    processors:
        - k8sattributes
        - transform/resolve-service-name
    exporters:
        - forward/pipe1
metrics/pipe1-output:
    receivers:
        - routing/pipe1
        - forward/pipe1
    processors:
        - transform/set-instrumentation-scope-runtime
        - filter/drop-if-input-source-prometheus
        - filter/drop-if-input-source-istio
        - filter/pipe1-filter-by-namespace-runtime-input
        - filter/drop-k8s-cluster-metrics
        - resource/insert-cluster-name
        - resource/delete-skip-enrichment-attribute
        - batch
    exporters:
        - otlp/pipe1
metrics/pipe2-input:
    receivers:
        - otlp
        - singleton_receiver_creator/kymastats
        - singleton_receiver_creator/k8s_cluster
    processors:
        - memory_limiter
    exporters:
        - routing/pipe2
metrics/pipe2-attributes-enrichment:
    receivers:
        - routing/pipe2
    processors:
        - k8sattributes
        - transform/resolve-service-name
    exporters:
        - forward/pipe2
metrics/pipe2-output:
    receivers:
        - routing/pipe2
        - forward/pipe2
    processors:
        - transform/set-instrumentation-scope-runtime
        - filter/pipe2-filter-by-namespace-runtime-input
        - filter/pipe2-filter-by-namespace-prometheus-input
        - filter/drop-k8s-cluster-metrics
        - filter/drop-diagnostic-metrics-if-input-source-istio
        - transform/set-instrumentation-scope-kyma
        - resource/insert-cluster-name
        - resource/delete-skip-enrichment-attribute
        - batch
    exporters:
        - otlp/pipe2

we should have something like this:

metrics/otlp-input:
    receivers:
        - otlp
    processors:
        - memory_limiter
    exporters:
        - routing/enrichement #enrichment or (pipe1 and pipe2)
metrics/kymastats-input:
    receivers:
        - singleton_receiver_creator/kymastats
    processors:
        - memory_limiter
        - transform/set-instrumentation-scope-kyma
    exporters:
        - routing/enrichement #enrichment or (pipe1 and pipe2)
metrics/k8s_cluster-input:
    receivers:
        - singleton_receiver_creator/k8s_cluster
    processors:
        - memory_limiter
        - transform/set-instrumentation-scope-runtime
    exporters:
        - routing/enrichement #enrichment or (pipe1 and pipe2)
metrics/enrichment:
    receivers:
        - routing/enrichement
    processors:
        - k8sattributes
        - transform/resolve-service-name
        - resource/delete-skip-enrichment-attribute
    exporters:
        - forward
metrics/pipe1-output:
    receivers:
        - routing/enrichement
        - forward
    processors:
        - filter/drop-if-input-source-prometheus
        - filter/drop-if-input-source-istio
        - filter/pipe1-filter-by-namespace-runtime-input
        - filter/drop-k8s-cluster-metrics
        - resource/insert-cluster-name
        - batch
    exporters:
        - otlp/pipe1
metrics/pipe2-output:
    receivers:
        - routing/enrichement
        - forward
    processors:
        - filter/pipe2-filter-by-namespace-runtime-input
        - filter/pipe2-filter-by-namespace-prometheus-input
        - filter/drop-k8s-cluster-metrics
        - filter/drop-diagnostic-metrics-if-input-source-istio
        - resource/insert-cluster-name
        - batch
    exporters:
        - otlp/pipe2

[shorim]

After implementing the logic for skipping the enrichment for the Trace Gateway, we should document this feature here

[github-actions[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.