Show appropriate status message via self monitor if a target is close to scrape limit

[rakesh-garimella]

Description As part of https://github.com/kyma-project/telemetry-manager/issues/976 the metric-agent should expose under a dedicated port the diagnostic metrics relevant for scraping metric targets via the prometheusreceiver. The new port should be protected by a networkpolicy so that user cannot reach it. For now the plain exposure is enough in order to enable operational diagnostic.

Criterias

• There is a new port under which the diagnostic metrics of the prometheusreceiver are accessible at /metrics
• No other metrics are exposed there
• The user cannot access the port

Implementation details

Example configuration for metrics agent

pipelines:
            metrics/istio:
                ...
                exporters:
                    - otlp
                    - prometheus
            metrics/prometheus:
                ...
                exporters:
                    - otlp
                    - prometheus
            metrics/runtime:
                ...
                exporters:
                    - otlp
                    - prometheus
 exporters:
        prometheus:
            endpoint: "127.0.0.1:1888" # Prometheus  exporter config
        ...

Probably we should not expose just all metrics but also filter out business metrics. So instead we should use a new dedicated pipeline having the same receivers as input, but doing a specialized filtering and exporting

[github-actions[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed due to the lack of recent activity. /lifecycle rotten

[github-actions[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.