Improved Prometheus Documentation

[stevehipwell]

Describe the feature request

The current Prometheus documentation is lacking in a number of areas which make it increasingly hard to set up (the documentation for Kiali and Jager isn't much better). I'll list the documentation areas in personal priority order below and hopefully as part of this someone could provide quick answers to help.

• How metric merging actually works?
  • I assume Prometheus annotations but as a Prometheus Operator user this isn't something I have ever used
  • Clarification of the fact that mTLS isn't applied to the endpoint
• How to use Prometheus Operator with istio?
  • Minimal Prometheus Operator configuration
  • Where the PodMonitor/ServiceMonitors can be found
• Is Prometheus federation still required when using metric merging?
  • If so confirmation of recording rules and aggregation to move all merged metrics

For bonus points the following would be great.

• Minimal Kiali Operator configuration (in CRD form)
• Minimal Jager Operator configuration (in CRD form)

Describe alternatives you've considered

I've read all the relevant existing documentation, the source code and searched the internet.

I was asked to create a new issue when I commented on issue https://github.com/istio/istio/issues/27940.

[x] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [x] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Additional context

[stevehipwell]

Is there anyone from the Istio team who can comment on this?

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-01-14. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

[jewertow]

How metric merging actually works?

I assume Prometheus annotations but as a Prometheus Operator user this isn't something I have ever used

Prometheus Operator also supports annotations. You can configure proper relabeling rules in Pod/Service monitor.

Clarification of the fact that mTLS isn't applied to the endpoint.

The documentation says about this fact in the TLS settings section:

The control plane, gateway, and Envoy sidecar metrics will all be scraped over plaintext.

How to use Prometheus Operator with istio?

Minimal Prometheus Operator configuration

Istio does not require any special configuration in Prometheus Operator to be monitored, so it does not make sense to copy Prometheus documentation.

Where the PodMonitor/ServiceMonitors can be found

You can find samples here. Regarding your first question, PodMonitor from this example relies on Prometheus annotations.