search

openshift / must-gather

A client tool for gathering information about an operator managed component.
Apache License 2.0
104 stars 189 forks source link

MON-3108: Add gather_metrics for Prometheus fetching #434

Closed machine424 closed 3 months ago

machine424 commented 4 months ago

This introduces a new gather_metrics script. The script leverages the new promtool tsdb dump-openmetrics command to extract metrics in OpenMetrics format.

The script accepts some arguments, run: $ ./collection-scripts/gather_metrics for more details.

The dumpoed metrics can be imported into a Prometheus instance using: $ promtool tsdb create-blocks-from openmetrics For more information, refer to https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-tsdb-create-blocks-from

Note that in some cases, using increase() or {i}rate() functions on the imported data may return negative or inaccurate results. This issue is under investigation and will be fixed upstream.

Recommendations:

the next step would be to add some out-of-the box metrics in gather_monitoring, see https://github.com/openshift/must-gather/pull/389

openshift-ci-robot commented 4 months ago

@machine424: This pull request references MON-3108 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/must-gather/pull/434): >This introduces a new `gather_metrics` script. The script leverages the new `promtool tsdb dump-openmetrics` command to extract metrics in OpenMetrics format. > >The script accepts some arguments, run: > $ ./collection-scripts/gather_metrics >for more details. > >The dumpoed metrics can be imported into a Prometheus instance using: > $ promtool tsdb create-blocks-from openmetrics >For more information, refer to https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-tsdb-create-blocks-from > >Note that in some cases, using `increase()` or `{i}rate()` functions on the imported data may return negative or inaccurate results. This issue is under investigation and will be fixed upstream. > >Recommendations: > >- Import these metrics into a blank Prometheus instance. >- Ensure that the major version of the target Prometheus instance matches the major version of the dumped instance. >- If the involved metrics are old, consider adjusting the --storage.tsdb.retention.time setting in the host instance to prevent their removal (especially if the receiving Prometheus is to configured to scrape new targets) Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fmust-gather). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci-robot commented 4 months ago

@machine424: This pull request references MON-3108 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/must-gather/pull/434): >This introduces a new `gather_metrics` script. The script leverages the new `promtool tsdb dump-openmetrics` command to extract metrics in OpenMetrics format. > >The script accepts some arguments, run: > $ ./collection-scripts/gather_metrics >for more details. > >The dumpoed metrics can be imported into a Prometheus instance using: > $ promtool tsdb create-blocks-from openmetrics >For more information, refer to https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-tsdb-create-blocks-from > >Note that in some cases, using `increase()` or `{i}rate()` functions on the imported data may return negative or inaccurate results. This issue is under investigation and will be fixed upstream. > >Recommendations: > >- Import these metrics into a blank Prometheus instance. >- Ensure that the major version of the target Prometheus instance matches the major version of the dumped instance. >- If the involved metrics are old, consider adjusting the --storage.tsdb.retention.time setting in the host instance to prevent their removal (especially if the receiving Prometheus is to configured to scrape new targets) > >--- > >the next step would be to add some out-of-the box metrics in `gather_monitoring`, see https://github.com/openshift/must-gather/pull/389 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fmust-gather). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
sferich888 commented 3 months ago

/lgtm

openshift-ci[bot] commented 3 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: machine424, sferich888

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[collection-scripts/OWNERS](https://github.com/openshift/must-gather/blob/master/collection-scripts/OWNERS)~~ [sferich888] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
machine424 commented 3 months ago

/retest

openshift-ci[bot] commented 3 months ago

@machine424: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
openshift-bot commented 3 months ago

[ART PR BUILD NOTIFIER]

Distgit: ose-must-gather This PR has been included in build ose-must-gather-container-v4.18.0-202407311714.p0.gb37c260.assembly.stream.el9. All builds following this will include this PR.