search

jenkinsci / prometheus-plugin

Jenkins Prometheus Plugin
https://plugins.jenkins.io/prometheus/
Apache License 2.0
184 stars 151 forks source link

Telegraf Unable to Parse Metrics - second TYPE line for metric name "jenkins_node_builds", or TYPE reported after samples #385

Closed zollo closed 2 years ago

zollo commented 2 years ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.319.3 OS: Linux - 3.10.0-1160.59.1.el7.x86_64 --- ace-editor:1.1 active-directory:2.25.1 ansicolor:1.0.0 ant:1.11 antisamy-markup-formatter:2.1 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 blueocean:1.24.8 blueocean-autofavorite:1.2.4 blueocean-bitbucket-pipeline:1.24.8 blueocean-commons:1.24.8 blueocean-config:1.24.8 blueocean-core-js:1.24.8 blueocean-dashboard:1.24.8 blueocean-display-url:2.4.1 blueocean-events:1.24.8 blueocean-git-pipeline:1.24.8 blueocean-github-pipeline:1.24.8 blueocean-i18n:1.24.8 blueocean-jira:1.24.8 blueocean-jwt:1.24.8 blueocean-personalization:1.24.8 blueocean-pipeline-api-impl:1.24.8 blueocean-pipeline-editor:1.24.8 blueocean-pipeline-scm-api:1.24.8 blueocean-rest:1.24.8 blueocean-rest-impl:1.24.8 blueocean-web:1.24.8 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.22 branch-api:2.7.0 build-blocker-plugin:1.7.7 build-name-setter:2.1.0 build-timeout:1.20 build-user-vars-plugin:1.7 build-with-parameters:1.5.1 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloudbees-bitbucket-branch-source:757.vddedc5f2589a_ cloudbees-disk-usage-simple:0.10 cloudbees-folder:6.708.ve61636eb_65a_5 command-launcher:1.6 conditional-buildstep:1.4.1 configuration-as-code:1414.v878271fc496f credentials:1087.v16065d268466 credentials-binding:1.27 display-url-api:2.3.5 docker-commons:1.19 docker-workflow:1.26 durable-task:1.37 echarts-api:5.3.0-2 email-ext:2.83 extended-choice-parameter:0.82 extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:1.7 favorite:2.3.2 font-awesome-api:6.0.0-1 git:4.10.3 git-client:3.11.0 git-server:1.9 github:1.33.1 github-api:1.123 github-branch-source:2.9.9 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 hidden-parameter:0.0.4 htmlpublisher:1.25 jackson2-api:2.13.2-260.v43d711474c77 jacoco:3.3.1 javadoc:1.6 javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jdk-tool:1.5 jenkins-design-language:1.24.8 jira:3.3 jjwt-api:0.11.2-9.c8b45b8bb173 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.54 ldap:2.0 lockable-resources:2.11 logstash:2.3.1-SNAPSHOT (private-261f22ce-root) mailer:408.vd726a_1130320 matrix-auth:2.6.8 matrix-project:1.20 maven-plugin:3.8 mercurial:2.15 metrics:4.1.6.1 momentjs:1.1.1 nodelabelparameter:1.9.0 okhttp-api:3.14.9 pam-auth:1.6 parameterized-trigger:2.39 pipeline-build-step:2.15 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:446.vf27b_0b_83500e pipeline-milestone-step:1.3.2 pipeline-model-api:1.9.1 pipeline-model-definition:1.9.1 pipeline-model-extensions:1.9.1 pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:1.9.1 pipeline-stage-view:2.19 pipeline-utility-steps:2.8.0 plain-credentials:1.7 plugin-util-api:2.15.0 popper-api:1.16.1-2 popper2-api:2.11.4-1 prometheus:2.0.10 pubsub-light:1.13 rebuild:1.32 resource-disposer:0.16 role-strategy:3.2.0 run-condition:1.5 saml:1.1.7 scm-api:595.vd5a_df5eb_0e39 script-security:1138.v8e727069a_025 simple-theme-plugin:0.7 slack:2.48 snakeyaml-api:1.29.1 sse-gateway:1.24 ssh-credentials:1.19 ssh-slaves:1.31.5 sshd:3.0.3 structs:308.v852b473a2b8c timestamper:1.13 token-macro:280.v97a_82642793c trilead-api:1.0.13 validating-string-parameter:2.8 variant:1.4 view-job-filters:2.3 wavefront:1.1.1 windows-slaves:1.8 workflow-aggregator:2.6 workflow-api:1143.v2d42f1e9dea_5 workflow-basic-steps:2.24 workflow-cps:2.94 workflow-cps-global-lib:2.19 workflow-durable-task-step:2.39 workflow-job:1145.v7f2433caa07f workflow-multibranch:2.24 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15b_c95a_ workflow-support:813.vb_d7c3d2984a_0 ws-cleanup:0.39 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller & All Agents Running CentOS 7

Reproduction steps

Expected Results

Telegraf should be able to parse the Prometheus Export.

Actual Results

2022-04-17T16:49:40Z E! [inputs.prometheus] Error in plugin: error reading metrics for https://localhost:8443/prometheus/: reading text format failed: text format parsing error in line 1803: second TYPE line for metric name "jenkins_node_builds", or TYPE reported after samples This is what we see over and over again in our Telegraf logs.

Anything else?

[[inputs.prometheus]]
  ## Prefix to attach to the measurement name
  name_prefix = "jenkins_"

  ## URL of each Jenkins server
  urls = ["https://localhost:8443/prometheus/"]

  ## Specify timeout duration for slower clients (default is 3s)
  # response_timeout = "3s"

  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

Here's what our Telegraf plugin config looks like.

Waschndolos commented 2 years ago

same as #375 I guess.

zollo commented 2 years ago

Hi all, just wanted to see if there's any other data I can provide to help track this issue down.

MarkRx commented 2 years ago

I believe the error is because agents use the jenkins_node_builds type (even though they define their type as jenkinsnode[agent]_builds and said nodes show up in the prometheus output before the type definition of jenkins_node_builds. Depending on the ordering of your nodes you may or may not encounter this issue.

Example:

# HELP jenkins_node_jknn2t_vnzdet_builds Generated from Dropwizard metric import (metric=jenkins.node.jknn2t-vnzdet.builds, type=com.codahale.metrics.Timer)
# TYPE jenkins_node_jknn2t_vnzdet_builds summary
jenkins_node_builds{node="jknn2t_vnzdet", quantile="0.5",} 0.0
jenkins_node_builds{node="jknn2t_vnzdet", quantile="0.75",} 0.0
jenkins_node_builds{node="jknn2t_vnzdet", quantile="0.95",} 0.0
jenkins_node_builds{node="jknn2t_vnzdet", quantile="0.98",} 0.0
jenkins_node_builds{node="jknn2t_vnzdet", quantile="0.99",} 0.0
jenkins_node_builds{node="jknn2t_vnzdet", quantile="0.999",} 0.0
jenkins_node_builds_count{node="jknn2t_vnzdet"} 0.0

. . .

# HELP jenkins_node_builds Generated from Dropwizard metric import (metric=jenkins.node.builds, type=com.codahale.metrics.Timer)
# TYPE jenkins_node_builds summary
jenkins_node_builds{node="master", quantile="0.5",} 0.991878353
jenkins_node_builds{node="master", quantile="0.75",} 0.991878353
jenkins_node_builds{node="master", quantile="0.95",} 0.991878353
jenkins_node_builds{node="master", quantile="0.98",} 0.991878353
jenkins_node_builds{node="master", quantile="0.99",} 0.991878353
jenkins_node_builds{node="master", quantile="0.999",} 0.991878353
jenkins_node_builds_count{node="master"} 23.0

This is not vaild. Either each individual node needs it's own type or all jenkins_node_builds need to be generated together with the TYPE definition coming first.

Note I am trying with 2.0.12-SNAPSHOT built yesteryday. https://github.com/jenkinsci/prometheus-plugin/pull/384 does not fix it.