search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
98 stars 52 forks source link

Opentelemetry jenkins metrics doesn't write in Elastic with error `failed to parse field [system.cpu.load] of type [double]` #604

Closed isav48 closed 1 year ago

isav48 commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.375 OS: Linux - 5.15.0-48-generic Java: 17.0.6-ea - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- Parameterized-Remote-Trigger:3.1.6.3 ace-editor:1.1 active-directory:2.29 analysis-model-api:10.21.0 ansible:1.1 ansicolor:1.0.2 ant:481.v7b_09e538fcca antisamy-markup-formatter:155.v795fb_8702324 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 artifactory:3.15.3 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.287-357.vf82d85a_6eefd aws-java-sdk-cloudformation:1.12.287-357.vf82d85a_6eefd aws-java-sdk-codebuild:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ecr:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ecs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-efs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-elasticbeanstalk:1.12.287-357.vf82d85a_6eefd aws-java-sdk-iam:1.12.287-357.vf82d85a_6eefd aws-java-sdk-logs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd aws-java-sdk-sns:1.12.287-357.vf82d85a_6eefd aws-java-sdk-sqs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ssm:1.12.287-357.vf82d85a_6eefd aws-secrets-manager-credentials-provider:1.198.v839f082578db aws-secrets-manager-secret-source:1.62.v120c8e5058ee bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.1-3 bouncycastle-api:2.26 branch-api:2.1051.v9985666b_f6cc build-name-setter:2.2.0 build-timeout:1.26 build-timestamp:1.0.3 build-user-vars-plugin:1.9 buildrotator:1.2 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.8.1 cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.800.v71307ca_b_986b cobertura:1.16 code-coverage-api:3.4.1 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-27.vb_fa_3896786a_7 conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1569.vb_72405b_80249 configuration-as-code-groovy:1.1 copyartifact:1.48 credentials:1214.v1de940103927 credentials-binding:523.vd859a_4b_122e6 custom-tools-plugin:0.8 dashboard-view:2.466.vdfefd95a_b_f8d data-tables-api:1.12.1-4 devops-config-plugin:test-3216-20230314.140853-build7 display-url-api:2.3.7 docker-build-publish:1.4.0 docker-commons:1.21 docker-java-api:3.2.13-37.vf3411c9828b9 docker-plugin:1.2.10 docker-workflow:563.vd5d2e5c4007f dtkit-api:3.0.2 durable-task:503.v57154d18d478 echarts-api:5.4.0-1 email-ext:2.93 emailext-template:1.5 envinject:2.892.v25453b_80e595 envinject-api:1.199.v3ce31253ed13 extended-choice-parameter:359.v35dcfdd0c20d extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:203.v683c09d993b_9 fluentd:0.1.5 font-awesome-api:6.2.1-1 forensics-api:1.17.0 fsscm:1.0 generic-webhook-trigger:1.86.2 git:5.0.0 git-client:4.0.0 git-parameter:0.9.18 git-server:99.va_0826a_b_cdfa_d github:1.36.1 github-api:1.303-400.v35c2d8258028 github-branch-source:1696.v3a_7603564d04 gitlab-plugin:1.6.0 google-oauth-plugin:1.0.7 gradle:2.2 groovy:453.vcdb_a_c5c99890 h2-api:1.4.199 handlebars:3.0.8 hashicorp-vault-plugin:359.v2da_3b_45f17d5 hidden-parameter:0.0.5 htmlpublisher:1.31 http_request:1.16 instance-identity:142.v04572ca_5b_265 ionicons-api:31.v4757b_6987003 ivy:2.4 jackson2-api:2.14.1-313.v504cdd45c18b jacoco:3.3.2 jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-8 jaxb:2.3.7-1 jdk-tool:63.v62d2fd4b_4793 jenkins-atp-two-runner-plugin:2.2.48 jersey2-api:2.38-1 jira-steps:2.0.141.vd0c6e6dc83f0 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.16-1 job-dsl:1.81 job-parameter-summary:0.5 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1166.va_436e268e972 junit-attachments:101.v82f494a_00e9e kubernetes:3743.v1fa_4c724c3b_7 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 ldap:659.v8ca_b_a_fe79fa_d lockable-resources:1106.vc9153f1a_ff25 mailer:438.v02c7f0a_12fa_4 mapdb-api:1.0.9-28.vf251ce40855d mask-passwords:3.3 matrix-auth:3.1.6 matrix-groovy-execution-strategy:1.0.8 matrix-project:785.v06b_7f47b_c631 maven-plugin:3.20 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a momentjs:1.1.1 monitoring:1.91.0 nc-jenkins-jobs-importer-plugin:2.0.0 nc-ui-ext:0.2 nodejs:1.5.1 oauth-credentials:0.5 oic-auth:2.5 okhttp-api:4.9.3-108.v0feda04578cf opentelemetry:2.10.0 pam-auth:1.10 parameter-separator:1.3 parameterized-scheduler:1.1 parameterized-trigger:2.45 permissive-script-security:0.7 pipeline-aggregator-view:1.11 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:629.vb_5627b_ee2104 pipeline-input-step:466.v6d0a_5df34f81 pipeline-maven:1257.v89e586d3c58c pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-rest-api:2.28 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-stage-view:2.27 pipeline-utility-steps:2.14.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:2.20.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 prism-api:1.29.0-2 publish-over:0.22 python:1.3 rebuild:1.34 resource-disposer:0.20 rich-text-publisher-plugin:1.4 role-strategy:584.vf8e515397ecd run-condition:1.5 saml:4.385.v4dea_91565e9d scm-api:631.v9143df5b_e4a_a script-security:1228.vd93135a_2fb_25 simple-theme-plugin:136.v23a_15f86c53d slave-setup:1.10 snakeyaml-api:1.33-90.v80dcb_3814d35 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:305.v8f4381501156 ssh-slaves:2.854.v7fd446b_337c9 sshd:3.275.v9e17c10f2571 structs:324.va_f5d6774f3a_d subversion:2.17.0 throttle-concurrents:2.10 timestamper:1.21 token-macro:321.vd7cc1f2a_52c8 trilead-api:2.84.v72119de229b_7 uno-choice:2.6.4 update-sites-manager:2.0.0 variant:59.vf075fe829ccb vaultCredentialStore:release-1-0-1-20210225.100622-build2 versioncolumn:95.v82f3985cd6e1 warnings-ng:9.22.0 windows-slaves:1.8.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:3601.v9b_36a_d99e1cc workflow-durable-task-step:1217.v38306d8fa_b_5c workflow-job:1254.v3f64639b_11dd workflow-multibranch:716.vc692a_e52371b_ workflow-remote-loader:1.5 workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.44 xunit:3.1.2 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

  1. Send metrics to APM

Expected Results

Get metrics in elastic

Actual Results

{"log.level":"info","@timestamp":"2023-03-14T15:30:49.802Z","log.logger":"beater.grpc","log.origin":{"file.name":"interceptors/logging.go","file.line":65},"message":"request accepted","service.name":"apm-server","source.address":"10.109.116.110:31721","grpc.request.method":"/opentelemetry.proto.collector.metrics.v1.MetricsService/Export","event.duration":1344775,"grpc.response.status_code":"OK","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-14T15:30:51.469Z","log.logger":"beater","log.origin":{"file.name":"beater/checkintegration.go","file.line":105},"message":"integration package status: installed","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-14T15:30:51.497Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":68},"message":"no longer blocking ingestion as all precondition checks are now satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:30:52.847Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [type] within [host.os] is not allowed","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:30:52.847Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'WyO-4IYBK6GMsY_n5ApI'. Preview of field's value: '{average={1m=0.99}}'","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:30:52.848Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'cSO-4IYBK6GMsY_n5ApK'. Preview of field's value: '{average={1m=1.03}}'","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:30:52.854Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'HCO-4IYBK6GMsY_n4wqk'. Preview of field's value: '{average={1m=0.96}}'","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:30:52.924Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'KyO-4IYBK6GMsY_n4wqk'. Preview of field's value: '{average={1m=1}}'","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:30:52.925Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'RiO-4IYBK6GMsY_n5ApI'. Preview of field's value: '{average={1m=1.15}}'","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:30:52.928Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'MyO-4IYBK6GMsY_n5ApH'. Preview of field's value: '{average={1m=1.19}}'","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-14T15:31:42.166Z","log.logger":"beater.grpc","log.origin":{"file.name":"interceptors/logging.go","file.line":65},"message":"request accepted","service.name":"apm-server","source.address":"10.109.116.107:41842","grpc.request.method":"/opentelemetry.proto.collector.metrics.v1.MetricsService/Export","event.duration":2849932,"grpc.response.status_code":"OK","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-14T15:31:43.204Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":448},"message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'kCO_4IYBK6GMsY_nqwrU'. Preview of field's value: '{average={1m=0.98}}'","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-14T15:31:49.797Z","log.logger":"beater.grpc","log.origin":{"file.name":"interceptors/logging.go","file.line":65},"message":"request accepted","service.name":"apm-server","source.address":"10.109.116.110:31721","grpc.request.method":"/opentelemetry.proto.collector.metrics.v1.MetricsService/Export","event.duration":2566410,"grpc.response.status_code":"OK","ecs.version":"1.6.0"}

Anything else?

No response

cyrille-leclerc commented 1 year ago

Hello @isav48 . Can you confirm that the log messages you shared are produced by the component Elastic APM Server? Can you also indicate

I'm no longer working for Elastic but I'll try to help or hopefully, someone from Elastic will contribute.

isav48 commented 1 year ago

Hello @cyrille-leclerc, Yes, it's output of Elastic APM Server 8.6.2 (kibana also 8.6.2) also i've tried 7.17.9 and got similar error Metrics sends directly to Elastic APM

isav48 commented 1 year ago

example of error from Elastic APM Server 7.17.9 look like more clear maybe it help you 

{
   "log.level":"warn",
   "@timestamp":"2023-03-14T18:01:50.811Z",
   "log.logger":"elasticsearch",
   "log.origin":{
      "file.name":"elasticsearch/client.go",
      "file.line":414
   },
   "message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.March, 14, 18, 2, 29, 745746712, time.UTC), Meta:{\"pipeline\":\"apm\"}, Fields:{\"_metric_descriptions\":{\"jenkins.agents.offline\":{\"type\":\"gauge\"},\"jenkins.agents.online\":{\"type\":\"gauge\"},\"jenkins.agents.total\":{\"type\":\"gauge\"},\"jenkins.disk.usage.bytes\":{\"type\":\"gauge\"},\"jenkins.queue.blocked\":{\"type\":\"gauge\"},\"jenkins.queue.buildable\":{\"type\":\"gauge\"},\"jenkins.queue.waiting\":{\"type\":\"gauge\"},\"jenkins.scm.event.active_threads\":{\"type\":\"counter\"},\"jenkins.scm.event.completed_tasks\":{\"type\":\"counter\"},\"jenkins.scm.event.pool_size\":{\"type\":\"counter\"},\"jenkins.scm.event.queued_tasks\":{\"type\":\"counter\"},\"login\":{\"type\":\"counter\"},\"login_success\":{\"type\":\"counter\"},\"process.cpu.load\":{\"type\":\"gauge\"},\"process.cpu.time\":{\"type\":\"counter\"},\"system.cpu.load\":{\"type\":\"gauge\"},\"system.cpu.load.average.1m\":{\"type\":\"gauge\"},\"system.memory.utilization\":{\"type\":\"gauge\"},\"system.paging.utilization\":{\"type\":\"gauge\"}},\"agent\":{\"name\":\"opentelemetry/java\",\"version\":\"1.18.0\"},\"container\":{\"id\":\"03f1a901d86390e3ad059a78414d64ee83d54bbd84f4df09b785bc118e5d3c69\"},\"ecs\":{\"version\":\"1.12.0\"},\"host\":{\"architecture\":\"amd64\",\"hostname\":\"jenkins-deployer-deployment-5ff7c95f-5f26h\",\"name\":\"jenkins-deployer-deployment-5ff7c95f-5f26h\",\"os\":{\"full\":\"Linux 5.15.0-48-generic\",\"platform\":\"linux\",\"type\":\"linux\"}},\"jenkins.agents.offline\":0,\"jenkins.agents.online\":1,\"jenkins.agents.total\":1,\"jenkins.disk.usage.bytes\":7518642176,\"jenkins.queue.blocked\":0,\"jenkins.queue.buildable\":0,\"jenkins.queue.waiting\":0,\"jenkins.scm.event.active_threads\":0,\"jenkins.scm.event.completed_tasks\":0,\"jenkins.scm.event.pool_size\":0,\"jenkins.scm.event.queued_tasks\":0,\"labels\":{\"jenkins_opentelemetry_plugin_version\":\"2.10.0\",\"jenkins_url\":\"https://deployer-stg.com/\",\"jenkins_version\":\"2.362\",\"process_runtime_description\":\"Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.6-ea+9-LTS\",\"service_namespace\":\"jenkins\"},\"login\":1,\"login_success\":1,\"metricset.name\":\"app\",\"observer\":{\"ephemeral_id\":\"3ccfb6a6-bae1-482d-b74e-7766a46be8ba\",\"hostname\":\"eb1805a05313\",\"id\":\"03f4b71a-f2bc-429d-9165-e3f2a9603ffe\",\"type\":\"apm-server\",\"version\":\"7.17.9\",\"version_major\":7},\"process.cpu.load\":0.0066820276497695855,\"process.cpu.time\":1072190000000,\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"java\"},\"name\":\"staging-deployer\",\"node\":{\"name\":\"03f1a901d86390e3ad059a78414d64ee83d54bbd84f4df09b785bc118e5d3c69\"},\"runtime\":{\"name\":\"OpenJDK Runtime Environment\",\"version\":\"17.0.6-ea+9-LTS\"},\"version\":\"2.362\"},\"system.cpu.load\":0.007741768064516129,\"system.cpu.load.average.1m\":1.88,\"system.memory.utilization\":0.4214859008789062,\"system.paging.utilization\":0}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"can't merge a non object mapping [system.cpu.load] with an object mapping\"}, dropping event!",
   "service.name":"apm-server",
   "ecs.version":"1.6.0"
}
cyrille-leclerc commented 1 year ago

Can you please test https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.12.0-rc1

It uses the instrumentation of the Otel Java Auto Instrumentation library to collect JVM / runtime metrics.

isav48 commented 1 year ago

@cyrille-leclerc unfortunately, same error still exist in apm log

{
   "log.level":"error",
   "@timestamp":"2023-03-29T06:53:25.524Z",
   "log.logger":"modelindexer",
   "log.origin":{
      "file.name":"modelindexer/indexer.go",
      "file.line":448
   },
   "message":"failed to index event (mapper_parsing_exception): failed to parse field [system.cpu.load] of type [double] in document with id 'o3okLIcBkPpjlQ4vjIaM'. Preview of field's value: '{average={1m=0.15}}'",
   "service.name":"apm-server",
   "ecs.version":"1.6.0"
}

{
   "log.level":"error",
   "@timestamp":"2023-03-29T06:53:25.525Z",
   "log.logger":"modelindexer",
   "log.origin":{
      "file.name":"modelindexer/indexer.go",
      "file.line":448
   },
   "message":"failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [type] within [host.os] is not allowed",
   "service.name":"apm-server",
   "ecs.version":"1.6.0"
}
loopping commented 1 year ago

elasticsearch doesn't allow these two fields as leaves, system.cpu.load.average.1m is contained in system.cpu.load

system.cpu.load
system.cpu.load.average.1m

one solution is to drop system.cpu.load.average.1m with an ingest pipeline such as:

{
  "apm_final_pipeline": {
    "description": "apm - final pipeline",
    "processors": [
      {
        "script": {
          "source": "ctx.remove('system.cpu.load.average.1m')"
        }
      }
    ]
  }
}
cyrille-leclerc commented 1 year ago

Thanks @loopping.

I'm wondering if the system.cpu.load is still relevant. OpenTelemetry has standardized on the following that don't have have this problem of having one metric name being ta subset of another metric name:

See https://github.com/jenkinsci/opentelemetry-plugin/blob/master/docs/monitoring-metrics.md#jvm-and-system-metrics

isav48 commented 1 year ago

latest plugin (ver 2.13) is working correct with apm now (load all metrics to elastic) @cyrille-leclerc Thank you