search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.02k stars 2.33k forks source link

Metric queueSize twice fails with opentelemetry collector with a prometheus metrics exporter #18194

Closed cmunger closed 1 year ago

cmunger commented 1 year ago

Component(s)

exporter/prometheus

What happened?

Describe the bug

When setting up an opentelemetry collector with a metrics prometheus exporter, the exporter will fail when called if the monitored application is a java application using a opentelemetry-log4j-appender-2.17 instrumentation since the application has 2 metrics named queueSize one for the logs and of for the spans processor

Steps to reproduce A java application using opentelementry java agent log4j2 instrumentation :

io.opentelemetry.instrumentation opentelemetry-log4j-appender-2.17 1.21.0-alpha runtime

Java Application must do in loop log statements every seconds:

public static void main(String[] args) throws InterruptedException {
    while (true) {
        LoggerFactory.getLogger("root").info("TestLog");
        LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(1));
    }
}

start the collector : docker run -p 4317:4317 -p 9464:9464 -v $(pwd)/otel-collector.yaml:/etc/otelcol/config.yaml otel/opentelemetry-collector start the java application call http://localhost:9464/metrics will trigger the bug in the opentelemetry collector and will produce an half populated prometheus output file

What did you expect to see? A prometheus output file with output :

# TYPE queueSize gauge
# HELP queueSize The number of spans queued,The number of logs queued
queueSize{spanProcessorType="BatchSpanProcessor"} 0
queueSize{logRecordProcessorType="BatchLogRecordProcessor"} 0

What did you see instead? A badly generated prometheus output file with missing queueSize{spanProcessorType="BatchSpanProcessor"} 0

# HELP default_queueSize The number of logs queued
# TYPE default_queueSize gauge
default_queueSize{logRecordProcessorType="BatchLogRecordProcessor"} 0

What version and what artifacts are you using? oentelemetry-collector 0.68, 0.69.0.70 opentelemetry-java-agent 1.21.0

Collector version

0.68,0.69,0.70

Environment information

OS: Ubuntu 20.04

OpenTelemetry Collector configuration

receivers:
  otlp/2:
    protocols:
      grpc:

processors:
  batch:

exporters:
  otlphttp:
    endpoint: http://localhost:8080
  prometheus:
    endpoint: "0.0.0.0:9464"
    namespace: "default"

service:
  pipelines:
    traces:
      receivers: [ otlp/2 ]
      processors: [ batch ]
      exporters: [ otlphttp ]
    metrics:
      receivers: [ otlp/2 ]
      processors: [ batch ]
      exporters: [ prometheus ]
    logs:
      receivers: [ otlp/2 ]
      processors: [ batch ]
      exporters: [ otlphttp ]

Log output

2023-02-01T11:19:41.663Z    info    service/telemetry.go:90 Setting up own telemetry...
2023-02-01T11:19:41.663Z    info    service/telemetry.go:116    Serving Prometheus metrics  {"address": ":8888", "level": "Basic"}
2023-02-01T11:19:41.665Z    info    exporter/exporter.go:290    Development component. May change in the future.    {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:41.665Z    warn    loggingexporter@v0.70.0/factory.go:109  'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior.  {"kind": "exporter", "data_type": "metrics", "name": "logging", "loglevel": "debug", "equivalent verbosity level": "Detailed"}
2023-02-01T11:19:41.666Z    info    service/service.go:128  Starting otelcol... {"Version": "0.70.0", "NumCPU": 8}
2023-02-01T11:19:41.666Z    info    extensions/extensions.go:41 Starting extensions...
2023-02-01T11:19:41.666Z    info    extensions/extensions.go:44 Extension is starting...    {"kind": "extension", "name": "pprof"}
2023-02-01T11:19:41.666Z    info    pprofextension@v0.70.0/pprofextension.go:71 Starting net/http/pprof server  {"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":"localhost:1777"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2023-02-01T11:19:41.666Z    info    extensions/extensions.go:48 Extension started.  {"kind": "extension", "name": "pprof"}
2023-02-01T11:19:41.666Z    info    extensions/extensions.go:44 Extension is starting...    {"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z    info    zpagesextension@v0.70.0/zpagesextension.go:64   Registered zPages span processor on tracer provider {"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z    info    zpagesextension@v0.70.0/zpagesextension.go:74   Registered Host's zPages    {"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z    info    zpagesextension@v0.70.0/zpagesextension.go:86   Starting zPages extension   {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}}
2023-02-01T11:19:41.666Z    info    extensions/extensions.go:48 Extension started.  {"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z    info    extensions/extensions.go:44 Extension is starting...    {"kind": "extension", "name": "health_check"}
2023-02-01T11:19:41.666Z    info    healthcheckextension@v0.70.0/healthcheckextension.go:45 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2023-02-01T11:19:41.666Z    warn    internal/warning.go:51  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-02-01T11:19:41.667Z    info    extensions/extensions.go:48 Extension started.  {"kind": "extension", "name": "health_check"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:86 Starting exporters...
2023-02-01T11:19:41.667Z    info    service/pipelines.go:90 Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "otlphttp"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:94 Exporter started.   {"kind": "exporter", "data_type": "logs", "name": "otlphttp"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:90 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "otlphttp"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:94 Exporter started.   {"kind": "exporter", "data_type": "traces", "name": "otlphttp"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:90 Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-02-01T11:19:41.667Z    warn    internal/warning.go:51  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:94 Exporter started.   {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:90 Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:94 Exporter started.   {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:98 Starting processors...
2023-02-01T11:19:41.667Z    info    service/pipelines.go:102    Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "logs"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:106    Processor started.  {"kind": "processor", "name": "batch", "pipeline": "logs"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:102    Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "traces"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:106    Processor started.  {"kind": "processor", "name": "batch", "pipeline": "traces"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:102    Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:106    Processor started.  {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:110    Starting receivers...
2023-02-01T11:19:41.667Z    info    service/pipelines.go:114    Receiver is starting... {"kind": "receiver", "name": "otlp/2", "pipeline": "traces"}
2023-02-01T11:19:41.667Z    warn    internal/warning.go:51  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "receiver", "name": "otlp/2", "pipeline": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-02-01T11:19:41.667Z    info    otlpreceiver@v0.70.0/otlp.go:94 Starting GRPC server    {"kind": "receiver", "name": "otlp/2", "pipeline": "traces", "endpoint": "0.0.0.0:4317"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:118    Receiver started.   {"kind": "receiver", "name": "otlp/2", "pipeline": "traces"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:114    Receiver is starting... {"kind": "receiver", "name": "otlp/2", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:118    Receiver started.   {"kind": "receiver", "name": "otlp/2", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:114    Receiver is starting... {"kind": "receiver", "name": "otlp/2", "pipeline": "logs"}
2023-02-01T11:19:41.667Z    info    service/pipelines.go:118    Receiver started.   {"kind": "receiver", "name": "otlp/2", "pipeline": "logs"}
2023-02-01T11:19:41.667Z    info    healthcheck/handler.go:129  Health Check state change   {"kind": "extension", "name": "health_check", "status": "ready"}
2023-02-01T11:19:41.667Z    info    service/service.go:145  Everything is ready. Begin running and processing data.
2023-02-01T11:19:54.693Z    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 2}
2023-02-01T11:19:54.693Z    info    ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> container.id: Str(2164232)
     -> deployment.environment: Str(pre)
     -> host.arch: Str(amd64)
     -> host.name: Str(cmunger-HP-Compaq-Elite-8300-CMT)
     -> os.description: Str(Linux 5.4.0-137-generic)
     -> os.type: Str(linux)
     -> process.command_line: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java -javaagent:./opentelemetry-javaagent.jar -Dotel.javaagent.configuration-file=./opentelemetry-one.properties -javaagent:/home/cmunger/dev/idea-IC-221.5080.210/lib/idea_rt.jar=39569:/home/cmunger/dev/idea-IC-221.5080.210/bin -Dfile.encoding=UTF-8)
     -> process.executable.path: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java)
     -> process.pid: Int(2186394)
     -> process.runtime.description: Str(Azul Systems, Inc. OpenJDK 64-Bit Server VM 11.0.17+8-LTS)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(11.0.17+8-LTS)
     -> service.name: Str(monitoring-poc-one)
     -> service.namespace: Str(poc)
     -> service.version: Str(1.1)
     -> telemetry.auto.version: Str(1.21.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.21.0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0
    {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:55.495Z    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 17}
2023-02-01T11:19:55.495Z    info    ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> container.id: Str(2164232)
     -> deployment.environment: Str(pre)
     -> host.arch: Str(amd64)
     -> host.name: Str(cmunger-HP-Compaq-Elite-8300-CMT)
     -> os.description: Str(Linux 5.4.0-137-generic)
     -> os.type: Str(linux)
     -> process.command_line: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java -javaagent:./opentelemetry-javaagent.jar -Dotel.javaagent.configuration-file=./opentelemetry-one.properties -javaagent:/home/cmunger/dev/idea-IC-221.5080.210/lib/idea_rt.jar=39569:/home/cmunger/dev/idea-IC-221.5080.210/bin -Dfile.encoding=UTF-8)
     -> process.executable.path: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java)
     -> process.pid: Int(2186394)
     -> process.runtime.description: Str(Azul Systems, Inc. OpenJDK 64-Bit Server VM 11.0.17+8-LTS)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(11.0.17+8-LTS)
     -> service.name: Str(monitoring-poc-one)
     -> service.namespace: Str(poc)
     -> service.version: Str(1.1)
     -> telemetry.auto.version: Str(1.21.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.21.0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.runtime-metrics 1.21.0-alpha
Metric #0
Descriptor:
     -> Name: process.runtime.jvm.buffer.count
     -> Description: The number of buffers in the pool
     -> Unit: {buffers}
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1
Metric #1
Descriptor:
     -> Name: process.runtime.jvm.classes.unloaded
     -> Description: Number of classes unloaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
Metric #2
Descriptor:
     -> Name: process.runtime.jvm.classes.current_loaded
     -> Description: Number of classes currently loaded
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5426
Metric #3
Descriptor:
     -> Name: process.runtime.jvm.buffer.usage
     -> Description: Memory that the Java virtual machine is using for this buffer pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8192
Metric #4
Descriptor:
     -> Name: process.runtime.jvm.threads.count
     -> Description: Number of executing threads
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> daemon: Bool(false)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1
NumberDataPoints #1
Data point attributes:
     -> daemon: Bool(true)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 16
Metric #5
Descriptor:
     -> Name: process.runtime.jvm.memory.usage_after_last_gc
     -> Description: Measure of memory used after the most recent garbage collection event on this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 10485760
Metric #6
Descriptor:
     -> Name: process.runtime.jvm.memory.limit
     -> Description: Measure of max obtainable memory
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1073741824
NumberDataPoints #1
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 122908672
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8405385216
NumberDataPoints #3
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 122912768
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5836800
Metric #7
Descriptor:
     -> Name: process.runtime.jvm.memory.usage
     -> Description: Measure of memory used
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 34584624
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 3931568
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8812928
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5678240
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2079616
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 71303168
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1271808
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 10485760
Metric #8
Descriptor:
     -> Name: process.runtime.jvm.memory.init
     -> Description: Measure of initial memory requested
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 499122176
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 27262976
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
Metric #9
Descriptor:
     -> Name: process.runtime.jvm.classes.loaded
     -> Description: Number of classes loaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5426
Metric #10
Descriptor:
     -> Name: process.runtime.jvm.memory.committed
     -> Description: Measure of memory committed
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 35700736
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 4325376
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8847360
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 398458880
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 117440512
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 10485760
Metric #11
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.utilization
     -> Description: Recent cpu utilization for the whole system
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0.000000
Metric #12
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.load_1m
     -> Description: Average CPU load of the whole system for the last minute
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0.890000
Metric #13
Descriptor:
     -> Name: process.runtime.jvm.cpu.utilization
     -> Description: Recent cpu utilization for the process
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0.000000
Metric #14
Descriptor:
     -> Name: process.runtime.jvm.buffer.limit
     -> Description: Total capacity of the buffers in this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8192
ScopeMetrics #2
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
    {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:56.003Z    error   prometheusexporter@v0.70.0/log.go:34    error gathering metrics: collected metric default_queueSize label:<name:"job" value:"poc/monitoring-poc-one" > label:<name:"logRecordProcessorType" value:"BatchLogRecordProcessor" > gauge:<value:0 >  has help "The number of logs queued" but should have "The number of spans queued"
    {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
    github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.70.0/log.go:34
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
    github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/http.go:139
net/http.HandlerFunc.ServeHTTP
    net/http/server.go:2109
net/http.(*ServeMux).ServeHTTP
    net/http/server.go:2487
go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
    go.opentelemetry.io/collector@v0.70.0/config/confighttp/compression.go:162
net/http.HandlerFunc.ServeHTTP
    net/http/server.go:2109
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.37.0/handler.go:210
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
    go.opentelemetry.io/collector@v0.70.0/config/confighttp/clientinfohandler.go:39
net/http.serverHandler.ServeHTTP
    net/http/server.go:2947
net/http.(*conn).serve
    net/http/server.go:1991
2023-02-01T11:19:56.297Z    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 18}
2023-02-01T11:19:56.298Z    info    ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> container.id: Str(2164232)
     -> deployment.environment: Str(pre)
     -> host.arch: Str(amd64)
     -> host.name: Str(cmunger-HP-Compaq-Elite-8300-CMT)
     -> os.description: Str(Linux 5.4.0-137-generic)
     -> os.type: Str(linux)
     -> process.command_line: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java -javaagent:./opentelemetry-javaagent.jar -Dotel.javaagent.configuration-file=./opentelemetry-one.properties -javaagent:/home/cmunger/dev/idea-IC-221.5080.210/lib/idea_rt.jar=39569:/home/cmunger/dev/idea-IC-221.5080.210/bin -Dfile.encoding=UTF-8)
     -> process.executable.path: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java)
     -> process.pid: Int(2186394)
     -> process.runtime.description: Str(Azul Systems, Inc. OpenJDK 64-Bit Server VM 11.0.17+8-LTS)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(11.0.17+8-LTS)
     -> service.name: Str(monitoring-poc-one)
     -> service.namespace: Str(poc)
     -> service.version: Str(1.1)
     -> telemetry.auto.version: Str(1.21.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.21.0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.runtime-metrics 1.21.0-alpha
Metric #0
Descriptor:
     -> Name: process.runtime.jvm.buffer.count
     -> Description: The number of buffers in the pool
     -> Unit: {buffers}
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #1
Descriptor:
     -> Name: process.runtime.jvm.classes.unloaded
     -> Description: Number of classes unloaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #2
Descriptor:
     -> Name: process.runtime.jvm.classes.current_loaded
     -> Description: Number of classes currently loaded
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 6654
Metric #3
Descriptor:
     -> Name: process.runtime.jvm.buffer.usage
     -> Description: Memory that the Java virtual machine is using for this buffer pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #4
Descriptor:
     -> Name: process.runtime.jvm.threads.count
     -> Description: Number of executing threads
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> daemon: Bool(false)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2
NumberDataPoints #1
Data point attributes:
     -> daemon: Bool(true)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 15
Metric #5
Descriptor:
     -> Name: process.runtime.jvm.memory.usage_after_last_gc
     -> Description: Measure of memory used after the most recent garbage collection event on this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10485760
Metric #6
Descriptor:
     -> Name: process.runtime.jvm.memory.limit
     -> Description: Measure of max obtainable memory
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 1073741824
NumberDataPoints #1
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 122908672
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 8405385216
NumberDataPoints #3
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 122912768
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 5836800
Metric #7
Descriptor:
     -> Name: process.runtime.jvm.memory.usage
     -> Description: Measure of memory used
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 41074464
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 4784424
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10261248
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 4569216
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2776448
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 56623104
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 1320704
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10485760
Metric #8
Descriptor:
     -> Name: process.runtime.jvm.memory.init
     -> Description: Measure of initial memory requested
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 499122176
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 27262976
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #9
Descriptor:
     -> Name: process.runtime.jvm.classes.loaded
     -> Description: Number of classes loaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 6654
Metric #10
Descriptor:
     -> Name: process.runtime.jvm.memory.committed
     -> Description: Measure of memory committed
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 42287104
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 5242880
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10289152
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 232783872
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2818048
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 390070272
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10485760
Metric #11
Descriptor:
     -> Name: process.runtime.jvm.gc.duration
     -> Description: Duration of JVM garbage collection actions
     -> Unit: ms
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> action: Str(end of minor GC)
     -> gc: Str(G1 Young Generation)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Count: 1
Sum: 15.000000
Min: 15.000000
Max: 15.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
Metric #12
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.utilization
     -> Description: Recent cpu utilization for the whole system
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0.613251
Metric #13
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.load_1m
     -> Description: Average CPU load of the whole system for the last minute
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0.890000
Metric #14
Descriptor:
     -> Name: process.runtime.jvm.cpu.utilization
     -> Description: Recent cpu utilization for the process
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0.533951
Metric #15
Descriptor:
     -> Name: process.runtime.jvm.buffer.limit
     -> Description: Total capacity of the buffers in this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
ScopeMetrics #2
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 1
    {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:58.003Z    error   prometheusexporter@v0.70.0/log.go:34    error gathering metrics: collected metric default_queueSize label:<name:"job" value:"poc/monitoring-poc-one" > label:<name:"spanProcessorType" value:"BatchSpanProcessor" > gauge:<value:0 >  has help "The number of spans queued" but should have "The number of logs queued"
    {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
    github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.70.0/log.go:34
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
    github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/http.go:139
net/http.HandlerFunc.ServeHTTP
    net/http/server.go:2109
net/http.(*ServeMux).ServeHTTP
    net/http/server.go:2487
go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
    go.opentelemetry.io/collector@v0.70.0/config/confighttp/compression.go:162
net/http.HandlerFunc.ServeHTTP
    net/http/server.go:2109
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.37.0/handler.go:210
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
    go.opentelemetry.io/collector@v0.70.0/config/confighttp/clientinfohandler.go:39
net/http.serverHandler.ServeHTTP
    net/http/server.go:2947
net/http.(*conn).serve
    net/http/server.go:1991
^C2023-02-01T11:20:08.494Z  info    otelcol/collector.go:234    Received signal from OS {"signal": "interrupt"}
2023-02-01T11:20:08.494Z    info    service/service.go:154  Starting shutdown...
2023-02-01T11:20:08.494Z    info    healthcheck/handler.go:129  Health Check state change   {"kind": "extension", "name": "health_check", "status": "unavailable"}
2023-02-01T11:20:08.494Z    info    service/pipelines.go:130    Stopping receivers...
2023-02-01T11:20:08.494Z    info    service/pipelines.go:137    Stopping processors...
2023-02-01T11:20:08.494Z    info    service/pipelines.go:144    Stopping exporters...
2023-02-01T11:20:08.494Z    info    extensions/extensions.go:55 Stopping extensions...
2023-02-01T11:20:08.494Z    info    zpagesextension@v0.70.0/zpagesextension.go:109  Unregistered zPages span processor on tracer provider   {"kind": "extension", "name": "zpages"}
2023-02-01T11:20:08.494Z    info    service/service.go:168  Shutdown complete.

Additional context

This issue is similar to this one : https://github.com/open-telemetry/opentelemetry-java/issues/4382

github-actions[bot] commented 1 year ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Aneurysm9 commented 1 year ago

Can you please add the logging exporter to your metrics pipeline at the detailed verbosity? Understanding the metrics being presented to the prometheus exporter here will be important.

cmunger commented 1 year ago

updated with the full log output where you can see the issue with the 2 queueSize gauges, sorry the log is pretty big..

Aneurysm9 commented 1 year ago

2023-02-01T11:19:58.003Z error prometheusexporter@v0.70.0/log.go:34 error gathering metrics: collected metric default_queueSize label:<name:"job" value:"poc/monitoring-poc-one" > label:<name:"spanProcessorType" value:"BatchSpanProcessor" > gauge: has help "The number of spans queued" but should have "The number of logs queued"

Looks like metrics with the same name from different instrumentation scopes with different descriptions not being handled well. The metrics given to the exporter:

ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0
cmunger commented 1 year ago

Yep exactly the problem, this issue have been also encountered in the java instrumentation project. This is kind of a show stopper as it will happen in any basic java project with logging instrumentation and tracing making the export of metrics using prometheus and the collector complety unusable.

mviitane commented 1 year ago

I'm having the same problem, and as a workaround, decided to drop the queueSize metrics (for both spans and logs) using the filter processor. I'm sure there are more elegant alternatives but this gets rid of the problem.

  filter:
    metrics:
      exclude:
        match_type: strict
        metric_names:
          - queueSize
cmunger commented 1 year ago

thanks for the filter tip, working nicely ! hope a correct solution will be provided by the team.

github-actions[bot] commented 1 year ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 1 year ago

This issue has been closed as inactive because it has been stale for 120 days with no activity.