search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.04k stars 2.35k forks source link

ClickHouse - ScopeAttributes aren't stored #27425

Open StarpTech opened 1 year ago

StarpTech commented 1 year ago

Component(s)

No response

What happened?

Description

Hi, I found out that scope attributes attached to a Meter aren't forwarded to ClickHouse. It seems, they aren't passed here https://github.com/wundergraph/opentelemetry-collector-contrib/blob/dbb685a63b0527c1255663e11daba27ca94e60dc/exporter/clickhouseexporter/internal/histogram_metrics.go#L178

Steps to Reproduce

Create a meter with the following option and measure anything.

routerMeter := h.meterProvider.Meter(cosmoRouterMeterName,
    metric.WithInstrumentationVersion("0.0.1"),
    metric.WithInstrumentationAttributes(attribute.String("router.version", "0.0.1")),
)

Expected Result

I should see attributes in the ScopeAttributes column.

Actual Result

No attributes.

Collector version

0.84.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Pop!_OS 22.04 LTS x86_64 Compiler(if manually compiled): (e.g., "go 14.2") go version go1.20.4 linux/amd64

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: '0.0.0.0:4318'

exporters:
  clickhouse:
    endpoint: ${CLICKHOUSE_ENDPOINT}
    timeout: 10s
    database: cosmo
    traces_table_name: otel_traces
    metrics_table_name: otel_metrics
    ttl_days: 30
    sending_queue:
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 10s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [clickhouse]
    traces:
      receivers: [otlp]
      exporters: [clickhouse]

Log output

No response

Additional context

No response

github-actions[bot] commented 1 year ago

Pinging code owners for exporter/clickhouse: @hanjm @dmitryax @Frapschen. See Adding Labels via Comments if you do not have permissions to add labels yourself.

crobert-1 commented 1 year ago

I don't have much context here, but it looks like the scope attributes are actually getting passed in where you've referenced. The scopeInstr variable is the instrumentation scope, which includes a getter method Attributes() (the scope instrumentation attributes). From the flow of metrics in this exporter, it looks like after all the histogram metrics are all added to a list together, they get inserted into Clickhouse using the insert method. From here we find the instrumentation scope attributes are converted into a key-value pair map, and then theoretically inserted into the DB using the SQL command. The SQL insert command being executed is found here.

I'll have to defer to code owners to confirm what I've found. They might have some more context as to why things are different than expected here.

cwegener commented 1 year ago

@StarpTech Can you make the collector config shorter, so that it is easier for others to reproduce?

StarpTech commented 1 year ago

Done

Frapschen commented 1 year ago

@StarpTech Does this issue still remain or should I close it?

StarpTech commented 1 year ago

Hi, I can still reproduce it.

github-actions[bot] commented 10 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 8 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

StarpTech commented 8 months ago

@Frapschen Can the code owners provide any feedback?

Frapschen commented 7 months ago

@StarpTech The integration test(https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/30209) has verified that the scope data has been sent to clickhouse: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/30209/files#diff-1ec84547ed93f2c8bfb21c371ca0b5304f01371e748d4b02bf397313a4b1dfa4R406-R412

StarpTech commented 7 months ago

I'll try to reproduce it again and come back.

github-actions[bot] commented 5 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

SpencerTorres commented 4 months ago

Hey @StarpTech! Were you able to reproduce this?

github-actions[bot] commented 2 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

SpencerTorres commented 2 months ago

@StarpTech let me know if this is still an issue. Thanks!

github-actions[bot] commented 2 days ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.