[exporter/clickhouse] create database fail if clickhouse cluster

maniac1978 commented 1 day ago

Component(s)

exporter/clickhouse

What happened?

Description

OTel collector can't start with clickhouse exporter when clickhouse cluster in config and the database doesn't exist. OTel collector starts successfully if there is a standalone clickhouse server in the configuration, the database is successfully created. If clickhouse cluster: Error: cannot start pipelines: create database: code: 81, message: Database otel does not exist collector server run finished with error: cannot start pipelines: create database: code: 81, message: Database otel does not exist

Steps to Reproduce

Describe the connection to the clickhouse cluster in the exporter.

Expected Result

Successful database creation

Actual Result

OTel collector can't start with error: Database otel does not exist

Collector version

0.114.0

Environment information

Environment

OS: Ubuntu 22.04.5 LTS Compiler(if manually compiled): go version go1.22.1 linux/amd64

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        cors:
          allowed_origins:
            - "https://*.somedomain.com:3000"
            - "https://*.somedomain.com"
        endpoint: 0.0.0.0:4318
        tls:
          cert_file: /etc/otelcol/cert.crt
          key_file: /etc/otelcol/privkey.key

processors:
  batch:

exporters:
  clickhouse:
    endpoint: tcp://192.168.201.97:9000,tcp://192.168.201.98:9000,tcp://192.168.201.99:9000
    username: user
    password: password
    database: otel
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    metrics_table_name: otel_metrics
    cluster_name: tlmtr_dev_cluster
    table_engine:
      name: ReplicatedMergeTree

service:

  pipelines:

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]

Log output

Nov 26 08:15:46 mapp-dev-tlmtr1 systemd[1]: Started OpenTelemetry Collector.
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.640Z#011info#011service@v0.114.0/service.go:166#011Setting up own telemetry...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.640Z#011info#011telemetry/metrics.go:70#011Serving metrics#011{"address": "localhost:8888", "metrics level": "Normal"}
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.659Z#011info#011service@v0.114.0/service.go:238#011Starting otelcol...#011{"Version": "", "NumCPU": 2}
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.659Z#011info#011extensions/extensions.go:39#011Starting extensions...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011error#011graph/graph.go:426#011Failed to start component#011{"error": "create database: code: 81, message: Database otel does not exist", "type": "Exporter"
, "id": "clickhouse"}
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011info#011service@v0.114.0/service.go:303#011Starting shutdown...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011info#011extensions/extensions.go:66#011Stopping extensions...
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024-11-26T08:15:46.662Z#011info#011service@v0.114.0/service.go:317#011Shutdown complete.
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: Error: cannot start pipelines: create database: code: 81, message: Database otel does not exist
Nov 26 08:15:46 mapp-dev-tlmtr1 otelcol[18967]: 2024/11/26 08:15:46 collector server run finished with error: cannot start pipelines: create database: code: 81, message: Database otel does not exist
Nov 26 08:15:46 mapp-dev-tlmtr1 systemd[1]: otelcol.service: Main process exited, code=exited, status=1/FAILURE
Nov 26 08:15:46 mapp-dev-tlmtr1 systemd[1]: otelcol.service: Failed with result 'exit-code'.
Nov 26 08:15:51 mapp-dev-tlmtr1 systemd[1]: Stopped OpenTelemetry Collector.

Additional context

I tried to connect clickhouse clusters with versions: ClickHouse server version 23.8.9.54 (official build) with Zookeeper. ClickHouse server version 24.8.7.41 (official build) with ClickHouseKeeper.

github-actions[bot] commented 1 day ago

Pinging code owners:

exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

maniac1978 commented 1 day ago

It looks like I was in a hurry to claim that the database is being created with a standalone clickhouse server. With version 0.114.0, the database is not created at all, even with a standalone clickhouse server.

The database was successfully created with version otelcol 0.95. But I had to go to the latest version, because 0.95 when connecting to the clickhouse cluster, gave an error "* '' has invalid keys: table_engine".

And I did not check the creation of the database on version 0.114.0 with standalone clickhouse. Now I tried to connect it to a standalone clickhouse server and got the same error: collector server run finished with error: cannot start pipelines: create database: code: 81, message: Database otel does not exist.

So it was in this issue: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/23664

maniac1978 commented 1 day ago

I ran the otelcol version 0.95 with the same configuration and the database was created on standalone clickhouse server.

SpencerTorres commented 16 hours ago

Hey! Thanks for submitting this. Could be a duplicate of #35713, check that thread and see if you can provide similar debugging info and I can look into it

open-telemetry / opentelemetry-collector-contrib