[exporter/Clickhouse]Failed to create schema in a clickhouse cluster

[Wudadada]

Component(s)

exporter/clickhouse

What happened?

Description

Failed to start the otelcol-contrib because creating database failed as the error log says.

Steps to Reproduce

my clickhouse deployed as a 3-node cluster.

Expected Result

Successfully start the otelcol-contrib

Actual Result

Failed as the log say

Collector version

v0.111.0

Environment information

Environment

OS: (e.g., "CentOS8")

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 100000

exporters:
  clickhouse:
    endpoint: tcp://10.105.212.248:9000?dial_timeout=10s
    database: otel
    async_insert: true
    ttl: 72h
    compress: lz4
    create_schema: true
    metrics_table_name: otel_metrics
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    username: "default"
    password: "123"
    cluster_name: cluster_2S_1R

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]

  telemetry:
    logs:
      level: "debug"

Log output

Oct 09 17:12:35 yptjkcshj-Linux-005 otelcol-contrib[1840871]: 2024/10/09 17:12:35 collector server run finished with error: cannot start pipelines: create database: code: 81, message: Database otel does not exist
Oct 09 17:12:35 yptjkcshj-Linux-005 systemd[1]: otelcol-contrib.service: Main process exited, code=exited, status=1/FAILURE
Oct 09 17:12:35 yptjkcshj-Linux-005 systemd[1]: otelcol-contrib.service: Failed with result 'exit-code'.
Oct 09 17:12:36 yptjkcshj-Linux-005 systemd[1]: otelcol-contrib.service: Service RestartSec=100ms expired, scheduling restart.
Oct 09 17:12:36 yptjkcshj-Linux-005 systemd[1]: otelcol-contrib.service: Scheduled restart job, restart counter is at 5.
Oct 09 17:12:36 yptjkcshj-Linux-005 systemd[1]: Stopped OpenTelemetry Collector Contrib.
Oct 09 17:12:36 yptjkcshj-Linux-005 systemd[1]: otelcol-contrib.service: Start request repeated too quickly.
Oct 09 17:12:36 yptjkcshj-Linux-005 systemd[1]: otelcol-contrib.service: Failed with result 'exit-code'.
Oct 09 17:12:36 yptjkcshj-Linux-005 systemd[1]: Failed to start OpenTelemetry Collector Contrib.

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

  See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Wudadada]

after I delete this row 'database: otel' in config.xml，otel-col can start and create schema，but the schema it created is not on cluster but standalone

[SpencerTorres]

after I delete this row 'database: otel' in config.xml，otel-col can start and create schema，but the schema it created is not on cluster but standalone

This is likely because it will fall back to the default database, which already exists by default

[SpencerTorres]

You may need to check the system.query_log table for a query that looks something like:

CREATE DATABASE IF NOT EXISTS otel cluster_2S_1R

For now, you can manually create your tables using the example DDL here, and then start your exporter with the create_schema config option set to false

[SpencerTorres]

Let me know if there are any more logs surrounding this in the otelcol output, the error message doesn't make sense. I don't see how this "create database" would error with the database already existing

[Wudadada]

Let me know if there are any more logs surrounding this in the otelcol output, the error message doesn't make sense. I don't see how this "create database" would error with the database already existing

Thank you for your reply, logs were lost and i reproduced again, but it seems nothing more useful log.