Difficulty connecting to Clickhouse Cloud over HTTPS and/or via Chproxy

[smunukutla-mycarrier]

We are currently trying to configure otel-collector to receive traces from Kafka and export to a Clickhouse cloud instance via Chproxy. We have tried different ways of connecting but are failing to establish a connection with/without Chproxy to Clickhouse cloud.

Context on our clickhouse setup:

• We have a clickhouse cloud instance which we have been able to connect successfully over https via other tools.
• We noticed latency and packet drops so we used Chproxy with caching to alleviate the pain. We have been able to connect to Chproxy over https with other tools like Jaeger-clickhouse plugin.

Here is the qryn otel-collector configuration for reference:

receivers:
      kafka/qryn_traces: 
        topic: otlp_traces
        encoding: otlp_proto
        group_id: qryn_traces
        auth:
          sasl:
            username: ${KAFKA_USER}
            password: ${KAFKA_PASSWORD}
            mechanism: ${KAFKA_MECH}
          tls:
            ca_file: /var/tls/ca.crt
        brokers:  
          - kafka.monitoring.svc.cluster.local:9093
    processors:
      resourcedetection:
        detectors: [env, system]
      cumulativetodelta:
      batch:
        send_batch_size: 1000
        timeout: 10s
    exporters:
      qryn: 
        dsn: https://${CH_ADDR}:8443/qryntraces?skip_verify=false&secure=true&username=${CH_USER}&password=${CH_PASS}
        timeout: 10s
        sending_queue:
          queue_size: 1000
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
          max_elapsed_time: 300s
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        traces:
          receivers: [kafka/qryn_traces]
          processors: [batch]
          exporters: [qryn]

We have tried different DSNs: Scenario 1 (secure):

DSN: https://${CH_ADDR}:8443/qryntraces?skip_verify=false&secure=true&username=${CH_USER}&password=${CH_PASS}

(With skip_verify=false and skip_verify=true)

Error: 
Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "traces", "name": "qryn", "error": "[handshake] unexpected packet [72] from server", "interval": "3.264886497s"}

Scenario 2 (insecure):

DSN: http://${CH_ADDR}:8443/qryntraces?skip_verify=false&secure=false&username=${CH_USER}&password=${CH_PASS}

DSN: tcp://${CH_ADDR}:9440/qryntraces?skip_verify=false&secure=false&username=${CH_USER}&password=${CH_PASS}

Error:
exporterhelper/retry_sender.go:118  Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "traces", "name": "qryn", "error": "[read: EOF](read:%20EOF)", "errorVerbose": "read:\n    github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n        /go/pkg/mod/github.com/!click!house/ch-go@v0.58.2/proto/reader.go:62\n  - EOF", "interval": "10.972525583s"}

In the scenarios above, we have tried using both Clickhouse Cloud as well as Chproxy addresses and creds with no success. We haven't faced any issues with Kafka receiver but just qryn exporter trying to connect to clickhouse over http(s). Any help/guidance would be really appreciated. Please let me know if you need any further information.

[akvlad]

Hello. After a short test there's a working configuration file:

receivers:
  zipkin:

exporters:
  qryn:
    dsn: tcp://localhost:9000/qryn_bnch?username=default
    clustered_clickhouse: false
    timeout: 10s
    sending_queue:
      queue_size: 100
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    logs:
      format: json

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [zipkin]
      exporters: [qryn]

Traces are successfully sent. The collector is definitely working.

Are you sure 9440 is a non-secure port? it's tls endpoint by default. skip_verify=false&secure=false seems strange. Please consider rechecking.

[akvlad]

please try : tcp://${CH_ADDR}:9440/qryntraces?secure=true&username=${CH_USER}&password=${CH_PASS}

or clickhouse://${CH_USER}:${CH_PASS}@${CH_ADDR}:9440/qryntraces?secure=true

[rita7lopes]

Based on our email thread, we believe this issue was resolved with our suggestion. Therefore, I am closing it.