[Bug]: why jaeger don't connect clickhouse databases;

[ericinfra]

What happened?

I do it in the simplest way, ,https://github.com/jaegertracing/jaeger-clickhouse/blob/main/guide-kubernetes.md

kubectl get cm jaeger-clickhouse -o yaml
apiVersion: v1
data:
  config.yaml: |
    address: clickhouse-jaeger:9000
    username: clickhouse_operator
    password: clickhouse_operator_password
    spans_table:
    spans_index_table:
    operations_table:
kind: ConfigMap

But the report cannot connect to the clickhouse

The error log is as follows:

[ERROR] jaeger-clickhouse: Failed to create a storage: @module=jaeger-clickhouse EXTRA_VALUE_AT_END="could not connect to database: \"dial tcp: missing address\"" timestamp=2023-02-09T09:02:29.083Z

kubectl logs jaeger-clickhouse-854dfc4c5d-fkcq9

Defaulted container "jaeger" out of: jaeger, install-plugin (init)
2023/02/09 09:02:29 maxprocs: Leaving GOMAXPROCS=36: CPU quota undefined
{"level":"info","ts":1675933349.0723028,"caller":"flags/service.go:119","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1675933349.0723774,"caller":"flags/service.go:125","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
{"level":"info","ts":1675933349.07262,"caller":"flags/admin.go:129","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1675933349.0726702,"caller":"flags/admin.go:143","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1675933349.0726848,"caller":"flags/admin.go:121","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
2023-02-09T09:02:29.075Z [WARN]  plugin configured with a nil SecureConfig
2023-02-09T09:02:29.075Z [DEBUG] starting plugin: path=/plugin/jaeger-clickhouse args=["/plugin/jaeger-clickhouse", "--config", "/plugin-config/config.yaml"]
2023-02-09T09:02:29.076Z [DEBUG] plugin started: path=/plugin/jaeger-clickhouse pid=23
2023-02-09T09:02:29.076Z [DEBUG] waiting for RPC address: path=/plugin/jaeger-clickhouse
2023-02-09T09:02:29.083Z [ERROR] jaeger-clickhouse: Failed to create a storage: @module=jaeger-clickhouse EXTRA_VALUE_AT_END="could not connect to database: \"dial tcp: missing address\"" timestamp=2023-02-09T09:02:29.083Z
{"level":"fatal","ts":1675933349.0847218,"caller":"./main.go:109","msg":"Failed to init storage factory","error":"grpc-plugin builder failed to create a store: error attempting to connect to plugin rpc client: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.","stacktrace":"main.main.func1\n\t./main.go:109\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t./main.go:240\nruntime.main\n\truntime/proc.go:250"}
2023-02-09T09:02:29.084Z [ERROR] plugin process exited: path=/plugin/jaeger-clickhouse pid=23 error="exit status 1"

clickhouse and database is normal

# kubectl get pod
NAME                                             READY   STATUS             RESTARTS       AGE
busybox                                          1/1     Running            25 (30m ago)   25h
chi-jaeger-cluster1-0-0-0                        1/1     Running            0              18m
jaeger-clickhouse-854dfc4c5d-fkcq9               0/1     CrashLoopBackOff   6 (5m6s ago)   10m

# kubectl exec -it statefulset.apps/chi-jaeger-cluster1-0-0 -- clickhouse-client  -h clickhouse-jaeger --user clickhouse_operator --password clickhouse_operator_password
ClickHouse client version 22.1.3.7 (official build).
Connecting to clickhouse-jaeger:9000 as user clickhouse_operator.
Connected to ClickHouse server version 22.1.3 revision 54455.

:) use jaeger

USE jaeger

Query id: c0bb766d-3fbe-4e4f-9fe8-68ac5a0b2345

Ok.

0 rows in set. Elapsed: 0.001 sec. 

 :) show databases;

SHOW DATABASES

Query id: 19942759-efd9-4d13-adf7-26acd678425b

┌─name───────────────┐
│ INFORMATION_SCHEMA │
│ default            │
│ information_schema │
│ jaeger             │
│ system             │
└────────────────────┘

5 rows in set. Elapsed: 0.002 sec. 

SELECT
    query_id,
    client_hostname,
    initial_address
FROM system.processes

Query id: 13ca8ec2-1449-45ca-a44a-89e37fa070b7

┌─query_id─────────────────────────────┬─client_hostname────────────────────┬─initial_address────┐
│ 13ca8ec2-1449-45ca-a44a-89e37fa070b7 │ clickhouse-client-5574484945-b7zx9 │ ::ffff:10.100.3.42 │
└──────────────────────────────────────┴────────────────────────────────────┴────────────────────┘

1 rows in set. Elapsed: 0.002 sec. 

Steps to reproduce

I do it in the simplest way, https://github.com/jaegertracing/jaeger-clickhouse/blob/main/guide-kubernetes.md

Is this document missing the necessary steps, but I also tried to create the table manually

my jaeger-operator version is 1.41.0

Expected behavior

Is it an image version problem?

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

No response

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

No response

[pradipsakh]

@ericinfra , I am facing the same problem. Did you found solution for this ?

[ericinfra]

@ericinfra , I am facing the same problem. Did you found solution for this ?

@pradipsakh

Sorry, I went back to my hometown last week This problem is related to two configurations

1.Note: must first create the following database and sql in clickhouse

CREATE DATABASE IF NOT EXISTS jaeger ON CLUSTER '$CLUSTER_NAME' ENGINE=Atomic;

2.The config.yaml of jaeger needs to clearly write spans_table, and it cannot be defaulted. My yaml file is as follows::

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: jaeger-clickhouse
  namespace: observability
  labels:
    jaeger-clickhouse: demo
data:
  config.yaml: |
    address: clickhouse.test-clickhouse:9000
    username: jaeger
    password: jaeger123
    database: jaeger
    replication: true
    spans_table: jaeger_spans
    spans_index_table: jaeger_index
    operations_table: jaeger_operations
    ttl: 15
EOF

cat <<EOF | kubectl create -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-clickhouse
  labels:
    jaeger-clickhouse: demo
spec:
  storage:
    type: grpc-plugin
    grpcPlugin:
      image: ghcr.io/jaegertracing/jaeger-clickhouse:0.13.0
    options:
      grpc-storage-plugin:
        binary: /plugin/jaeger-clickhouse
        configuration-file: /plugin-config/config.yaml
        log-level: debug
  volumeMounts:
    - name: plugin-config
      mountPath: /plugin-config
  volumes:
    - name: plugin-config
      configMap:
        name: jaeger-clickhouse
EOF

3.You need to ensure that all the nodes of the clickhouse are normal. Maybe the jaeger link to the clickhouse needs to be improved, so you need to ensure that each node of the clickhouse is normal. I configured 16G memory for each node of the clickhouse and it is normal.

Wish you all the best

[zhixiangjoy]

I meet the same problem, I solved by add "//" before ck address, like this: address: //clickhouse.test-clickhouse:9000.