Does it have to be a distributed table, is it not a normal MergeTree engine table?

[kkrgwbj]

when testing, Startup found the following error "error":"Please create distributed table for sso_login_log_new."

MacBook-Pro-3.local :) use test;

USE test

Query id: 3eb70227-bd0f-448c-b8f1-fd1d2d24ba1a

Ok.

0 rows in set. Elapsed: 0.001 sec.

MacBook-Pro-3.local :) show tables;

SHOW TABLES

Query id: 03fb4256-f9e9-4640-aa10-ffb9b64d40e8

┌─name──────────────┐ │ sso_login_log_new │ └───────────────────┘

1 rows in set. Elapsed: 0.002 sec.

MacBook-Pro-3.local :)

[yuzhichang]

Please show the config file of clickhouse_sinker.

[kkrgwbj]

{ "clickhouse": { "hosts": [ [ "192.168.10.68" ] ], "port": 9000, "username": "hnmcc_wx", "password": "hnmcc123",

"db": "test",

"secure": false,

"insecureSkipVerify": false,

"retryTimes": 0,

"maxOpenConns": 1

},

"kafka": { "brokers": "192.168.10.122:9092",

"security":{
    "security.protocol": "SASL_PLAINTEXT",
    "sasl.kerberos.service.name": "kafka",
    "sasl.mechanism":"GSSAPI",
    "sasl.jaas.config":"com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true debug=true keyTab=\"/etc/security/mmmtest.keytab\" principal=\"mmm@ALANWANG.COM\";"
},

"version": "2.5.0"

},

"task": { "name": "test_dynamic_schema",

"kafkaClient": "sarama",

"topic": "sso-log-topic",

"earliest": true,

"consumerGroup": "group",

"parser": "json",

"tableName": "sso_login_log_new",

"dims": [
  {

    "name": "reqTime",

    "type": "DateTime"
  },
  {
    "name": "mobile",
    "type": "String"
  }
],
"flushInterval": 5,

"bufferSize": 262114,

"timeZone": "",

"timeUnit": 1.0

},

"logLevel": "debug" }

[yuzhichang]

The error is triggered if there's cluster name in config but the distributed table doesn't exist. https://github.com/housepower/clickhouse_sinker/blob/16bcf3e88d79d9fd6a21f27fa8178aaae0396fea/output/clickhouse.go#L310

[kkrgwbj]

thanks!but ,find other error...

{"level":"debug","ts":"2022-02-11T11:48:10.013+0800","msg":"consumer/broker/0 disconnecting due to error processing FetchRequest: EOF","name":"sarama"} {"level":"debug","ts":"2022-02-11T11:48:10.013+0800","msg":"Closed connection to broker [192.168.10.122:9092]","name":"sarama"} {"level":"debug","ts":"2022-02-11T11:48:10.013+0800","msg":"kafka: error while consuming sso-log-topic/0: EOF","name":"sarama"} {"level":"debug","ts":"2022-02-11T11:48:10.419+0800","msg":"ClientID is the default of 'sarama', you should consider setting it to something application-specific.","name":"sarama"} {"level":"debug","ts":"2022-02-11T11:48:10.420+0800","msg":"Connected to broker at [192.168.10.122:9092 %!s(int32=0)] (registered as #%!d(MISSING))","name":"sarama"} {"level":"debug","ts":"2022-02-11T11:48:10.420+0800","msg":"Error while sending ApiVersionsRequest to broker 192.168.10.122:9092: kafka: insufficient data to decode packet, more bytes expected","name":"sarama"}

[yuzhichang]

https://housepower.github.io/clickhouse_sinker/configuration/config.html

    // kafka version, if you use sarama, the version must be specified
    "version": "2.5.0"

I suggest to use Franz instead:

"kafkaClient": "franz",

Franz is the best Kafka Go-SDK currently. You don't need to specify the kafka broker version.

[kkrgwbj]

thank you solved it