search

housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
514 stars 118 forks source link

Small updates to allow TLS connections for AWS MSK, etc. #169

Closed genzgd closed 1 year ago

genzgd commented 1 year ago

This PR contains a few fixes to allow Kafka TLS connections to services like AWS MSK.

  1. Don't require TLS.caCertFiles. TLS connections will work without this if the Kafka certificate is signed by a root certificate recognized by the OS (like globally trusted roots).
  2. If secure is true, don't convert the hostname into an IPv4 address. In most cases the hostname and not the IP is required for TLS authentication.

In addition

  1. The clickhouse-go and ch-go dependencies have to updated to the latest release.
  2. The "Dial Timeout" for clickhouse-go has been made configurable, since 1 second is sometimes not enough to wake up some cloud services.
  3. The default memory used by the Franz Kafka client has been reduced to 20MB, as the previous values used would result in OOM errors even with environments with 16GB of memory.

This build has been successfully tested again AWS MSK and ClickHouse Cloud