Release-207 intergration test: Clickhouse dosen't work in instana

[yaoyao12138]

On instana env:

ts=2021-10-25T02:24:59.528792996Z level=info logger=migration msg="Clickhouse connection clickhouse://9.30.182.151:9000?database=shared&x-migrations-table=migrations_create"
ts=2021-10-25T02:24:59.548045324Z level=error logger=migration msg="clickhouse (application): 9.30.182.151, shard=1: failed to create tables. migration failed in line 0: CREATE TABLE IF NOT EXISTS shared.chains (\n  client_id String,\n  heartbeat UInt64,\n  heartbeatDate Date MATERIALIZED toDate(heartbeat/1000),\n  destination_infrastructure_id String,\n  destination_endpoint_id String,\n  destination_endpoint_label String,\n  destination_service_id String,\n  destination_service_label String,\n  destination_application_id String,\n  destination_endpoint_type String,\n  destination_technologies Array(String),\n  weight UInt64,\n  dummy String\n) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/shared/chains', '{replica}')\nPARTITION BY toRelativeWeekNum(heartbeatDate)\nORDER BY (client_id, heartbeat)\nSETTINGS index_granularity=8192;\n (details: code: 225, message: Can't create replicated table without ZooKeeper)" stacktrace="reflect.Value.call\n\t/usr/local/go/src/reflect/value.go:543\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:339\ngithub.com/chuckpreslar/emission.(*Emitter).Emit.func1\n\t/tmp/build/0c511968/repo/vendor/github.com/chuckpreslar/emission/emitter.go:209"
ts=2021-10-25T02:24:59.548156115Z level=error logger=controller-runtime.manager.controller.core msg="reconcile error" reconcilergroup=instana.io reconcilerkind=Core name=instana-core namespace=instana-core error="clickhouse (application): 9.30.182.151, shard=1: failed to create tables. migration failed in line 0: CREATE TABLE IF NOT EXISTS shared.chains (\n  client_id String,\n  heartbeat UInt64,\n  heartbeatDate Date MATERIALIZED toDate(heartbeat/1000),\n  destination_infrastructure_id String,\n  destination_endpoint_id String,\n  destination_endpoint_label String,\n  destination_service_id String,\n  destination_service_label String,\n  destination_application_id String,\n  destination_endpoint_type String,\n  destination_technologies Array(String),\n  weight UInt64,\n  dummy String\n) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/shared/chains', '{replica}')\nPARTITION BY toRelativeWeekNum(heartbeatDate)\nORDER BY (client_id, heartbeat)\nSETTINGS index_granularity=8192;\n (details: code: 225, message: Can't create replicated table without ZooKeeper)" stacktrace="github.com/instana/infrastructure/self-hosted-k8s-operator/operator/controllers.(*CoreReconciler).Reconcile\n\t/tmp/build/0c511968/repo/self-hosted-k8s-operator/operator/controllers/core_controller.go:214\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/tmp/build/0c511968/repo/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/tmp/build/0c511968/repo/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/tmp/build/0c511968/repo/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"

Also, on the db env:

root@yao-instana-vm-32-instana-db:~# systemctl status clickhouse
● clickhouse.service - ClickHouse Server (analytic DBMS for big data)
     Loaded: loaded (/lib/systemd/system/clickhouse.service; disabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-10-24 00:20:56 PDT; 19h ago
   Main PID: 2296217 (clckhouse-watch)
      Tasks: 100 (limit: 38431)
     Memory: 117.6M
     CGroup: /system.slice/clickhouse.service
             ├─2296217 clickhouse-watchdog                   --config=/etc/clickhouse/server.xml
             └─2296222 /opt/clickhouse/bin/clickhouse-server --config=/etc/clickhouse/server.xml

Oct 24 00:20:56 yao-instana-vm-32-instana-db.fyre.ibm.com systemd[1]: Started ClickHouse Server (analytic DBMS for big data).
Oct 24 00:20:56 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296217]: Processing configuration file '/etc/clickhouse/server.xml'.
Oct 24 00:20:56 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296217]: Including configuration file '/etc/clickhouse/dynamic.xml'.
Oct 24 00:20:56 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296217]: Include not found: clickhouse_compression
Oct 24 00:20:56 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296217]: Logging information to /var/log/instana/clickhouse/clickhouse-server.log
Oct 24 00:20:56 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296217]: Logging errors to /var/log/instana/clickhouse/clickhouse-server.err.log
Oct 24 00:20:57 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296222]: Include not found: clickhouse_compression
Oct 24 00:20:57 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296222]: Include not found: networks
Oct 24 00:20:57 yao-instana-vm-32-instana-db.fyre.ibm.com clickhouse-server[2296222]: Include not found: networks

root@yao-instana-vm-32-instana-db:/etc/clickhouse# vi  /var/log/instana/clickhouse/clickhouse-server.log

2021.10.24 17:44:56.215161 [ 2300188 ] {98e1caf2-1c0a-48bc-9263-e410ab765a67} <Error> TCPHandler: Code: 225, e.displayText() = DB::Exception: Can't create replicated table without ZooKeeper, Stack trace:

[yaoyao12138]

Reason:
In the config file, the element in zookeeper inclof server.xml https://github.com/instana/onprem-distkit/blob/release-207/clickhouse/1-shard-clickhouse-setup/clickhouse-1/server.xml#L162 is different from the name of dynamic.xml. https://github.com/instana/onprem-distkit/blob/release-207/clickhouse/1-shard-clickhouse-setup/clickhouse-1/dynamic.xml#L25. Fixed:
Unite this two and changed the the element in zookeeper inclof server.xml, the clickhouse works.
PR: https://github.com/instana/onprem-distkit/pull/32
cc @frankwg

[yaoyao12138]

PR merged, closed this.