operator pod restart causes clickhouse statefulsets to be recreated (without configuration change)

[FarhadF]

Is this expected that restart of clickhouse operator pod causes all the nodes to be recreated? in case this is not a bug, what configuration change should I do to prevent this? Please advise

steps to reproduce:

1. Make a clickhouse sharded + replicated cluster with some tables via operator version 0.16

   kubectl get pods --namespace clickhouse
   NAME                              READY   STATUS    RESTARTS   AGE
   chi-clickhouse-sandbox-ch-0-0-0   2/2     Running   0          91m
   chi-clickhouse-sandbox-ch-1-0-0   2/2     Running   0          88m
   chi-clickhouse-sandbox-ch-2-0-0   2/2     Running   0          85m
   chi-clickhouse-sandbox-ch-3-0-0   2/2     Running   0          81m
   chi-clickhouse-sandbox-ch-4-0-0   2/2     Running   0          78m
   chi-clickhouse-sandbox-ch-5-0-0   2/2     Running   0          75m

2. when everything is stable remove clickhouse-operator pod (simulating a restart in case of instance failure)

   kubectl delete pod/clickhouse-operator-55dcf7f88-g44jw --namespace kube-system

3. check the clickhouse pods status after a new operator pod is running (operator will start replacing nodes, note the terminating status on node 0-0-0)

   kubectl get pods --namespace clickhouse
   NAME                              READY   STATUS        RESTARTS   AGE
   chi-clickhouse-sandbox-ch-0-0-0   2/2     Terminating   0          94m
   chi-clickhouse-sandbox-ch-1-0-0   2/2     Running       0          91m
   chi-clickhouse-sandbox-ch-2-0-0   2/2     Running       0          88m
   chi-clickhouse-sandbox-ch-3-0-0   2/2     Running       0          85m
   chi-clickhouse-sandbox-ch-4-0-0   2/2     Running       0          82m
   chi-clickhouse-sandbox-ch-5-0-0   2/2     Running       0          79m

Operator logs after restart:

I1026 16:24:43.957435       1 announcer.go:226] clickhouse_operator.go:87:Run():start
I1026 16:24:43.957507       1 announcer.go:224] clickhouse_operator.go:95:Run():Starting clickhouse-operator. Version:0.16.0 GitSHA:4767ddc6 BuiltAt:2021-10-03T18:11:19
I1026 16:24:43.958397       1 announcer.go:218] Parameters num: 11
I1026 16:24:43.958408       1 announcer.go:218] OPERATOR_CONTAINER_CPU_LIMIT=8
I1026 16:24:43.958411       1 announcer.go:218] OPERATOR_CONTAINER_CPU_REQUEST=0
I1026 16:24:43.958413       1 announcer.go:218] OPERATOR_CONTAINER_MEM_LIMIT=20411842560
I1026 16:24:43.958417       1 announcer.go:218] OPERATOR_CONTAINER_MEM_REQUEST=0
I1026 16:24:43.958421       1 announcer.go:218] OPERATOR_POD_IP=10.20.101.148
I1026 16:24:43.958425       1 announcer.go:218] OPERATOR_POD_NAME=clickhouse-operator-55dcf7f88-fppmn
I1026 16:24:43.958428       1 announcer.go:218] OPERATOR_POD_NAMESPACE=kube-system
I1026 16:24:43.958431       1 announcer.go:218] OPERATOR_POD_NODE_NAME=ip-xxx
I1026 16:24:43.958433       1 announcer.go:218] OPERATOR_POD_SERVICE_ACCOUNT=clickhouse-operator
I1026 16:24:43.958436       1 announcer.go:218] WATCH_NAMESPACE=
I1026 16:24:43.958438       1 announcer.go:218] WATCH_NAMESPACES=
I1026 16:24:43.961168       1 announcer.go:220] File-based ClickHouseOperatorConfigurations
I1026 16:24:43.961227       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: 
WatchNamespaces (0):
CHCommonConfigsPath: config.d
CHHostConfigsPath: conf.d
CHUsersConfigsPath: users.d
CHCommonConfigs (0):
CHHostConfigs (0):
CHUsersConfigs (0):
CHITemplatesPath: templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1026 16:24:43.999005       1 announcer.go:220] Unified (but not post-processed yet) CHOP config
I1026 16:24:43.999052       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: 
WatchNamespaces (0):
CHCommonConfigsPath: config.d
CHHostConfigsPath: conf.d
CHUsersConfigsPath: users.d
CHCommonConfigs (0):
CHHostConfigs (0):
CHUsersConfigs (0):
CHITemplatesPath: templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1026 16:24:43.999233       1 announcer.go:220] Final CHOP config
I1026 16:24:43.999286       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: kube-system
WatchNamespaces (0):
CHCommonConfigsPath: /etc/clickhouse-operator/config.d
CHHostConfigsPath: /etc/clickhouse-operator/conf.d
CHUsersConfigsPath: /etc/clickhouse-operator/users.d
CHCommonConfigs (4):
  - [01-clickhouse-01-listen.xml]=<yandex>
    <!-- Listen wildcard address to allow accepting connections from other containers and host network. -->
    <listen_host>::</listen_host>
    <listen_host>0.0.0.0</listen_host>
    <listen_try>1</listen_try>
</yandex>

  - [01-clickhouse-02-logger.xml]=<yandex>
    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>1</console>
    </logger>
</yandex>

  - [01-clickhouse-03-query_log.xml]=<yandex>
    <query_log replace="1">
        <database>system</database>
        <table>query_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>
    <query_thread_log remove="1"/>
</yandex>

  - [01-clickhouse-04-part_log.xml]=<yandex>
    <part_log replace="1">
        <database>system</database>
        <table>part_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </part_log>
</yandex>

CHHostConfigs (0):
CHUsersConfigs (3):
  - [01-clickhouse-user.xml]=<yandex>
    <users>
        <clickhouse_operator>
            <networks>
                <ip>127.0.0.1</ip>
                <ip>0.0.0.0/0</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448</password_sha256_hex>
            <profile>clickhouse_operator</profile>
            <quota>default</quota>
        </clickhouse_operator>
    </users>
    <profiles>
        <clickhouse_operator>
            <log_queries>0</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
        </clickhouse_operator>
    </profiles>
</yandex>

  - [02-clickhouse-default-profile.xml]=<yandex>
  <profiles>
    <default>
      <log_queries>1</log_queries>
      <connect_timeout_with_failover_ms>1000</connect_timeout_with_failover_ms>
      <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>
      <parallel_view_processing>1</parallel_view_processing>
    </default>
  </profiles>
</yandex>

  - [03-database-ordinary.xml]=<!--  Remove it for ClickHouse versions before 20.4 -->
<yandex>
    <profiles>
        <default>
            <default_database_engine>Ordinary</default_database_engine>
        </default>
    </profiles>
</yandex>

CHITemplatesPath: /etc/clickhouse-operator/templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1026 16:24:43.999303       1 announcer.go:218] Log option 'logtostderr' change value from 'true' to 'true'
I1026 16:24:43.999315       1 announcer.go:218] Log option 'alsologtostderr' change value from 'false' to 'false'
I1026 16:24:43.999318       1 announcer.go:218] Log option 'v' change value from '1' to '1'
I1026 16:24:43.999331       1 announcer.go:220] Additional log options applied
I1026 16:24:43.999345       1 announcer.go:220] clickhouse_operator.go:102:Run():Log options parsed
I1026 16:24:43.999372       1 announcer.go:226] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: kube-system
WatchNamespaces (0):
CHCommonConfigsPath: /etc/clickhouse-operator/config.d
CHHostConfigsPath: /etc/clickhouse-operator/conf.d
CHUsersConfigsPath: /etc/clickhouse-operator/users.d
CHCommonConfigs (4):
  - [01-clickhouse-01-listen.xml]=<yandex>
    <!-- Listen wildcard address to allow accepting connections from other containers and host network. -->
    <listen_host>::</listen_host>
    <listen_host>0.0.0.0</listen_host>
    <listen_try>1</listen_try>
</yandex>

  - [01-clickhouse-02-logger.xml]=<yandex>
    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>1</console>
    </logger>
</yandex>

  - [01-clickhouse-03-query_log.xml]=<yandex>
    <query_log replace="1">
        <database>system</database>
        <table>query_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>
    <query_thread_log remove="1"/>
</yandex>

  - [01-clickhouse-04-part_log.xml]=<yandex>
    <part_log replace="1">
        <database>system</database>
        <table>part_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </part_log>
</yandex>

CHHostConfigs (0):
CHUsersConfigs (3):
  - [01-clickhouse-user.xml]=<yandex>
    <users>
        <clickhouse_operator>
            <networks>
                <ip>127.0.0.1</ip>
                <ip>0.0.0.0/0</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448</password_sha256_hex>
            <profile>clickhouse_operator</profile>
            <quota>default</quota>
        </clickhouse_operator>
    </users>
    <profiles>
        <clickhouse_operator>
            <log_queries>0</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
        </clickhouse_operator>
    </profiles>
</yandex>

  - [02-clickhouse-default-profile.xml]=<yandex>
  <profiles>
    <default>
      <log_queries>1</log_queries>
      <connect_timeout_with_failover_ms>1000</connect_timeout_with_failover_ms>
      <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>
      <parallel_view_processing>1</parallel_view_processing>
    </default>
  </profiles>
</yandex>

  - [03-database-ordinary.xml]=<!--  Remove it for ClickHouse versions before 20.4 -->
<yandex>
    <profiles>
        <default>
            <default_database_engine>Ordinary</default_database_engine>
        </default>
    </profiles>
</yandex>

CHITemplatesPath: /etc/clickhouse-operator/templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1026 16:24:43.999774       1 announcer.go:220] clickhouse_operator.go:145:Run():Starting CHI controller
I1026 16:24:43.999842       1 announcer.go:220] Starting ClickHouseInstallation controller
I1026 16:24:43.999860       1 announcer.go:218] waitForCacheSync():Syncing caches for ClickHouseInstallation controller
I1026 16:24:44.041384       1 announcer.go:218] ENQUEUE new ReconcileCHI cmd=add for clickhouse/clickhouse
I1026 16:24:44.099978       1 announcer.go:218] waitForCacheSync():Caches are synced for ClickHouseInstallation controller
I1026 16:24:44.194395       1 announcer.go:218] controller.go:431:Run():ClickHouseInstallation controller: starting workers number: 11
I1026 16:24:44.194417       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 1 out of 11
I1026 16:24:44.194447       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 2 out of 11
I1026 16:24:44.194466       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 3 out of 11
I1026 16:24:44.194477       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 4 out of 11
I1026 16:24:44.194489       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 5 out of 11
I1026 16:24:44.194500       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 6 out of 11
I1026 16:24:44.194508       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 7 out of 11
I1026 16:24:44.194518       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 8 out of 11
I1026 16:24:44.194530       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 9 out of 11
I1026 16:24:44.194537       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 10 out of 11
I1026 16:24:44.194549       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 11 out of 11
I1026 16:24:44.194570       1 announcer.go:220] controller.go:439:Run():ClickHouseInstallation controller: workers started
W1026 16:24:54.194857       1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template1 referenced in useTemplates. Skip it.
W1026 16:24:54.194881       1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template2 referenced in useTemplates. Skip it.
I1026 16:24:54.254636       1 announcer.go:220] markReconcileStart():clickhouse/clickhouse:reconcile started
I1026 16:24:54.311355       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:0|rep:0|host:0-0
I1026 16:24:54.311379       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:1|rep:0|host:1-0
I1026 16:24:54.311386       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:2|rep:0|host:2-0
I1026 16:24:54.311394       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:3|rep:0|host:3-0
I1026 16:24:54.311405       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:4|rep:0|host:4-0
I1026 16:24:54.311412       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:5|rep:0|host:5-0
W1026 16:24:54.324867       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1026 16:24:54.325227       1 announcer.go:245] unable to create PDB poddisruptionbudgets.policy "clickhouse" already exists
I1026 16:24:54.325288       1 announcer.go:218] CreateServiceCHI():clickhouse/clickhouse:clickhouse/service-clickhouse
I1026 16:24:54.325572       1 announcer.go:224] updateService():clickhouse/clickhouse:reuse Port 8123 values
I1026 16:24:54.325587       1 announcer.go:224] updateService():clickhouse/clickhouse:reuse Port 9000 values
I1026 16:24:54.334938       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/service-clickhouse
I1026 16:24:54.406379       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1026 16:24:54.491873       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-usersd
I1026 16:24:54.540246       1 announcer.go:218] CreateServiceCluster():clickhouse/clickhouse:clickhouse/cluster-clickhouse-sandbox-ch
I1026 16:24:54.540296       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-0
I1026 16:24:54.540323       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 0-0 started
I1026 16:24:54.651800       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-0-0 use custom template clickhouse-v20.6
I1026 16:24:54.651909       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-0-0
W1026 16:24:54.651929       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:24:54.651939       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:24:54.651947       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:24:54.651953       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:24:54.653749       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-0-0
I1026 16:24:54.653775       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 0 cluster sandbox-ch
I1026 16:24:54.713910       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local]
I1026 16:24:55.240695       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/0-0-OK
I1026 16:24:55.248699       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-0-0
I1026 16:24:55.341134       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-0-0 use custom template clickhouse-v20.6
I1026 16:24:55.341234       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-0-0
W1026 16:24:55.341258       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:24:55.341272       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:24:55.341283       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:24:55.341296       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:24:55.343085       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-0-0
I1026 16:24:55.343141       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - started
I1026 16:24:55.419557       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.921502699s 78.497301ms/1m30s
I1026 16:26:25.372986       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 11=>12
I1026 16:26:30.395914       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:12 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1026 16:26:35.423351       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:12 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1026 16:26:35.468028       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - completed
I1026 16:26:35.518667       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-0-0 use custom template clickhouse-v20.6
I1026 16:26:35.518752       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-0-0
W1026 16:26:35.518770       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:26:35.518781       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:26:35.518790       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:26:35.518799       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:26:35.520743       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-0-0
I1026 16:26:35.520803       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - started
I1026 16:26:35.577133       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.236060397s/1m30s
I1026 16:26:35.591682       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 12=>13
I1026 16:27:36.175435       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:13 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1026 16:27:56.359493       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:13 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1026 16:27:56.408410       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - completed
I1026 16:27:56.547274       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-0-0 for Set chi-clickhouse-sandbox-ch-0-0
I1026 16:27:56.795998       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-0-0
I1026 16:27:56.980647       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:0/0 cluster:sandbox-ch
I1026 16:27:57.183474       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 0-0
I1026 16:27:57.183527       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1026 16:27:57.183651       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1026 16:27:57.183706       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1026 16:27:57.183746       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1026 16:27:57.183873       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.191013       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1026 16:27:57.191034       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.191043       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.213577       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1026 16:27:57.213600       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.213609       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.222120       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1026 16:27:57.222141       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.222149       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.230912       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1026 16:27:57.230931       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.230940       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.260802       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1026 16:27:57.260828       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.260840       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.289686       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1026 16:27:57.289706       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
E1026 16:27:57.289721       1 announcer.go:267] cluster.go:95:QueryAny():FAILED to run query on all hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1026 16:27:57.289745       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1026 16:27:57.289765       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1026 16:27:57.289785       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1026 16:27:57.289802       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1026 16:27:57.289925       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.292047       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1026 16:27:57.292065       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.292074       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.294353       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1026 16:27:57.294372       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.294380       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.297286       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1026 16:27:57.297305       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.297313       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.299782       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1026 16:27:57.299806       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.299817       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.301573       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1026 16:27:57.301589       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1026 16:27:57.301600       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1026 16:27:57.303411       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1026 16:27:57.303427       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
E1026 16:27:57.303443       1 announcer.go:267] cluster.go:95:QueryAny():FAILED to run query on all hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1026 16:27:57.303476       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1026 16:27:57.303496       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1026 16:27:57.303514       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 0 cluster sandbox-ch
I1026 16:27:57.349226       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1026 16:27:57.961478       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 0-0 completed
I1026 16:27:57.978432       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.137  0xc00090d510 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:74cad381-a9ac-4f74-92e5-210c4abbc769,APIVersion:,ResourceVersion:3860773,FieldPath:,}} {10.20.101.30  0xc00090d520 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:134e1ce5-e4f4-4a42-ae16-d192efce06b8,APIVersion:,ResourceVersion:3862550,FieldPath:,}} {10.20.2.168  0xc00090d530 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:92be44d2-1376-45a5-9e87-530932c34192,APIVersion:,ResourceVersion:3861607,FieldPath:,}} {10.20.2.217  0xc00090d540 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:d0a81e14-138c-4c2e-9ae0-0f4e03ced434,APIVersion:,ResourceVersion:3860044,FieldPath:,}} {10.20.51.244  0xc00090d550 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:28bd33c6-c6af-4106-b4a9-48e9343ca279,APIVersion:,ResourceVersion:3863358,FieldPath:,}} {10.20.51.50  0xc00090d560 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:5f909f77-91fc-4f0e-88f0-3ed009b64214,APIVersion:,ResourceVersion:3880410,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
W1026 16:27:57.988996       1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template1 referenced in useTemplates. Skip it.
W1026 16:27:57.989039       1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template2 referenced in useTemplates. Skip it.
I1026 16:27:58.188651       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-1
I1026 16:27:58.188689       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 1-0 started
I1026 16:27:58.388253       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-1-0 use custom template clickhouse-v20.6
I1026 16:27:58.388347       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-1-0
W1026 16:27:58.388366       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:27:58.388384       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:27:58.388395       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:27:58.388403       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:27:58.390673       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-1-0
I1026 16:27:58.390703       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 1 cluster sandbox-ch
I1026 16:27:58.773909       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local]
I1026 16:27:58.776027       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/1-0-OK
I1026 16:27:58.948811       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-1-0
I1026 16:27:59.223872       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-1-0 use custom template clickhouse-v20.6
I1026 16:27:59.223962       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-1-0
W1026 16:27:59.223977       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:27:59.223984       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:27:59.223993       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:27:59.224004       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:27:59.227810       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-1-0
I1026 16:27:59.227870       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - started
I1026 16:27:59.418436       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.805344579s 194.655421ms/1m30s
I1026 16:29:29.251176       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 14=>15
I1026 16:29:34.277091       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:15 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1026 16:29:39.295850       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:15 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1026 16:29:39.379523       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - completed
I1026 16:29:39.453423       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-1-0 use custom template clickhouse-v20.6
I1026 16:29:39.453519       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-1-0
W1026 16:29:39.453543       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:29:39.453556       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:29:39.453566       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:29:39.453577       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:29:39.455338       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-1-0
I1026 16:29:39.455389       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - started
I1026 16:29:39.523631       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.299849364s/1m30s
I1026 16:29:39.539679       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 15=>16
W1026 16:29:43.261016       1 reflector.go:436] pkg/client/informers/externalversions/factory.go:117: watch of *v1.ClickHouseOperatorConfiguration ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind \"ClickHouseOperatorConfiguration\" is registered for version \"clickhouse.altinity.com/v1\" in scheme \"pkg/client/clientset/versioned/scheme/register.go:30\"") has prevented the request from succeeding
I1026 16:30:35.211811       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:16 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1026 16:31:00.520551       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:16 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1026 16:31:00.576730       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - completed
I1026 16:31:00.703689       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-1-0 for Set chi-clickhouse-sandbox-ch-1-0
I1026 16:31:00.905969       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-1-0
I1026 16:31:01.146821       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:1/0 cluster:sandbox-ch
I1026 16:31:01.343279       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 1-0
I1026 16:31:01.343336       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1026 16:31:01.343444       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1026 16:31:01.343478       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1026 16:31:01.343495       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1026 16:31:01.343582       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1026 16:31:01.380476       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1026 16:31:01.380527       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names1: default
I1026 16:31:01.380550       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1026 16:31:01.380565       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql1: CREATE DATABASE IF NOT EXISTS "default"
I1026 16:31:01.380584       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1026 16:31:01.380603       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1026 16:31:01.380731       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1026 16:31:01.406765       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1026 16:31:01.406797       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations_local
I1026 16:31:01.406812       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days_local
I1026 16:31:01.406833       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations_local
I1026 16:31:01.406851       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user_local
I1026 16:31:01.406866       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1_local
I1026 16:31:01.406888       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations
I1026 16:31:01.406913       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days
I1026 16:31:01.406936       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations
I1026 16:31:01.406950       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user
I1026 16:31:01.406963       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1
I1026 16:31:01.406977       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1026 16:31:01.406992       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192
I1026 16:31:01.407011       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192
I1026 16:31:01.407026       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192
I1026 16:31:01.407046       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1026 16:31:01.407061       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1026 16:31:01.407089       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand())
I1026 16:31:01.407114       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand())
I1026 16:31:01.407138       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand())
I1026 16:31:01.407167       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand())
I1026 16:31:01.407195       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())
I1026 16:31:01.407225       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Creating distributed objects at 1-0: [default default.campaign_in_app_event_aggregations_local default.campaign_in_app_purchase_event_days_local default.campaign_postback_aggregations_local default.user_event_aggregations_user_local default.user_event_aggregations_user1_local default.campaign_in_app_event_aggregations default.campaign_in_app_purchase_event_days default.campaign_postback_aggregations default.user_event_aggregations_user default.user_event_aggregations_user1]
I1026 16:31:01.407251       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:
[CREATE DATABASE IF NOT EXISTS "default" CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())]
I1026 16:31:01.428012       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 1 cluster sandbox-ch
I1026 16:31:01.519987       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1026 16:31:02.151860       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 1-0 completed
I1026 16:31:02.178906       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.137  0xc00044c710 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:74cad381-a9ac-4f74-92e5-210c4abbc769,APIVersion:,ResourceVersion:3860773,FieldPath:,}} {10.20.101.30  0xc00044c740 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:134e1ce5-e4f4-4a42-ae16-d192efce06b8,APIVersion:,ResourceVersion:3862550,FieldPath:,}} {10.20.2.168  0xc00044c750 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:92be44d2-1376-45a5-9e87-530932c34192,APIVersion:,ResourceVersion:3861607,FieldPath:,}} {10.20.2.53  0xc00044c760 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:f1a152f1-0514-4734-a8ad-b6002ee54872,APIVersion:,ResourceVersion:3881154,FieldPath:,}} {10.20.51.244  0xc00044c770 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:28bd33c6-c6af-4106-b4a9-48e9343ca279,APIVersion:,ResourceVersion:3863358,FieldPath:,}} {10.20.51.50  0xc00044c790 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:5f909f77-91fc-4f0e-88f0-3ed009b64214,APIVersion:,ResourceVersion:3880410,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
W1026 16:31:02.212756       1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template1 referenced in useTemplates. Skip it.
W1026 16:31:02.212782       1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template2 referenced in useTemplates. Skip it.
I1026 16:31:02.432303       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-2
I1026 16:31:02.432345       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 2-0 started
I1026 16:31:02.565879       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-2-0 use custom template clickhouse-v20.6
I1026 16:31:02.566050       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-2-0
W1026 16:31:02.566075       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:31:02.566093       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:31:02.566107       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:31:02.566119       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:31:02.568594       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-2-0
I1026 16:31:02.568619       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 2 cluster sandbox-ch
I1026 16:31:02.931747       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local]
I1026 16:31:02.939053       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/2-0-OK
I1026 16:31:03.119770       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-2-0
I1026 16:31:03.403927       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-2-0 use custom template clickhouse-v20.6
I1026 16:31:03.404023       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-2-0
W1026 16:31:03.404041       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:31:03.404052       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:31:03.404063       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:31:03.404076       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:31:03.405857       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-2-0
I1026 16:31:03.405925       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-2-0) - started
I1026 16:31:03.585763       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.818095041s 181.904959ms/1m30s
I1026 16:32:33.436611       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 16=>17
I1026 16:32:38.452692       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-2-0:OK  :ObservedGeneration:17 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749 UpdateRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749
I1026 16:32:43.481389       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-2-0:OK  :ObservedGeneration:17 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749 UpdateRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749
I1026 16:32:43.532903       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-2-0) - completed
I1026 16:32:43.614608       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-2-0 use custom template clickhouse-v20.6
I1026 16:32:43.614713       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-2-0
W1026 16:32:43.614731       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1026 16:32:43.614746       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1026 16:32:43.614759       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1026 16:32:43.614768       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1026 16:32:43.616463       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-2-0
I1026 16:32:43.616546       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-2-0) - started
I1026 16:32:43.689606       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.28574793s/1m30s
I1026 16:32:43.705791       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 17=>18

[alex-zaitsev]

@FarhadF , It is complaining of missing templates. It could be a reason of changing CHI and restarting:

W1026 16:24:54.194857 1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template1 referenced in useTemplates. Skip it. W1026 16:24:54.194881 1 announcer.go:245] normalizer.go:104:CreateTemplatedCHI():clickhouse/clickhouse:UNABLE to find template clickhouse/template2 referenced in useTemplates. Skip it.

Could you also show your CHI, please?

[FarhadF]

@alex-zaitsev sure

tl;dr

could it be caused by this? I dont see any other references to it. My CHI is a modified version of example 99-max

  useTemplates:
    - name: template1
      namespace: clickhouse
      useType: merge
    - name: template2
      namespace: clickhouse
      # No namespace specified - use CHI namespace

full yaml:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse"
  namespace: "clickhouse"
  labels:
    label1: clickhouse
  annotations:

spec:
  stop: "no"
  restart: "RollingUpdate"
  troubleshoot: "no"

  namespaceDomainPattern:  "%s.svc.cluster.local"

  templating:
    policy: "manual"

  reconciling:
    policy: "nowait"
    configMapPropagationTimeout: 90
    cleanup:
      unknownObjects:
        statefulSet: Delete
        pvc: Delete
        configMap: Delete
        service: Delete
      reconcileFailedObjects:
        statefulSet: Retain
        pvc: Retain
        configMap: Retain
        service: Retain

  # List of templates used by a CHI
  useTemplates:
    - name: template1
      namespace: clickhouse
      useType: merge
    - name: template2
      namespace: clickhouse
      # No namespace specified - use CHI namespace

  defaults:
    replicasUseFQDN: "no"
    distributedDDL:
      profile: default
    templates:
      hostTemplate: host-template-custom-ports
      podTemplate: clickhouse-v20.6
      dataVolumeClaimTemplate: default-volume-claim
      logVolumeClaimTemplate: default-volume-claim
      serviceTemplate: chi-service-template
      clusterServiceTemplate: cluster-service-template
      shardServiceTemplate: shard-service-template
      replicaServiceTemplate: replica-service-template

  configuration:
    zookeeper:
      nodes:
        - host: zookeeper-0.zookeepers.zookeeper.svc.cluster.local
          port: 2181
        - host: zookeeper-1.zookeepers.zookeeper.svc.cluster.local
          port: 2181
        - host: zookeeper-2.zookeepers.zookeeper.svc.cluster.local
          port: 2181
      session_timeout_ms: 30000
      operation_timeout_ms: 10000
      root: "/"
      identity: "user:password"
    users:
     #admin/profile: admin
      #admin/networks/ip:
      #  - "0.0.0.0/0"
      #  - "::/0"
      #admin/quotas: default
      admin/access_management: 1
      admin/networks/ip:
        - "0.0.0.0/0"
        - "::/0"
      admin/password_sha256_hex: "xxx"
      admin/host_regexp: "^.*$"

    #profiles:
    #  admin/admin: "1"
    #  default/max_memory_usage: "1000000000"
    quotas:
      default/interval/duration: "3600"
      #     <quotas>
      #       <default>
      #         <interval>
      #           <duration>3600</duration>
      #         </interval>
      #       </default>
      #     </quotas>
    settings:
      compression/case/method: zstd
      #      <compression>
      #        <case>
      #          <method>zstd</method>
      #        </case>
      #      </compression>
      disable_internal_dns_cache: 1
      #      <disable_internal_dns_cache>1</disable_internal_dns_cache>
    files:
      dict1.xml: |
        <yandex>
            <!-- ref to file /etc/clickhouse-data/config.d/source1.csv -->
        </yandex>
      source1.csv: |
        a1,b1,c1,d1
        a2,b2,c2,d2
    clusters:
      - name: sandbox-ch
        templates:
          podTemplate: clickhouse-v20.6
          dataVolumeClaimTemplate: default-volume-claim
          logVolumeClaimTemplate: default-volume-claim
        layout:
          shardsCount: 6
          replicasCount: 1

  templates:
    hostTemplates:
      - name: host-template-custom-ports
        spec:
          tcpPort: 9000
          httpPort: 8123
          interserverHTTPPort: 9009

    serviceTemplates:
      - name: chi-service-template
        # generateName understands different sets of macroses,
        # depending on the level of the object, for which Service is being created:
        #
        # For CHI-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        #
        # For Cluster-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        # 3. {cluster} - cluster name
        # 4. {clusterID} - short hashed cluster name (BEWARE, this is an experimental feature)
        # 5. {clusterIndex} - 0-based index of the cluster in the CHI (BEWARE, this is an experimental feature)
        #
        # For Shard-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        # 3. {cluster} - cluster name
        # 4. {clusterID} - short hashed cluster name (BEWARE, this is an experimental feature)
        # 5. {clusterIndex} - 0-based index of the cluster in the CHI (BEWARE, this is an experimental feature)
        # 6. {shard} - shard name
        # 7. {shardID} - short hashed shard name (BEWARE, this is an experimental feature)
        # 8. {shardIndex} - 0-based index of the shard in the cluster (BEWARE, this is an experimental feature)
        #
        # For Replica-level Service:
        # 1. {chi} - ClickHouseInstallation name
        # 2. {chiID} - short hashed ClickHouseInstallation name (BEWARE, this is an experimental feature)
        # 3. {cluster} - cluster name
        # 4. {clusterID} - short hashed cluster name (BEWARE, this is an experimental feature)
        # 5. {clusterIndex} - 0-based index of the cluster in the CHI (BEWARE, this is an experimental feature)
        # 6. {shard} - shard name
        # 7. {shardID} - short hashed shard name (BEWARE, this is an experimental feature)
        # 8. {shardIndex} - 0-based index of the shard in the cluster (BEWARE, this is an experimental feature)
        # 9. {replica} - replica name
        # 10. {replicaID} - short hashed replica name (BEWARE, this is an experimental feature)
        # 11. {replicaIndex} - 0-based index of the replica in the shard (BEWARE, this is an experimental feature)
        # 12. {chiScopeIndex} - 0-based index of the host in the chi (BEWARE, this is an experimental feature)
        # 13. {chiScopeCycleIndex} - 0-based index of the host's cycle in the chi-scope (BEWARE, this is an experimental feature)
        # 14. {chiScopeCycleOffset} - 0-based offset of the host in the chi-scope cycle (BEWARE, this is an experimental feature)
        # 15. {clusterScopeIndex} - 0-based index of the host in the cluster (BEWARE, this is an experimental feature)
        # 16. {clusterScopeCycleIndex} - 0-based index of the host's cycle in the cluster-scope (BEWARE, this is an experimental feature)
        # 17. {clusterScopeCycleOffset} - 0-based offset of the host in the cluster-scope cycle (BEWARE, this is an experimental feature)
        # 18. {shardScopeIndex} - 0-based index of the host in the shard (BEWARE, this is an experimental feature)
        # 19. {replicaScopeIndex} - 0-based index of the host in the replica (BEWARE, this is an experimental feature)
        # 20. {clusterScopeCycleHeadPointsToPreviousCycleTail} - 0-based cluster-scope index of previous cycle tail
        generateName: "service-{chi}"
        # type ObjectMeta struct from k8s.io/meta/v1
        metadata:
          labels:
            custom.label: "custom.value"
          annotations:
            # For more details on Internal Load Balancer check
            # https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"

            # NLB Load Balancer
            service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

        # type ServiceSpec struct from k8s.io/core/v1
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
          type: LoadBalancer

      - name: replica-service-template
        # type ServiceSpec struct from k8s.io/core/v1
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
            - name: interserver
              port: 9009
          type: ClusterIP
          clusterIP: None

      - name: preserve-client-source-ip
        # For more details on Preserving Client Source IP check
        # https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
        spec:
          selector:
            app: example
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
            - name: interserver
              port: 9009
          externalTrafficPolicy: Local
          type: LoadBalancer

    volumeClaimTemplates:
      - name: default-volume-claim
        # type PersistentVolumeClaimSpec struct from k8s.io/core/v1
        spec:
          # 1. If storageClassName is not specified, default StorageClass
          # (must be specified by cluster administrator) would be used for provisioning
          # 2. If storageClassName is set to an empty string (‘’), no storage class will be used
          # dynamic provisioning is disabled for this PVC. Existing, “Available”, PVs
          # (that do not have a specified storageClassName) will be considered for binding to the PVC
          #storageClassName: gold
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 501Gi
      - name: volume-claim-retain-pvc
        # Keep PVC from being deleted
        # Retaining PVC will also keep backing PV from deletion. This is useful in case we need to keep data intact.
        reclaimPolicy: Retain
        # type PersistentVolumeClaimSpec struct from k8s.io/core/v1
        spec:
          # 1. If storageClassName is not specified, default StorageClass
          # (must be specified by cluster administrator) would be used for provisioning
          # 2. If storageClassName is set to an empty string (‘’), no storage class will be used
          # dynamic provisioning is disabled for this PVC. Existing, “Available”, PVs
          # (that do not have a specified storageClassName) will be considered for binding to the PVC
          #storageClassName: gold
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 501Gi

    podTemplates:
      # multiple pod templates makes possible to update version smoothly
      # pod template for ClickHouse v18.16.1
      - name: clickhouse-v20.7
        # We may need to label nodes with clickhouse=allow label for this example to run
        # See ./label_nodes.sh for this purpose
        zone:
          key: "clickhouse"
          values:
            - "allow"
        # Shortcut version for AWS installations
        #zone:
        #  values:
        #    - "us-east-1a"

        # Possible values for podDistribution are:
        # Unspecified - empty value
        # ClickHouseAntiAffinity - AntiAffinity by ClickHouse instances.
        #   Pod pushes away other ClickHouse pods, which allows one ClickHouse instance per topologyKey-specified unit
        #   CH - (push away) - CH - (push away) - CH
        # ShardAntiAffinity - AntiAffinity by shard name.
        #   Pod pushes away other pods of the same shard (replicas of this shard),
        #   which allows one replica of a shard instance per topologyKey-specified unit.
        #   Other shards are allowed - it does not push all shards away, but CH-instances of the same shard only.
        #   Used for data loss avoidance - keeps all copies of the shard on different topologyKey-specified units.
        #   shard1,replica1 - (push away) - shard1,replica2 - (push away) - shard1,replica3
        # ReplicaAntiAffinity - AntiAffinity by replica name.
        #   Pod pushes away other pods of the same replica (shards of this replica),
        #   which allows one shard of a replica per topologyKey-specified unit.
        #   Other replicas are allowed - it does not push all replicas away, but CH-instances of the same replica only.
        #   Used to evenly distribute load from "full cluster scan" queries.
        #   shard1,replica1 - (push away) - shard2,replica1 - (push away) - shard3,replica1
        # AnotherNamespaceAntiAffinity - AntiAffinity by "another" namespace.
        #   Pod pushes away pods from another namespace, which allows same-namespace pods per topologyKey-specified unit.
        #   ns1 - (push away) - ns2 - (push away) - ns3
        # AnotherClickHouseInstallationAntiAffinity - AntiAffinity by "another" ClickHouseInstallation name.
        #   Pod pushes away pods from another ClickHouseInstallation,
        #   which allows same-ClickHouseInstallation pods per topologyKey-specified unit.
        #   CHI1 - (push away) - CHI2 - (push away) - CHI3
        # AnotherClusterAntiAffinity - AntiAffinity by "another" cluster name.
        #   Pod pushes away pods from another Cluster,
        #   which allows same-cluster pods per topologyKey-specified unit.
        #   cluster1 - (push away) - cluster2 - (push away) - cluster3
        # MaxNumberPerNode - AntiAffinity by cycle index.
        #   Pod pushes away pods from the same cycle,
        #   which allows to specify maximum number of ClickHouse instances per topologyKey-specified unit.
        #   Used to setup circular replication.
        # NamespaceAffinity - Affinity by namespace.
        #   Pod attracts pods from the same namespace,
        #   which allows pods from same namespace per topologyKey-specified unit.
        #   ns1 + (attracts) + ns1
        # ClickHouseInstallationAffinity - Affinity by ClickHouseInstallation name.
        #   Pod attracts pods from the same ClickHouseInstallation,
        #   which allows pods from the same CHI per topologyKey-specified unit.
        #   CHI1 + (attracts) + CHI1
        # ClusterAffinity - Affinity by cluster name.
        #   Pod attracts pods from the same cluster,
        #   which allows pods from the same Cluster per topologyKey-specified unit.
        #   cluster1 + (attracts) + cluster1
        # ShardAffinity - Affinity by shard name.
        #   Pod attracts pods from the same shard,
        #   which allows pods from the same Shard per topologyKey-specified unit.
        #   shard1 + (attracts) + shard1
        # ReplicaAffinity - Affinity by replica name.
        #   Pod attracts pods from the same replica,
        #   which allows pods from the same Replica per topologyKey-specified unit.
        #   replica1 + (attracts) + replica1
        # PreviousTailAffinity - Affinity to overlap cycles. Used to make cycle pod distribution
        #   cycle head + (attracts to) + previous cycle tail
        podDistribution:
          - type: ClickHouseAntiAffinity
          - type: MaxNumberPerNode
            number: 1
            # Apply podDistribution on per-host basis
            topologyKey: "kubernetes.io/hostname"
            # Apply podDistribution on per-zone basis
            #topologyKey: "kubernetes.io/zone"

        # type PodSpec struct {} from k8s.io/core/v1
        spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:21.10.2.15
              volumeMounts:
                - name: default-volume-claim
                  mountPath: /var/lib/clickhouse
              resources:
                requests:
                  memory: "20G"
                  cpu: "3001m"
                limits:
                  memory: "20G"
                  cpu: "3001m"

      # pod template for ClickHouse v19.11.3.11
      - name: clickhouse-v20.6
        # type PodSpec struct {} from k8s.io/core/v1
        spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:21.10.2.15
              volumeMounts:
                - name: default-volume-claim
                  mountPath: /var/lib/clickhouse
              resources:
                requests:
                  memory: "20G"
                  cpu: "3001m"
                limits:
                  memory: "20G"
                  cpu: "3001m"```

[FarhadF]

update: I removed the template part from last comment and applied CHI, it recreated the pods. After recreation was done I deleted the operator pod (restart simulation) and it started to recreate clickhouse pods again.

This log line could be the cause:

I1027 10:54:59.121404       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-0-0

labels on the ch-0-0 statefulset:

Name:               chi-clickhouse-sandbox-ch-0-0
Namespace:          clickhouse
CreationTimestamp:  Mon, 25 Oct 2021 18:22:32 +0200
Selector:           clickhouse.altinity.com/app=chop,clickhouse.altinity.com/chi=clickhouse,clickhouse.altinity.com/cluster=sandbox-ch,clickhouse.altinity.com/namespace=clickhouse,clickhouse.altinity.com/replica=0,clickhouse.altinity.com/shard=0
Labels:             clickhouse.altinity.com/app=chop
                    clickhouse.altinity.com/chi=clickhouse
                    clickhouse.altinity.com/cluster=sandbox-ch
                    clickhouse.altinity.com/namespace=clickhouse
                    clickhouse.altinity.com/object-version=7022bf5cf69679801e3e8ac0e6d750195f9d7912
                    clickhouse.altinity.com/replica=0
                    clickhouse.altinity.com/settings-version=06a596b5d8f930701f0661e49273cccbc4db2fbe
                    clickhouse.altinity.com/shard=0
                    clickhouse.altinity.com/zookeeper-version=0801121bef43f105f5af22b132fd5e9e7d776d84
                    label1=clickhouse

full log after restarting the operator:

I1027 10:54:48.103121       1 announcer.go:226] clickhouse_operator.go:87:Run():start
I1027 10:54:48.103181       1 announcer.go:224] clickhouse_operator.go:95:Run():Starting clickhouse-operator. Version:0.16.0 GitSHA:4767ddc6 BuiltAt:2021-10-03T18:11:19
I1027 10:54:48.104043       1 announcer.go:218] Parameters num: 11
I1027 10:54:48.104052       1 announcer.go:218] OPERATOR_CONTAINER_CPU_LIMIT=8
I1027 10:54:48.104054       1 announcer.go:218] OPERATOR_CONTAINER_CPU_REQUEST=0
I1027 10:54:48.104056       1 announcer.go:218] OPERATOR_CONTAINER_MEM_LIMIT=15261237248
I1027 10:54:48.104059       1 announcer.go:218] OPERATOR_CONTAINER_MEM_REQUEST=0
I1027 10:54:48.104061       1 announcer.go:218] OPERATOR_POD_IP=10.20.51.183
I1027 10:54:48.104063       1 announcer.go:218] OPERATOR_POD_NAME=clickhouse-operator-55dcf7f88-rqwlb
I1027 10:54:48.104065       1 announcer.go:218] OPERATOR_POD_NAMESPACE=kube-system
I1027 10:54:48.104068       1 announcer.go:218] OPERATOR_POD_NODE_NAME=xxx
I1027 10:54:48.104070       1 announcer.go:218] OPERATOR_POD_SERVICE_ACCOUNT=clickhouse-operator
I1027 10:54:48.104073       1 announcer.go:218] WATCH_NAMESPACE=
I1027 10:54:48.104075       1 announcer.go:218] WATCH_NAMESPACES=
I1027 10:54:48.106668       1 announcer.go:220] File-based ClickHouseOperatorConfigurations
I1027 10:54:48.106712       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: 
WatchNamespaces (0):
CHCommonConfigsPath: config.d
CHHostConfigsPath: conf.d
CHUsersConfigsPath: users.d
CHCommonConfigs (0):
CHHostConfigs (0):
CHUsersConfigs (0):
CHITemplatesPath: templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1027 10:54:48.127874       1 announcer.go:220] Unified (but not post-processed yet) CHOP config
I1027 10:54:48.127917       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: 
WatchNamespaces (0):
CHCommonConfigsPath: config.d
CHHostConfigsPath: conf.d
CHUsersConfigsPath: users.d
CHCommonConfigs (0):
CHHostConfigs (0):
CHUsersConfigs (0):
CHITemplatesPath: templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1027 10:54:48.128090       1 announcer.go:220] Final CHOP config
I1027 10:54:48.128134       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: kube-system
WatchNamespaces (0):
CHCommonConfigsPath: /etc/clickhouse-operator/config.d
CHHostConfigsPath: /etc/clickhouse-operator/conf.d
CHUsersConfigsPath: /etc/clickhouse-operator/users.d
CHCommonConfigs (4):
  - [01-clickhouse-01-listen.xml]=<yandex>
    <!-- Listen wildcard address to allow accepting connections from other containers and host network. -->
    <listen_host>::</listen_host>
    <listen_host>0.0.0.0</listen_host>
    <listen_try>1</listen_try>
</yandex>

  - [01-clickhouse-02-logger.xml]=<yandex>
    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>1</console>
    </logger>
</yandex>

  - [01-clickhouse-03-query_log.xml]=<yandex>
    <query_log replace="1">
        <database>system</database>
        <table>query_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>
    <query_thread_log remove="1"/>
</yandex>

  - [01-clickhouse-04-part_log.xml]=<yandex>
    <part_log replace="1">
        <database>system</database>
        <table>part_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </part_log>
</yandex>

CHHostConfigs (0):
CHUsersConfigs (3):
  - [01-clickhouse-user.xml]=<yandex>
    <users>
        <clickhouse_operator>
            <networks>
                <ip>127.0.0.1</ip>
                <ip>0.0.0.0/0</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>xxx</password_sha256_hex>
            <profile>clickhouse_operator</profile>
            <quota>default</quota>
        </clickhouse_operator>
    </users>
    <profiles>
        <clickhouse_operator>
            <log_queries>0</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
        </clickhouse_operator>
    </profiles>
</yandex>

  - [02-clickhouse-default-profile.xml]=<yandex>
  <profiles>
    <default>
      <log_queries>1</log_queries>
      <connect_timeout_with_failover_ms>1000</connect_timeout_with_failover_ms>
      <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>
      <parallel_view_processing>1</parallel_view_processing>
    </default>
  </profiles>
</yandex>

  - [03-database-ordinary.xml]=<!--  Remove it for ClickHouse versions before 20.4 -->
<yandex>
    <profiles>
        <default>
            <default_database_engine>Ordinary</default_database_engine>
        </default>
    </profiles>
</yandex>

CHITemplatesPath: /etc/clickhouse-operator/templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1027 10:54:48.128150       1 announcer.go:218] Log option 'logtostderr' change value from 'true' to 'true'
I1027 10:54:48.128158       1 announcer.go:218] Log option 'alsologtostderr' change value from 'false' to 'false'
I1027 10:54:48.128161       1 announcer.go:218] Log option 'v' change value from '1' to '1'
I1027 10:54:48.128167       1 announcer.go:220] Additional log options applied
I1027 10:54:48.128178       1 announcer.go:220] clickhouse_operator.go:102:Run():Log options parsed
I1027 10:54:48.128209       1 announcer.go:226] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: kube-system
WatchNamespaces (0):
CHCommonConfigsPath: /etc/clickhouse-operator/config.d
CHHostConfigsPath: /etc/clickhouse-operator/conf.d
CHUsersConfigsPath: /etc/clickhouse-operator/users.d
CHCommonConfigs (4):
  - [01-clickhouse-01-listen.xml]=<yandex>
    <!-- Listen wildcard address to allow accepting connections from other containers and host network. -->
    <listen_host>::</listen_host>
    <listen_host>0.0.0.0</listen_host>
    <listen_try>1</listen_try>
</yandex>

  - [01-clickhouse-02-logger.xml]=<yandex>
    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>1</console>
    </logger>
</yandex>

  - [01-clickhouse-03-query_log.xml]=<yandex>
    <query_log replace="1">
        <database>system</database>
        <table>query_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>
    <query_thread_log remove="1"/>
</yandex>

  - [01-clickhouse-04-part_log.xml]=<yandex>
    <part_log replace="1">
        <database>system</database>
        <table>part_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </part_log>
</yandex>

CHHostConfigs (0):
CHUsersConfigs (3):
  - [01-clickhouse-user.xml]=<yandex>
    <users>
        <clickhouse_operator>
            <networks>
                <ip>127.0.0.1</ip>
                <ip>0.0.0.0/0</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448</password_sha256_hex>
            <profile>clickhouse_operator</profile>
            <quota>default</quota>
        </clickhouse_operator>
    </users>
    <profiles>
        <clickhouse_operator>
            <log_queries>0</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
        </clickhouse_operator>
    </profiles>
</yandex>

  - [02-clickhouse-default-profile.xml]=<yandex>
  <profiles>
    <default>
      <log_queries>1</log_queries>
      <connect_timeout_with_failover_ms>1000</connect_timeout_with_failover_ms>
      <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>
      <parallel_view_processing>1</parallel_view_processing>
    </default>
  </profiles>
</yandex>

  - [03-database-ordinary.xml]=<!--  Remove it for ClickHouse versions before 20.4 -->
<yandex>
    <profiles>
        <default>
            <default_database_engine>Ordinary</default_database_engine>
        </default>
    </profiles>
</yandex>

CHITemplatesPath: /etc/clickhouse-operator/templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-{chi})\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: 
CHCredentialsSecretName: 
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold: 
VModule: 
LogBacktraceAt: 
ReconcileThreadsNumber: 10
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
I1027 10:54:48.128565       1 announcer.go:220] clickhouse_operator.go:145:Run():Starting CHI controller
I1027 10:54:48.128825       1 announcer.go:220] Starting ClickHouseInstallation controller
I1027 10:54:48.128848       1 announcer.go:218] waitForCacheSync():Syncing caches for ClickHouseInstallation controller
I1027 10:54:48.139860       1 announcer.go:218] ENQUEUE new ReconcileCHI cmd=add for clickhouse/clickhouse
I1027 10:54:48.229033       1 announcer.go:218] waitForCacheSync():Caches are synced for ClickHouseInstallation controller
I1027 10:54:48.295599       1 announcer.go:218] controller.go:431:Run():ClickHouseInstallation controller: starting workers number: 11
I1027 10:54:48.295636       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 1 out of 11
I1027 10:54:48.295661       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 2 out of 11
I1027 10:54:48.295670       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 3 out of 11
I1027 10:54:48.295683       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 4 out of 11
I1027 10:54:48.295695       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 5 out of 11
I1027 10:54:48.295708       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 6 out of 11
I1027 10:54:48.295718       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 7 out of 11
I1027 10:54:48.295765       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 8 out of 11
I1027 10:54:48.295774       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 9 out of 11
I1027 10:54:48.295787       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 10 out of 11
I1027 10:54:48.295801       1 announcer.go:218] controller.go:433:Run():ClickHouseInstallation controller: starting worker 11 out of 11
I1027 10:54:48.295819       1 announcer.go:220] controller.go:439:Run():ClickHouseInstallation controller: workers started
I1027 10:54:58.345602       1 announcer.go:220] markReconcileStart():clickhouse/clickhouse:reconcile started
I1027 10:54:58.386672       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:0|rep:0|host:0-0
I1027 10:54:58.386696       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:1|rep:0|host:1-0
I1027 10:54:58.386705       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:2|rep:0|host:2-0
I1027 10:54:58.386715       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:3|rep:0|host:3-0
I1027 10:54:58.386723       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:4|rep:0|host:4-0
I1027 10:54:58.386732       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:sandbox-ch|sha:5|rep:0|host:5-0
W1027 10:54:58.399351       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1027 10:54:58.399639       1 announcer.go:245] unable to create PDB poddisruptionbudgets.policy "clickhouse" already exists
I1027 10:54:58.399689       1 announcer.go:218] CreateServiceCHI():clickhouse/clickhouse:clickhouse/service-clickhouse
I1027 10:54:58.399951       1 announcer.go:224] updateService():clickhouse/clickhouse:reuse Port 8123 values
I1027 10:54:58.399971       1 announcer.go:224] updateService():clickhouse/clickhouse:reuse Port 9000 values
I1027 10:54:58.410476       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/service-clickhouse
I1027 10:54:58.477444       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 10:54:58.520399       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-usersd
I1027 10:54:58.560521       1 announcer.go:218] CreateServiceCluster():clickhouse/clickhouse:clickhouse/cluster-clickhouse-sandbox-ch
I1027 10:54:58.560567       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-0
I1027 10:54:58.560634       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 0-0 started
I1027 10:54:58.717716       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-0-0 use custom template clickhouse-v20.6
I1027 10:54:58.717801       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-0-0
W1027 10:54:58.717824       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 10:54:58.717838       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 10:54:58.717850       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 10:54:58.717862       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 10:54:58.719517       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-0-0
I1027 10:54:58.719543       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 0 cluster sandbox-ch
I1027 10:54:58.744291       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local]
I1027 10:54:58.754569       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/0-0-OK
I1027 10:54:58.762698       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-0-0
I1027 10:54:59.119558       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-0-0 use custom template clickhouse-v20.6
I1027 10:54:59.119637       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-0-0
W1027 10:54:59.119657       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 10:54:59.119671       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 10:54:59.119683       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 10:54:59.119695       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 10:54:59.121404       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-0-0
I1027 10:54:59.121472       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - started
I1027 10:54:59.520741       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.598749971s 401.250029ms/1m30s
I1027 10:56:29.147355       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 15=>16
I1027 10:56:34.170325       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:16 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1027 10:56:39.187069       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:16 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1027 10:56:39.219094       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - completed
I1027 10:56:39.263959       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-0-0 use custom template clickhouse-v20.6
I1027 10:56:39.264107       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-0-0
W1027 10:56:39.264136       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 10:56:39.264147       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 10:56:39.264161       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 10:56:39.264177       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-0-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 10:56:39.266599       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-0-0
I1027 10:56:39.266658       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - started
I1027 10:56:39.305488       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.185998395s/1m30s
I1027 10:56:39.317694       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 16=>17
I1027 10:57:39.737767       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:17 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1027 10:57:54.837585       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-0-0:OK  :ObservedGeneration:17 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c UpdateRevision:chi-clickhouse-sandbox-ch-0-0-69d8b74b4c
I1027 10:57:54.889435       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-0-0) - completed
I1027 10:57:55.029119       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-0-0 for Set chi-clickhouse-sandbox-ch-0-0
I1027 10:57:55.229803       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-0-0
I1027 10:57:55.480752       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:0/0 cluster:sandbox-ch
I1027 10:57:55.658397       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 0-0
I1027 10:57:55.658497       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1027 10:57:55.658811       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 10:57:55.658864       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1027 10:57:55.658884       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1027 10:57:55.658974       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.668892       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1027 10:57:55.668913       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.668922       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.678328       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1027 10:57:55.678347       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.678357       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.686255       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1027 10:57:55.686275       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.686284       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.694403       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1027 10:57:55.694425       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.694436       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.704915       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1027 10:57:55.704935       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.704944       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.712141       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
W1027 10:57:55.712160       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
E1027 10:57:55.712174       1 announcer.go:267] cluster.go:95:QueryAny():FAILED to run query on all hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 10:57:55.712212       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1027 10:57:55.712234       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1027 10:57:55.712253       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1027 10:57:55.712281       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1027 10:57:55.712442       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.714217       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 10:57:55.714234       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.714245       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.716485       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 10:57:55.716500       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.716508       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.718442       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 10:57:55.718458       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.718465       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.720495       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 10:57:55.720512       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.720521       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.723279       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 10:57:55.723297       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
I1027 10:57:55.723306       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 10:57:55.725057       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 10:57:55.725077       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 170. DB::Exception: Requested cluster 'sandbox-ch' not found. (BAD_GET) (version 21.10.2.15 (official build))
E1027 10:57:55.725097       1 announcer.go:267] cluster.go:95:QueryAny():FAILED to run query on all hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 10:57:55.725145       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1027 10:57:55.725173       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1027 10:57:55.725196       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 0 cluster sandbox-ch
I1027 10:57:55.831353       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 10:57:56.437186       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 0-0 completed
I1027 10:57:56.454428       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.196  0xc0007addd0 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:7c83f4a3-ad45-46e4-83f2-3da0d986af65,APIVersion:,ResourceVersion:4115108,FieldPath:,}} {10.20.101.242  0xc0007adde0 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:a8765b92-6cba-408d-943d-9353a1c24894,APIVersion:,ResourceVersion:4113564,FieldPath:,}} {10.20.2.108  0xc0007addf0 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:a605f3d2-5556-4b15-a0f9-e8c0242a906f,APIVersion:,ResourceVersion:4114327,FieldPath:,}} {10.20.2.16  0xc0007ade00 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:da148245-47ea-47ea-af48-4072a2054cbf,APIVersion:,ResourceVersion:4112766,FieldPath:,}} {10.20.51.112  0xc0007ade10 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:6a3624dd-570f-4b88-bcb2-17477a81746f,APIVersion:,ResourceVersion:4117551,FieldPath:,}} {10.20.51.126  0xc0007ade20 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:0a1bc85c-a0c9-4b74-bd44-1abfd0edb303,APIVersion:,ResourceVersion:4115885,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
I1027 10:57:56.657498       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-1
I1027 10:57:56.657536       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 1-0 started
I1027 10:57:56.866234       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-1-0 use custom template clickhouse-v20.6
I1027 10:57:56.866324       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-1-0
W1027 10:57:56.866343       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 10:57:56.866354       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 10:57:56.866362       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 10:57:56.866371       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 10:57:56.868482       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-1-0
I1027 10:57:56.868514       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 1 cluster sandbox-ch
I1027 10:57:57.247249       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local]
I1027 10:57:57.249322       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/1-0-OK
I1027 10:57:57.426286       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-1-0
I1027 10:57:57.658476       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-1-0 use custom template clickhouse-v20.6
I1027 10:57:57.658555       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-1-0
W1027 10:57:57.658575       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 10:57:57.658588       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 10:57:57.658597       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 10:57:57.658605       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 10:57:57.660420       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-1-0
I1027 10:57:57.660478       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - started
I1027 10:57:57.862533       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.795877224s 204.122776ms/1m30s
W1027 10:58:48.900155       1 reflector.go:436] pkg/client/informers/externalversions/factory.go:117: watch of *v1.ClickHouseOperatorConfiguration ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind \"ClickHouseOperatorConfiguration\" is registered for version \"clickhouse.altinity.com/v1\" in scheme \"pkg/client/clientset/versioned/scheme/register.go:30\"") has prevented the request from succeeding
I1027 10:59:27.677250       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 18=>19
I1027 10:59:32.698725       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:19 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1027 10:59:37.716787       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:19 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1027 10:59:37.748086       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - completed
I1027 10:59:37.784828       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-1-0 use custom template clickhouse-v20.6
I1027 10:59:37.784977       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-1-0
W1027 10:59:37.785004       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 10:59:37.785016       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 10:59:37.785033       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 10:59:37.785045       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-1-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 10:59:37.788304       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-1-0
I1027 10:59:37.788388       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - started
I1027 10:59:37.825740       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.167327457s/1m30s
I1027 10:59:37.840388       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 19=>20
I1027 11:00:38.337929       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:20 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1027 11:00:53.456678       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-1-0:OK  :ObservedGeneration:20 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6 UpdateRevision:chi-clickhouse-sandbox-ch-1-0-bdf846bf6
I1027 11:00:53.493308       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-1-0) - completed
I1027 11:00:53.647229       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-1-0 for Set chi-clickhouse-sandbox-ch-1-0
I1027 11:00:53.844558       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-1-0
I1027 11:00:54.069685       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:1/0 cluster:sandbox-ch
I1027 11:00:54.272027       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 1-0
I1027 11:00:54.272088       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1027 11:00:54.272202       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:00:54.272238       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1027 11:00:54.272253       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1027 11:00:54.272333       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:00:54.288827       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1027 11:00:54.288850       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names1: default
I1027 11:00:54.288863       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1027 11:00:54.288874       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql1: CREATE DATABASE IF NOT EXISTS "default"
I1027 11:00:54.288888       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1027 11:00:54.288903       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1027 11:00:54.289035       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:00:54.312317       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1027 11:00:54.312347       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations_local
I1027 11:00:54.312368       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days_local
I1027 11:00:54.312387       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations_local
I1027 11:00:54.312410       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user_local
I1027 11:00:54.312434       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1_local
I1027 11:00:54.312459       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations
I1027 11:00:54.312477       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days
I1027 11:00:54.312501       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations
I1027 11:00:54.312522       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user
I1027 11:00:54.312544       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1
I1027 11:00:54.312567       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1027 11:00:54.312593       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192
I1027 11:00:54.312621       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192
I1027 11:00:54.312646       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192
I1027 11:00:54.312672       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:00:54.312695       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:00:54.312722       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand())
I1027 11:00:54.312747       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand())
I1027 11:00:54.312773       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand())
I1027 11:00:54.312795       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand())
I1027 11:00:54.312818       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())
I1027 11:00:54.312841       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Creating distributed objects at 1-0: [default default.campaign_in_app_event_aggregations_local default.campaign_in_app_purchase_event_days_local default.campaign_postback_aggregations_local default.user_event_aggregations_user_local default.user_event_aggregations_user1_local default.campaign_in_app_event_aggregations default.campaign_in_app_purchase_event_days default.campaign_postback_aggregations default.user_event_aggregations_user default.user_event_aggregations_user1]
I1027 11:00:54.312867       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:
[CREATE DATABASE IF NOT EXISTS "default" CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())]
I1027 11:00:54.331783       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 1 cluster sandbox-ch
I1027 11:00:54.441546       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 11:00:55.049554       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 1-0 completed
I1027 11:00:55.064084       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.196  0xc0002c9520 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:7c83f4a3-ad45-46e4-83f2-3da0d986af65,APIVersion:,ResourceVersion:4115108,FieldPath:,}} {10.20.101.242  0xc0002c9540 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:a8765b92-6cba-408d-943d-9353a1c24894,APIVersion:,ResourceVersion:4113564,FieldPath:,}} {10.20.2.108  0xc0002c9560 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:a605f3d2-5556-4b15-a0f9-e8c0242a906f,APIVersion:,ResourceVersion:4114327,FieldPath:,}} {10.20.2.212  0xc0002c9570 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:f9b592a2-ddc9-48ba-8bd4-a61d9947abad,APIVersion:,ResourceVersion:4118275,FieldPath:,}} {10.20.51.112  0xc0002c9580 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:6a3624dd-570f-4b88-bcb2-17477a81746f,APIVersion:,ResourceVersion:4117551,FieldPath:,}} {10.20.51.126  0xc0002c9590 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:0a1bc85c-a0c9-4b74-bd44-1abfd0edb303,APIVersion:,ResourceVersion:4115885,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
I1027 11:00:55.285350       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-2
I1027 11:00:55.285389       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 2-0 started
I1027 11:00:55.481969       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-2-0 use custom template clickhouse-v20.6
I1027 11:00:55.482352       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-2-0
W1027 11:00:55.482389       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:00:55.482409       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:00:55.482426       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:00:55.482440       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:00:55.485444       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-2-0
I1027 11:00:55.485467       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 2 cluster sandbox-ch
I1027 11:00:55.852820       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local]
I1027 11:00:55.861452       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/2-0-OK
I1027 11:00:56.038565       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-2-0
I1027 11:00:56.355055       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-2-0 use custom template clickhouse-v20.6
I1027 11:00:56.355134       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-2-0
W1027 11:00:56.355152       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:00:56.355163       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:00:56.355170       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:00:56.355177       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:00:56.357756       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-2-0
I1027 11:00:56.357860       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-2-0) - started
I1027 11:00:56.586661       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.768328318s 231.671682ms/1m30s
I1027 11:02:26.372101       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 20=>21
I1027 11:02:31.390443       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-2-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749 UpdateRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749
I1027 11:02:36.403428       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-2-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749 UpdateRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749
I1027 11:02:36.433464       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-2-0) - completed
I1027 11:02:36.477632       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-2-0 use custom template clickhouse-v20.6
I1027 11:02:36.477722       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-2-0
W1027 11:02:36.477742       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:02:36.477756       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:02:36.477766       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:02:36.477780       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-2-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:02:36.479963       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-2-0
I1027 11:02:36.480012       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-2-0) - started
I1027 11:02:36.518653       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.163655134s/1m30s
I1027 11:02:36.531603       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 21=>22
I1027 11:03:31.956248       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-2-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749 UpdateRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749
I1027 11:03:47.072378       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-2-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749 UpdateRevision:chi-clickhouse-sandbox-ch-2-0-794ddbb749
I1027 11:03:47.151453       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-2-0) - completed
I1027 11:03:47.339901       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-2-0 for Set chi-clickhouse-sandbox-ch-2-0
I1027 11:03:47.473754       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-2-0
I1027 11:03:47.683517       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:2/0 cluster:sandbox-ch
I1027 11:03:47.884112       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 2-0
I1027 11:03:47.884198       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1027 11:03:47.884383       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:03:47.884435       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1027 11:03:47.884454       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1027 11:03:47.884627       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:03:47.906903       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1027 11:03:47.906930       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names1: default
I1027 11:03:47.906945       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1027 11:03:47.906962       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql1: CREATE DATABASE IF NOT EXISTS "default"
I1027 11:03:47.906982       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1027 11:03:47.907004       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1027 11:03:47.907122       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:03:47.962030       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1027 11:03:47.962058       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations_local
I1027 11:03:47.962076       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days_local
I1027 11:03:47.962094       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations_local
I1027 11:03:47.962110       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user_local
I1027 11:03:47.962126       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1_local
I1027 11:03:47.962141       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations
I1027 11:03:47.962159       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days
I1027 11:03:47.962176       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations
I1027 11:03:47.962190       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user
I1027 11:03:47.962206       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1
I1027 11:03:47.962224       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1027 11:03:47.962247       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192
I1027 11:03:47.962268       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192
I1027 11:03:47.962285       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192
I1027 11:03:47.962301       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:03:47.962320       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:03:47.962345       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand())
I1027 11:03:47.962365       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand())
I1027 11:03:47.962382       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand())
I1027 11:03:47.962399       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand())
I1027 11:03:47.962417       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())
I1027 11:03:47.962436       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Creating distributed objects at 2-0: [default default.campaign_in_app_event_aggregations_local default.campaign_in_app_purchase_event_days_local default.campaign_postback_aggregations_local default.user_event_aggregations_user_local default.user_event_aggregations_user1_local default.campaign_in_app_event_aggregations default.campaign_in_app_purchase_event_days default.campaign_postback_aggregations default.user_event_aggregations_user default.user_event_aggregations_user1]
I1027 11:03:47.962464       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:
[CREATE DATABASE IF NOT EXISTS "default" CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())]
I1027 11:03:47.978925       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 2 cluster sandbox-ch
I1027 11:03:48.063296       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 11:03:48.667455       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 2-0 completed
I1027 11:03:48.678946       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.196  0xc000fbcb20 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:7c83f4a3-ad45-46e4-83f2-3da0d986af65,APIVersion:,ResourceVersion:4115108,FieldPath:,}} {10.20.101.218  0xc000fbcb30 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:5172ee09-1c7f-438b-8a2c-9ce14c0b9583,APIVersion:,ResourceVersion:4118977,FieldPath:,}} {10.20.2.108  0xc000fbcb40 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:a605f3d2-5556-4b15-a0f9-e8c0242a906f,APIVersion:,ResourceVersion:4114327,FieldPath:,}} {10.20.2.212  0xc000fbcb50 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:f9b592a2-ddc9-48ba-8bd4-a61d9947abad,APIVersion:,ResourceVersion:4118275,FieldPath:,}} {10.20.51.112  0xc000fbcb60 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:6a3624dd-570f-4b88-bcb2-17477a81746f,APIVersion:,ResourceVersion:4117551,FieldPath:,}} {10.20.51.126  0xc000fbcb80 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:0a1bc85c-a0c9-4b74-bd44-1abfd0edb303,APIVersion:,ResourceVersion:4115885,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
I1027 11:03:48.888290       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-3
I1027 11:03:48.888331       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 3-0 started
I1027 11:03:49.077239       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-3-0 use custom template clickhouse-v20.6
I1027 11:03:49.077330       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-3-0
W1027 11:03:49.077350       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:03:49.077358       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:03:49.077373       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:03:49.077381       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:03:49.079153       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-3-0
I1027 11:03:49.079176       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 3 cluster sandbox-ch
I1027 11:03:49.462246       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local]
I1027 11:03:49.470949       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/3-0-OK
I1027 11:03:49.644167       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-3-0
I1027 11:03:49.879045       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-3-0 use custom template clickhouse-v20.6
I1027 11:03:49.879161       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-3-0
W1027 11:03:49.879187       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:03:49.879203       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:03:49.879218       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:03:49.879232       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:03:49.882618       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-3-0
I1027 11:03:49.882713       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-3-0) - started
I1027 11:03:50.077020       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.801926222s 198.073778ms/1m30s
I1027 11:05:19.902992       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 20=>21
I1027 11:05:24.917488       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-3-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb UpdateRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb
I1027 11:05:29.930690       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-3-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb UpdateRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb
I1027 11:05:29.967630       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-3-0) - completed
I1027 11:05:30.020457       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-3-0 use custom template clickhouse-v20.6
I1027 11:05:30.020578       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-3-0
W1027 11:05:30.020606       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:05:30.020616       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:05:30.020627       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:05:30.020667       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-3-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:05:30.025337       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-3-0
I1027 11:05:30.025418       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-3-0) - started
I1027 11:05:30.103845       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.224899554s/1m30s
I1027 11:05:30.122021       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 21=>22
I1027 11:06:25.493201       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-3-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb UpdateRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb
I1027 11:06:55.732963       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-3-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb UpdateRevision:chi-clickhouse-sandbox-ch-3-0-8f9fd95bb
I1027 11:06:55.793439       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-3-0) - completed
I1027 11:06:55.943842       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-3-0 for Set chi-clickhouse-sandbox-ch-3-0
I1027 11:06:56.108619       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-3-0
I1027 11:06:56.335603       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:3/0 cluster:sandbox-ch
I1027 11:06:56.536560       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 3-0
I1027 11:06:56.536629       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1027 11:06:56.536756       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:06:56.536799       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1027 11:06:56.536817       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1027 11:06:56.537008       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:06:56.558407       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1027 11:06:56.558435       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names1: default
I1027 11:06:56.558452       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1027 11:06:56.558464       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql1: CREATE DATABASE IF NOT EXISTS "default"
I1027 11:06:56.558477       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1027 11:06:56.558490       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1027 11:06:56.558599       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:06:56.624875       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1027 11:06:56.624908       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations_local
I1027 11:06:56.624927       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days_local
I1027 11:06:56.624940       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations_local
I1027 11:06:56.624954       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user_local
I1027 11:06:56.624970       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1_local
I1027 11:06:56.624983       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations
I1027 11:06:56.624997       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days
I1027 11:06:56.625016       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations
I1027 11:06:56.625029       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user
I1027 11:06:56.625040       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1
I1027 11:06:56.625054       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1027 11:06:56.625072       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192
I1027 11:06:56.625087       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192
I1027 11:06:56.625152       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192
I1027 11:06:56.625176       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:06:56.625192       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:06:56.625210       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand())
I1027 11:06:56.625234       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand())
I1027 11:06:56.625257       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand())
I1027 11:06:56.625276       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand())
I1027 11:06:56.625294       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())
I1027 11:06:56.625311       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Creating distributed objects at 3-0: [default default.campaign_in_app_event_aggregations_local default.campaign_in_app_purchase_event_days_local default.campaign_postback_aggregations_local default.user_event_aggregations_user_local default.user_event_aggregations_user1_local default.campaign_in_app_event_aggregations default.campaign_in_app_purchase_event_days default.campaign_postback_aggregations default.user_event_aggregations_user default.user_event_aggregations_user1]
I1027 11:06:56.625331       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:
[CREATE DATABASE IF NOT EXISTS "default" CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())]
I1027 11:06:56.644612       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 3 cluster sandbox-ch
I1027 11:06:56.713610       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 11:06:57.319271       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 3-0 completed
I1027 11:06:57.349996       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.196  0xc00068bb20 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:7c83f4a3-ad45-46e4-83f2-3da0d986af65,APIVersion:,ResourceVersion:4115108,FieldPath:,}} {10.20.101.218  0xc00068bb40 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:5172ee09-1c7f-438b-8a2c-9ce14c0b9583,APIVersion:,ResourceVersion:4118977,FieldPath:,}} {10.20.2.212  0xc00068bb60 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:f9b592a2-ddc9-48ba-8bd4-a61d9947abad,APIVersion:,ResourceVersion:4118275,FieldPath:,}} {10.20.2.246  0xc00068bb80 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:df7ce43b-65c3-41aa-8c66-a2e93f018b49,APIVersion:,ResourceVersion:4119737,FieldPath:,}} {10.20.51.112  0xc00068bba0 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:6a3624dd-570f-4b88-bcb2-17477a81746f,APIVersion:,ResourceVersion:4117551,FieldPath:,}} {10.20.51.126  0xc00068bbc0 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:0a1bc85c-a0c9-4b74-bd44-1abfd0edb303,APIVersion:,ResourceVersion:4115885,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
I1027 11:06:57.541864       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-4
I1027 11:06:57.541904       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 4-0 started
I1027 11:06:57.738379       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-4-0 use custom template clickhouse-v20.6
I1027 11:06:57.738453       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-4-0
W1027 11:06:57.738476       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:06:57.738487       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:06:57.738495       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:06:57.738507       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:06:57.740321       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-4-0
I1027 11:06:57.740344       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 4 cluster sandbox-ch
I1027 11:06:58.125351       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local]
I1027 11:06:58.133225       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/4-0-OK
I1027 11:06:58.306454       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-4-0
I1027 11:06:58.539910       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-4-0 use custom template clickhouse-v20.6
I1027 11:06:58.539993       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-4-0
W1027 11:06:58.540009       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:06:58.540019       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:06:58.540027       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:06:58.540034       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:06:58.542329       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-4-0
I1027 11:06:58.542387       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-4-0) - started
I1027 11:06:58.744781       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.795059587s 204.940413ms/1m30s
I1027 11:08:28.552785       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 20=>21
I1027 11:08:33.570721       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-4-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959 UpdateRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959
W1027 11:08:36.144958       1 reflector.go:436] pkg/client/informers/externalversions/factory.go:117: watch of *v1.ClickHouseOperatorConfiguration ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind \"ClickHouseOperatorConfiguration\" is registered for version \"clickhouse.altinity.com/v1\" in scheme \"pkg/client/clientset/versioned/scheme/register.go:30\"") has prevented the request from succeeding
I1027 11:08:38.583813       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-4-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959 UpdateRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959
I1027 11:08:38.627477       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-4-0) - completed
I1027 11:08:38.669183       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-4-0 use custom template clickhouse-v20.6
I1027 11:08:38.669279       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-4-0
W1027 11:08:38.669299       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:08:38.669314       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:08:38.669325       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:08:38.669336       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-4-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:08:38.671093       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-4-0
I1027 11:08:38.671162       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-4-0) - started
I1027 11:08:38.709086       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.169244146s/1m30s
I1027 11:08:38.721957       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 21=>22
I1027 11:09:34.079492       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-4-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959 UpdateRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959
I1027 11:09:59.238951       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-4-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959 UpdateRevision:chi-clickhouse-sandbox-ch-4-0-6c7b495959
I1027 11:09:59.274315       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-4-0) - completed
I1027 11:09:59.430939       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-4-0 for Set chi-clickhouse-sandbox-ch-4-0
I1027 11:09:59.631048       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-4-0
I1027 11:09:59.864784       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:4/0 cluster:sandbox-ch
I1027 11:10:00.085839       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 4-0
I1027 11:10:00.085897       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1027 11:10:00.086004       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:10:00.086041       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1027 11:10:00.086058       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1027 11:10:00.086147       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:10:00.118571       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1027 11:10:00.118600       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names1: default
I1027 11:10:00.118617       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1027 11:10:00.118635       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql1: CREATE DATABASE IF NOT EXISTS "default"
I1027 11:10:00.118660       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1027 11:10:00.118684       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1027 11:10:00.118800       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 11:10:00.174838       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-2-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 11:10:00.174862       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-2-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
I1027 11:10:00.174875       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 11:10:00.259092       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-2-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 11:10:00.259115       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-2-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
I1027 11:10:00.259128       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 11:10:00.311269       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-0-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 11:10:00.311291       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-0-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
I1027 11:10:00.311300       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 11:10:00.334172       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-0-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 11:10:00.334193       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-0-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
I1027 11:10:00.334204       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 11:10:00.420310       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-0-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 11:10:00.420332       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-3-0:9000. DB::Exception: Received from chi-clickhouse-sandbox-ch-0-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
I1027 11:10:00.420341       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
E1027 11:10:00.466126       1 announcer.go:267] connection.go:128:QueryContext():FAILED Query(http://***:***@chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local:8123/) clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-2-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
 for SQL: SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
W1027 11:10:00.466147       1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local] skip to next. err: clickhouse: Code: 516. DB::Exception: Received from chi-clickhouse-sandbox-ch-2-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED) (version 21.10.2.15 (official build))
E1027 11:10:00.466163       1 announcer.go:267] cluster.go:95:QueryAny():FAILED to run query on all hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:10:00.466192       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1027 11:10:00.466210       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1027 11:10:00.466226       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Creating distributed objects at 4-0: [default]
I1027 11:10:00.466241       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:
[CREATE DATABASE IF NOT EXISTS "default"]
I1027 11:10:00.467365       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 4 cluster sandbox-ch
I1027 11:10:00.476052       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 11:10:00.839021       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 4-0 completed
I1027 11:10:00.855108       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.202  0xc000864840 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:42e19715-843d-4a04-bd55-5a24a4453e13,APIVersion:,ResourceVersion:4120473,FieldPath:,}} {10.20.101.218  0xc000864850 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:5172ee09-1c7f-438b-8a2c-9ce14c0b9583,APIVersion:,ResourceVersion:4118977,FieldPath:,}} {10.20.2.212  0xc000864860 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:f9b592a2-ddc9-48ba-8bd4-a61d9947abad,APIVersion:,ResourceVersion:4118275,FieldPath:,}} {10.20.2.246  0xc000864870 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:df7ce43b-65c3-41aa-8c66-a2e93f018b49,APIVersion:,ResourceVersion:4119737,FieldPath:,}} {10.20.51.112  0xc000864880 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:6a3624dd-570f-4b88-bcb2-17477a81746f,APIVersion:,ResourceVersion:4117551,FieldPath:,}} {10.20.51.126  0xc000864890 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:0a1bc85c-a0c9-4b74-bd44-1abfd0edb303,APIVersion:,ResourceVersion:4115885,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
I1027 11:10:01.059514       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-sandbox-ch-5
I1027 11:10:01.059564       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 5-0 started
I1027 11:10:01.263117       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-5-0 use custom template clickhouse-v20.6
I1027 11:10:01.263202       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-5-0
W1027 11:10:01.263221       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:10:01.263229       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:10:01.263237       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:10:01.263249       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:10:01.265145       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-sandbox-ch-5-0
I1027 11:10:01.265167       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 5 cluster sandbox-ch
I1027 11:10:01.682673       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:10:01.684262       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/5-0-OK
I1027 11:10:01.829425       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-5-0
I1027 11:10:02.074358       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-5-0 use custom template clickhouse-v20.6
I1027 11:10:02.074441       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-5-0
W1027 11:10:02.074460       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:10:02.074474       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:10:02.074485       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:10:02.074496       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:10:02.076200       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-5-0
I1027 11:10:02.076252       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-5-0) - started
I1027 11:10:02.268863       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Wait for ConfigMap propagation for 1m29.805430866s 194.569134ms/1m30s
I1027 11:11:32.104340       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 20=>21
I1027 11:11:37.123267       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-5-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56 UpdateRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56
I1027 11:11:42.138448       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-5-0:OK  :ObservedGeneration:21 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56 UpdateRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56
I1027 11:11:42.171632       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-5-0) - completed
I1027 11:11:42.220195       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-sandbox-ch-5-0 use custom template clickhouse-v20.6
I1027 11:11:42.220277       1 announcer.go:218] setupLogContainer():clickhouse/clickhouse:add log container for statefulSet chi-clickhouse-sandbox-ch-5-0
W1027 11:11:42.220293       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
W1027 11:11:42.220303       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse volumeClaimTemplateName:default-volume-claim already used
I1027 11:11:42.220311       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse-log mounted default-volume-claim on /var/lib/clickhouse
W1027 11:11:42.220321       1 announcer.go:245] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-sandbox-ch-5-0 container:clickhouse-log volumeClaimTemplateName:default-volume-claim already used
I1027 11:11:42.222169       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-5-0
I1027 11:11:42.222241       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-5-0) - started
I1027 11:11:42.260066       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m40.185772911s/1m30s
I1027 11:11:42.276630       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 21=>22
I1027 11:12:42.706517       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-5-0:WAIT:ObservedGeneration:22 Replicas:1 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56 UpdateRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56
I1027 11:12:47.742989       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-5-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56 UpdateRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56
I1027 11:13:07.899379       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-sandbox-ch-5-0:OK  :ObservedGeneration:22 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56 UpdateRevision:chi-clickhouse-sandbox-ch-5-0-84c564bc56
I1027 11:13:07.936962       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-sandbox-ch-5-0) - completed
I1027 11:13:08.085084       1 announcer.go:218] CreateServiceHost():clickhouse/clickhouse:clickhouse/chi-clickhouse-sandbox-ch-5-0 for Set chi-clickhouse-sandbox-ch-5-0
I1027 11:13:08.285252       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/chi-clickhouse-sandbox-ch-5-0
I1027 11:13:08.513228       1 announcer.go:218] migrateTables():clickhouse/clickhouse:Adding tables on shard/host:5/0 cluster:sandbox-ch
I1027 11:13:08.725786       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Migrating schema objects to host 5-0
I1027 11:13:08.725838       1 announcer.go:220] getCreateReplicaObjects():clickhouse/clickhouse:Single replica in a shard. Nothing to create a schema from.
I1027 11:13:08.725928       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:Extracting distributed table definitions from hosts [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:13:08.725961       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch dbs list
I1027 11:13:08.725971       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:dbs sql
SELECT 
        DISTINCT database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS create_db_query
FROM 
(
        SELECT DISTINCT arrayJoin([database, extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) database
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
)
I1027 11:13:08.726081       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:13:08.751322       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names1:
I1027 11:13:08.751345       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names1: default
I1027 11:13:08.751358       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql1:
I1027 11:13:08.751370       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql1: CREATE DATABASE IF NOT EXISTS "default"
I1027 11:13:08.751383       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:fetch table list
I1027 11:13:08.751395       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:tbl sql
SELECT DISTINCT 
        concat(database, '.', name) as name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database,
                name,
                create_table_query,
                2 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS name,
                t.create_table_query,
                1 AS order
        FROM clusterAllReplicas('sandbox-ch', system.tables) tables
        LEFT JOIN 
        (
                SELECT 
                        DISTINCT database, 
                        name, 
                        create_table_query 
                FROM clusterAllReplicas('sandbox-ch', system.tables)
                SETTINGS skip_unavailable_shards = 1
        ) t 
        USING (database, name)
        WHERE engine = 'Distributed' AND t.create_table_query != ''
        SETTINGS skip_unavailable_shards = 1
) tables
ORDER BY order
I1027 11:13:08.751504       1 announcer.go:218] Run query on: chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-sandbox-ch-0-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-1-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-2-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-3-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-4-0.clickhouse.svc.cluster.local chi-clickhouse-sandbox-ch-5-0.clickhouse.svc.cluster.local]
I1027 11:13:08.807964       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:names2:
I1027 11:13:08.807997       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations_local
I1027 11:13:08.808019       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days_local
I1027 11:13:08.808043       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations_local
I1027 11:13:08.808064       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user_local
I1027 11:13:08.808088       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1_local
I1027 11:13:08.808110       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_event_aggregations
I1027 11:13:08.808129       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_in_app_purchase_event_days
I1027 11:13:08.808148       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.campaign_postback_aggregations
I1027 11:13:08.808170       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user
I1027 11:13:08.808195       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:names2: default.user_event_aggregations_user1
I1027 11:13:08.808215       1 announcer.go:220] getCreateDistributedObjects():clickhouse/clickhouse:sql2:
I1027 11:13:08.808236       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192
I1027 11:13:08.808266       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192
I1027 11:13:08.808291       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192
I1027 11:13:08.808322       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:13:08.808351       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192
I1027 11:13:08.808377       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand())
I1027 11:13:08.808402       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand())
I1027 11:13:08.808427       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand())
I1027 11:13:08.808452       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand())
I1027 11:13:08.808477       1 announcer.go:218] getCreateDistributedObjects():clickhouse/clickhouse:sql2: CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())
I1027 11:13:08.808500       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:Creating distributed objects at 5-0: [default default.campaign_in_app_event_aggregations_local default.campaign_in_app_purchase_event_days_local default.campaign_postback_aggregations_local default.user_event_aggregations_user_local default.user_event_aggregations_user1_local default.campaign_in_app_event_aggregations default.campaign_in_app_purchase_event_days default.campaign_postback_aggregations default.user_event_aggregations_user default.user_event_aggregations_user1]
I1027 11:13:08.808532       1 announcer.go:218] HostCreateTables():clickhouse/clickhouse:
[CREATE DATABASE IF NOT EXISTS "default" CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations_local (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_event_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (purchase_value, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days_local (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_in_app_purchase_event_days_local', '{replica}') PARTITION BY toYYYYMM(postback_date) ORDER BY (dx, sdk_hash, postback_date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations_local (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/campaign_postback_aggregations_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (targeting_group_uuid, app_id, partner_uuid, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1_local (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/default/user_event_aggregations_user1_local', '{replica}') PARTITION BY toYYYYMM(date) ORDER BY (message, date) SETTINGS index_granularity = 8192 CREATE TABLE IF NOT EXISTS default.campaign_in_app_event_aggregations (`date` Date, `type` LowCardinality(String), `targeting_group_uuid` String, `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `country` LowCardinality(String), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `event_count` Int64 CODEC(T64), `purchase_value` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_event_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_in_app_purchase_event_days (`postback_date` Date, `targeting_group_uuid` String, `country` LowCardinality(String), `sdk_hash` LowCardinality(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `apk_version` LowCardinality(Nullable(String)), `dx` Int32 CODEC(T64), `purchase_sum` Float64, `purchase_count` Int64 CODEC(T64), `partner_uuid` LowCardinality(String), `campaign_uuid` String, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Nullable(Int32) CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_in_app_purchase_event_days_local', rand()) CREATE TABLE IF NOT EXISTS default.campaign_postback_aggregations (`sdk_hash` LowCardinality(String), `date` Date, `country` LowCardinality(String), `campaign_uuid` String, `targeting_group_uuid` String, `creative_set_uuid` String, `partner_uuid` LowCardinality(String), `creative_set_ab_test_group_id` Nullable(Int32) CODEC(T64), `postback_type` LowCardinality(String), `app_id` String, `ad_format` LowCardinality(String), `apk_version` LowCardinality(Nullable(String)), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `postback_count` Int64 CODEC(T64), `revenue` Float64, `created_at` DateTime, `testgroup` Nullable(Int32) CODEC(T64), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'campaign_postback_aggregations_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` Nullable(String), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user_local', rand()) CREATE TABLE IF NOT EXISTS default.user_event_aggregations_user1 (`sdk_hash` LowCardinality(String), `publisher_uuid` LowCardinality(String), `date` Date, `testgroup` LowCardinality(Nullable(String)), `country` LowCardinality(String), `message` LowCardinality(String), `apk_version` Nullable(String), `sdk_sub_id1` Nullable(String), `sdk_sub_id2` Nullable(String), `experiment_uuid` LowCardinality(Nullable(String)), `count` Int64 CODEC(T64), `created_at` DateTime, `age` Nullable(Int32) CODEC(T64), `gender` LowCardinality(Nullable(String)), `os_version` LowCardinality(Nullable(String)), `sdk_version` Int32 CODEC(T64), `ua_network` Nullable(String), `ua_channel` Nullable(String), `ua_sub_publisher_cleartext` Nullable(String), `ua_sub_publisher_encrypted` Nullable(String), `placement` Nullable(String)) ENGINE = Distributed('{cluster}', 'default', 'user_event_aggregations_user1_local', rand())]
I1027 11:13:08.826736       1 announcer.go:218] includeHost():clickhouse/clickhouse:Include into cluster host 0 shard 5 cluster sandbox-ch
I1027 11:13:08.884387       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 11:13:09.495744       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 5-0 completed
I1027 11:13:09.510578       1 announcer.go:218] clickhouse/service-clickhouse:endpointsInformer.UpdateFunc: IP ASSIGNED [{[{10.20.101.202  0xc000283540 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-4-0-0,UID:42e19715-843d-4a04-bd55-5a24a4453e13,APIVersion:,ResourceVersion:4120473,FieldPath:,}} {10.20.101.218  0xc000283550 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-2-0-0,UID:5172ee09-1c7f-438b-8a2c-9ce14c0b9583,APIVersion:,ResourceVersion:4118977,FieldPath:,}} {10.20.2.212  0xc000283560 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-1-0-0,UID:f9b592a2-ddc9-48ba-8bd4-a61d9947abad,APIVersion:,ResourceVersion:4118275,FieldPath:,}} {10.20.2.246  0xc000283570 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-3-0-0,UID:df7ce43b-65c3-41aa-8c66-a2e93f018b49,APIVersion:,ResourceVersion:4119737,FieldPath:,}} {10.20.51.112  0xc000283580 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-0-0-0,UID:6a3624dd-570f-4b88-bcb2-17477a81746f,APIVersion:,ResourceVersion:4117551,FieldPath:,}} {10.20.51.27  0xc000283590 &ObjectReference{Kind:Pod,Namespace:clickhouse,Name:chi-clickhouse-sandbox-ch-5-0-0,UID:b7b08ffe-1923-48de-890d-6c4813866849,APIVersion:,ResourceVersion:4121232,FieldPath:,}}] [] [{http 8123 TCP <nil>} {tcp 9000 TCP <nil>}]}]
I1027 11:13:09.881612       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 11:13:10.111881       1 announcer.go:220] updateCHI():clickhouse/clickhouse:remove items scheduled for deletion
I1027 11:13:11.082851       1 announcer.go:218] clear():clickhouse/clickhouse:Reconciled objects:
Service: clickhouse/service-clickhouse
Service: clickhouse/chi-clickhouse-sandbox-ch-0-0
Service: clickhouse/chi-clickhouse-sandbox-ch-1-0
Service: clickhouse/chi-clickhouse-sandbox-ch-2-0
Service: clickhouse/chi-clickhouse-sandbox-ch-3-0
Service: clickhouse/chi-clickhouse-sandbox-ch-4-0
Service: clickhouse/chi-clickhouse-sandbox-ch-5-0
ConfigMap: clickhouse/chi-clickhouse-common-configd
ConfigMap: clickhouse/chi-clickhouse-common-usersd
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-0-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-1-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-2-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-3-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-4-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-5-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-0-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-1-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-2-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-3-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-4-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-5-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-0-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-1-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-2-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-3-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-4-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-5-0-0
I1027 11:13:11.082925       1 announcer.go:218] clear():clickhouse/clickhouse:Existing objects:
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-0-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-1-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-2-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-3-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-4-0
StatefulSet: clickhouse/chi-clickhouse-sandbox-ch-5-0
ConfigMap: clickhouse/chi-clickhouse-common-configd
ConfigMap: clickhouse/chi-clickhouse-common-usersd
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-0-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-1-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-2-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-3-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-4-0
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-sandbox-ch-5-0
Service: clickhouse/chi-clickhouse-sandbox-ch-0-0
Service: clickhouse/chi-clickhouse-sandbox-ch-1-0
Service: clickhouse/chi-clickhouse-sandbox-ch-2-0
Service: clickhouse/chi-clickhouse-sandbox-ch-3-0
Service: clickhouse/chi-clickhouse-sandbox-ch-4-0
Service: clickhouse/chi-clickhouse-sandbox-ch-5-0
Service: clickhouse/service-clickhouse
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-0-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-1-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-2-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-3-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-4-0-0
PVC: clickhouse/default-volume-claim-chi-clickhouse-sandbox-ch-5-0-0
PV: /pvc-001f70bb-e71a-4e66-87e1-8947effab153
PV: /pvc-11f7a67e-5a9c-4b4e-b63f-9c5c42b27e40
PV: /pvc-13a97c63-8727-48ad-976d-a20a29c6ec3f
PV: /pvc-31a3b46c-8651-4708-ae2c-7d46b7c8b2c2
PV: /pvc-565f1105-c06d-41f2-94e5-5092882160a5
PV: /pvc-ae26ab10-1074-404e-b844-e320d75ce5da
I1027 11:13:11.082984       1 announcer.go:218] clear():clickhouse/clickhouse:Non-reconciled objects:
PV: /pvc-001f70bb-e71a-4e66-87e1-8947effab153
PV: /pvc-11f7a67e-5a9c-4b4e-b63f-9c5c42b27e40
PV: /pvc-13a97c63-8727-48ad-976d-a20a29c6ec3f
PV: /pvc-31a3b46c-8651-4708-ae2c-7d46b7c8b2c2
PV: /pvc-565f1105-c06d-41f2-94e5-5092882160a5
PV: /pvc-ae26ab10-1074-404e-b844-e320d75ce5da
I1027 11:13:11.083007       1 announcer.go:220] clear():clickhouse/clickhouse:remove items scheduled for deletion
I1027 11:13:11.315875       1 announcer.go:220] includeStopped():clickhouse/clickhouse:add CHI to monitoring
I1027 11:13:11.544644       1 announcer.go:220] markReconcileComplete():clickhouse/clickhouse:reconcile completed

[hodgesrm]

This problem has been replicated in an additional environment. I will add relevant YAML/log files when available.

[alex-zaitsev]

@FarhadF , do you set any labels outside of operator? This log message is important:

I1026 16:24:55.343085 1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-sandbox-ch-0-0

It means that operator discovered that labels that it expects on stateful set are different from the actual ones. That triggers re-creation. It may seem not needed, but podTemplate labels can not be changed without re-create, for example.

Also consider using 0.16.1 or latest 0.17.0. We constantly improve quality of an operator.

[alex-zaitsev]

@FarhadF , the restart is caused by

spec:
  ...
  restart: "RollingUpdate"

This is a setting the FORCES a restart. We use it in order to restart ClickHouse cluster on demand without service interruption.

Please remove it, that should help.