Externally generated config (named_collections.xml) break cluster

[Hubbitus]

Hello.

We use your Operator to manage Clickhouse cluster. Thank you for the product.

We are now trying to externally manage named_collections.xml file. The idea is simple - that should come from an external mechanism of access control. For the simplicity, let it be just Vault secret (we use ExternalSecret on ArgoCD deploy).

We then put it into file /etc/clickhouse-server/conf.d/custom-named-collections.xml in the pod:

    files:
      config.d/named-collections.xml:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secret-named-collections-xml
            key: custom-named-collections.xml

Meantime, clickhouse server pod goes into crashloop. I've inspected configs and see:

$ ll config.d
total 4
lrwxrwxrwx. 1 pasha pasha 35 Jul 12 21:41 custom-named-collections.xml -> ..data/custom-named-collections.xml

So, there are no other config files, normally generated on start (I assume generated by operator). Is it bug or I incorrectly try use it?

[Slach]

Unfortunatelly

  files:
      config.d/named-collections.xml:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secret-named-collections-xml
            key: custom-named-collections.xml

will place file in /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml

to allow use it as workaround, you need to add

spec:
  configuration:
    files:
      config.d/include_from.xml: |
        <clickhouse>
           <include_from>/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml</include_from>
        </clickhouse> 
      config.d/override_named_collection.xml: |
        <clickhouse>
          <named_collestions><collection_name incl="named_collections/collection_name"/></named_collection>
        </clickhouse>

[Hubbitus]

Hello. @Slach, thank you very much for the so quick response!

... will place file in /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml

Actually no.

I've used:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "gid"
spec:
  configuration:
    clusters:
      - name: "gid"
        layout:
          shardsCount: 1
          replicasCount: 2
        templates:
          podTemplate: clickhouse-stable
          volumeClaimTemplate: default-volume-claim
    files:
      config.d/named-collections.xml:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secret-named-collections-xml
            key: custom-named-collections.xml
      # To allow use named collections from our custom file (see https://github.com/Altinity/clickhouse-operator/issues/1456)
      config.d/include_from.xml: |
        <clickhouse>
           <include_from>/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml</include_from>
        </clickhouse>
      config.d/override_named_collection.xml: |
        <clickhouse>
          <named_collections><collection_name incl="named_collections/collection_name"/></named_collections>
        </clickhouse>
...

Please look:

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/config.d/
total 0
lrwxrwxrwx 1 root root 35 Jul 12 19:17 custom-named-collections.xml -> ..data/custom-named-collections.xml

And off course server did not start and even not print errors because is not configured properly:

$ kubectl logs chi-gid-gid-0-0-0
ClickHouse Database directory appears to contain a database; Skipping initialization
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-hostname-ports.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-macros.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-zookeeper.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/custom-named-collections.xml'.
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log

Even not all normal configs generated:

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/conf.d/
total 0
lrwxrwxrwx 1 root root 40 Jul 12 19:17 chop-generated-hostname-ports.xml -> ..data/chop-generated-hostname-ports.xml
lrwxrwxrwx 1 root root 32 Jul 12 19:17 chop-generated-macros.xml -> ..data/chop-generated-macros.xml
lrwxrwxrwx 1 root root 35 Jul 12 19:17 chop-generated-zookeeper.xml -> ..data/chop-generated-zookeeper.xml

[Slach]

which clickhouse-operator version do you use?

[Slach]

kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" -l app=clickhouse-operator

[Hubbitus]

Quite old. Unfortunately, I have no permissions to do so now. Will return later.

[Hubbitus]

Hello.

kub_dev get pods --all-namespaces -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" -l app=clickhouse-operator                                                                                                                         
altinity/clickhouse-operator:0.20.3 altinity/metrics-exporter:0.20.3

[Hubbitus]

We had updated operator to the latest version:

kub_dev get pods --all-namespaces -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" -l app=clickhouse-operator                                                                                                                         
altinity/clickhouse-operator:0.23.6 altinity/metrics-exporter:0.23.6

But still I see same file created and nothing more:

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/config.d/
total 0
lrwxrwxrwx 1 root root 35 Jul 15 15:52 custom-named-collections.xml -> ..data/custom-named-collections.xml

[Hubbitus]

@Slach, could you please help how I can debug operator? How to understand why other configs does not created?

[Slach]

Check logs from clickhouse-operator deployment

Could you share result of following command?

kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/secrets.d/

[Hubbitus]

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/secrets.d/
total 4
drwxr-xr-x 3 root root 4096 Jul 16 12:45 config.d
$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/secrets.d/config.d/
total 4
drwxr-xr-x 3 root root 4096 Jul 16 12:45 named-collections.xml

[Hubbitus]

Hmm, and that is also looking strange for me (seeing named-collections.xml as directory for example):

kubectl exec chi-gid-gid-0-0-0 -- cat /etc/clickhouse-server/secrets.d/config.d/named-collections.xml                                                                                                      
cat: /etc/clickhouse-server/secrets.d/config.d/named-collections.xml: Is a directory
command terminated with exit code 1

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/secrets.d/config.d/named-collections.xml                                                                                   
total 0
drwxrwxrwt 3 root root 100 Jul 16 12:45 clickhouse-secret-named-collections-xml

$ kubectl exec chi-gid-gid-0-0-0 -- cat /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml
cat: /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml: Is a directory
command terminated with exit code 1

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml
total 0
lrwxrwxrwx 1 root root 35 Jul 16 12:45 custom-named-collections.xml -> ..data/custom-named-collections.xml

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/
total 0
lrwxrwxrwx 1 root root 35 Jul 16 12:45 custom-named-collections.xml -> ..data/custom-named-collections.xml

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml
lrwxrwxrwx 1 root root 35 Jul 16 12:45 /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml -> ..data/custom-named-collections.xml

$ kubectl exec chi-gid-gid-0-0-0 -- cat /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml
...

File /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml actually our generated file.

[Slach]

look to /var/lib/clickhouse/preprocessed_configs

maybe

 <named_collections><collection_name incl="named_collections/collection_name"/></named_collections>

doesn't work

[Slach]

maybe you could achieve the same results with following manifest

kind: ClickHouseInstallation
spec:
  templates:
    podTemplates:
      - name: pod-with-secretMounts
        spec:
          containers:
          - name: clickhouse
            image: clickhouse/clickhouse-server:latest
            volumeMounts:
            - name:  clickhouse-secret-named-collections-xml
              mountPath: /etc/clickhouse-server/config.d/  

        volumes:
          - name: clickhouse-secret-named-collections-xml
            secret:    
              secretName: clickhouse-secret-named-collections-xml
              items: 
                - key: custom-named-collections.xml
                  path: custom-named-collections.xml

[Hubbitus]

Sorry, I did not understand your suggestion.

Now I roughly have (some unrelated details like users, permissions and options omitted):

---
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "gid"
spec:
  configuration:
    users:
      ...
    clusters:
      - name: "gid"
        layout:
          shardsCount: 1
          replicasCount: 2
        templates:
          podTemplate: clickhouse-stable
          volumeClaimTemplate: default-volume-claim
    files:
      config.d/named-collections.xml:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secret-named-collections-xml
            key: custom-named-collections.xml
      # To allow use named collections from our custom file (see https://github.com/Altinity/clickhouse-operator/issues/1456)
      config.d/include_from.xml: |
        <clickhouse>
           <include_from>/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml</include_from>
        </clickhouse>
      config.d/override_named_collection.xml: |
        <clickhouse>
          <named_collections><collection_name incl="named_collections/collection_name"/></named_collections>
        </clickhouse>
  templates:
    podTemplates:
      - name: clickhouse-stable
        spec:
          containers:
            - name: clickhouse
              image: ... #v24.2.1.2248
              volumeMounts:
                - name: kafkassl
                  mountPath: /var/private/ssl
                - name: named-collections
                  mountPath: /etc/clickhouse-server/config.d/
          volumes:
            - name: kafkassl
              secret:
                secretName: bi-clickhouse
            - name: named-collections
              secret:
                secretName: clickhouse-secret-named-collections-xml
          imagePullSecrets:
            - name: registrysecret
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: clickhouse-config-named-collections-xml-tpl
data:
  custom-named-collections.xml: |
    <clickhouse>
        <named_collections>
            {{- range $element := . }}
            {{- $k := $element | fromYaml }}
            <{{ $k.DB_COLLECTION }}>
                <user>{{ $k.DB_USERNAME }}</user>
                <password>{{ $k.DB_PASSWORD }}</password>
                <host>{{ $k.DB_HOST }}</host>
                <port>{{ $k.DB_PORT}}</port>
                <database>{{ $k.DB_NAME }}</database>
                <schema>{{ $k.DB_SCHEMA_NAME }}</schema>
            </{{ $k.DB_COLLECTION }}>
            {{ end }}
        </named_collections>
    </clickhouse>
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: clickhouse-secret-named-collections-xml
  namespace: gidplatform-dev
  labels:
    app.kubernetes.io/managed-by: ArgoCD
spec:
  refreshInterval: "30m"
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-backend-ansible
  target:
    name: clickhouse-secret-named-collections-xml
    template:
      engineVersion: v2
      templateFrom:
       - target: Data
         configMap:
           name: clickhouse-config-named-collections-xml-tpl
           items:
             - key: custom-named-collections.xml
               templateAs: Values
  dataFrom:
    - find:
        path: databases/GID_DM_CH_dev/instance/collections
        name:
          regexp: ".*"

Should I drop suggested early:

      config.d/include_from.xml: |
        <clickhouse>
           <include_from>/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml</include_from>
        </clickhouse>
      config.d/override_named_collection.xml: |
        <clickhouse>
          <named_collections><collection_name incl="named_collections/collection_name"/></named_collections>
        </clickhouse>

and replace something else?

[Hubbitus]

@Slach, could you please help me?

[Slach]

ok. let's try to figure out with your current manifest step by step and if will fail, then will try approach described in https://github.com/Altinity/clickhouse-operator/issues/1456#issuecomment-2230876227

is secrets.d/**.xml file exists inside your pod?

ls -la /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml return something?

[Hubbitus]

ok. let's try to figure out with your current manifest step by step

Thank you very much!

and if will fail, then will try approach described in https://github.com/Altinity/clickhouse-operator/issues/1456#issuecomment-2230876227

I do not argue to try it right now, but looks like do not understand what should I change exactly

is secrets.d/**.xml file exists inside your pod? Yes.

I've posted details in https://github.com/Altinity/clickhouse-operator/issues/1456#issuecomment-2230831288

ls -la /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml return something?

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server//etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml
ls: cannot access '/etc/clickhouse-server//etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml': No such file or directory
command terminated with exit code 2

$ kubectl exec chi-gid-gid-0-0-0 -- find / -name custom-named-collections.xml
/etc/clickhouse-server/config.d/custom-named-collections.xml
/etc/clickhouse-server/config.d/..2024_07_18_17_03_58.168711024/custom-named-collections.xml
/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml
/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/..2024_07_18_17_03_58.560993557/custom-named-collections.xml

$ kubectl exec chi-gid-gid-0-0-0 -- ls -l /etc/clickhouse-server/config.d/custom-named-collections.xml /etc/clickhouse-server/config.d/..2024_07_18_17_03_58.168711024/custom-named-collections.xml /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/..2024_07_18_17_03_58.560993557/custom-named-collections.xml
-rw-r--r-- 1 root root 10376 Jul 18 17:03 /etc/clickhouse-server/config.d/..2024_07_18_17_03_58.168711024/custom-named-collections.xml
lrwxrwxrwx 1 root root    35 Jul 18 17:03 /etc/clickhouse-server/config.d/custom-named-collections.xml -> ..data/custom-named-collections.xml
-rw-r--r-- 1 root root 10376 Jul 18 17:03 /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/..2024_07_18_17_03_58.560993557/custom-named-collections.xml
lrwxrwxrwx 1 root root    35 Jul 18 17:03 /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml -> ..data/custom-named-collections.xml

[b2kos]

maybe you could achieve the same results with following manifest

kind: ClickHouseInstallation
spec:
 templates:
 podTemplates:
 - name: pod-with-secretMounts
        spec:
 containers:
 - name: clickhouse
   image: clickhouse/clickhouse-server:latest
   volumeMounts:
   - name: clickhouse-secret-named-collections-xml
     mountPath: /etc/clickhouse-server/config.d/ 

   volumes:
   - name: clickhouse-secret-named-collections-xml
     secret: 
       secretName: clickhouse-secret-named-collections-xml
       items: 
       - key: custom-named-collections.xml
         path: custom-named-collections.xml

This will cause a reboot

ClickHouse Database directory appears to contain a database; Skipping initialization
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-hostname-ports.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-macros.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-zookeeper.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/custom-named-collections.xml'.
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log

[Slach]

ls: cannot access '/etc/clickhouse-server//etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml': No such file or directory command terminated with exit code 2

this is wrong path (double /etc/clickhouse-server/ in prefix) I requested /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml and

you found /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml

you need to use it

for

<include_from>/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/clickhouse-secret-named-collections-xml/custom-named-collections.xml</include_from>

ok, you decided to use better approach with volumeMounts

following final manifest should works

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "gid"
spec:
  configuration:
    users:
      ...
    clusters:
      - name: "gid"
        layout:
          shardsCount: 1
          replicasCount: 2
        templates:
          podTemplate: clickhouse-stable
          volumeClaimTemplate: default-volume-claim
  templates:
    podTemplates:
      - name: clickhouse-stable
        spec:
          containers:
            - name: clickhouse
              image: registry.gid.team/devops/base_image:a4d17dd27005e64044ad4ce2ed1192cefd992537a51d4be9042c653f-1710314190650 #v24.2.1.2248
              volumeMounts:
                - name: kafkassl
                  mountPath: /var/private/ssl
                - name: merged-config
                  mountPath: /etc/clickhouse-server/config.d/
          volumes:
            - name: kafkassl
              secret:
                secretName: bi-clickhouse
            - name: merged-config
              projected:
                sources:
                - configMap:
                    name: chi-git-common-configd 
                - secret:
                    secretName: clickhouse-secret-named-collections-xml
          imagePullSecrets:
            - name: registrysecret

look to projected

after apply please share

ls -la /etc/clickhouse-server/config.d/

[Hubbitus]

Hello, @Slach. Thank you for the answer.

this is wrong path (double /etc/clickhouse-server/ in prefix) I requested /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml

I'm sorry for the mistake. And yes, file have been found.

Solution with projected volume mount:

following final manifest should works ...

looks better on first look. At least, server does not stuck on first logs of loading configs.

But fails later:

Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-hostname-ports.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-macros.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-zookeeper.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-01-listen.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-02-logger.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-03-query_log.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-04-part_log.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-05-trace_log.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/chop-generated-remote_servers.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/include_from.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/macros.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/operator_metrics.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/override_named_collection.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/z_log_disable.xml'.
Including configuration file '/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml'.
Poco::Exception. Code: 1000, e.code() = 0, Exception: Failed to preprocess config '/etc/clickhouse-server/config.xml': File not found: /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml, Stack trace (when copying this message,
always include the lines below):
0. DB::ConfigProcessor::processConfig(bool*, zkutil::ZooKeeperNodeCache*, std::shared_ptr<Poco::Event> const&) @ 0x000000001369a5ca
1. DB::ConfigProcessor::loadConfig(bool) @ 0x000000001369aff4
2. BaseDaemon::initialize(Poco::Util::Application&) @ 0x000000000d243f00
3. DB::Server::initialize(Poco::Util::Application&) @ 0x000000000d0baed8
4. Poco::Util::Application::run() @ 0x00000000158bbd9a
5. DB::Server::run() @ 0x000000000d0bac91
6. Poco::Util::ServerApplication::run(int, char**) @ 0x00000000158c4c19
7. mainEntryClickHouseServer(int, char**) @ 0x000000000d0b7d4a
8. main @ 0x00000000078c47d8
9. ? @ 0x0000717c9dff3083
10. _start @ 0x00000000060c312e
 (version 24.2.1.2248 (official build))

after apply please share ls -la /etc/clickhouse-server/config.d/

Sorry, pod died too fast, I can't do ls.

[Slach]

File not found: /etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml,

do you still use files with secretKeyRef ?

share your latest manifest kubectl get chi -n gidplatform-dev gid -o yaml

[Hubbitus]

do you still use files with secretKeyRef ?

Yes, I have:

      # To allow use named collections from our custom file (see https://github.com/Altinity/clickhouse-operator/issues/1456)
      config.d/include_from.xml: |
        <clickhouse>
           <include_from>/etc/clickhouse-server/secrets.d/config.d/named-collections.xml/custom-named-collections.xml</include_from>
        </clickhouse>
      config.d/override_named_collection.xml: |
        <clickhouse>
          <named_collections><collection_name incl="named_collections/collection_name"/></named_collections>
        </clickhouse>

Should it be removed?

I've tried. It looks much better, but got error named collection is absent:

2024.07.23 12:25:27.485487 [ 697 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 669. DB::Exception: There is no named collection `bonus_db`: Cannot attach table `datamart`.`bonus__public__banner` from metadata file /var/lib/clickhouse/store/1ef/1efccde9-1207-4192-840e-0843fa5a4b77/bonus__public__banner.sql from query ATTACH TABLE datamart.bonus__public__banner UUID '50b6ed9c-cd5e-42e1-803c-e0d85d0a518e' (`id` Nullable(UUID), `ogon_banner_id` Nullable(Int64), `ogon_partner_id` Nullable(Int64), `position` Nullable(Int32), `title` Nullable(String), `description` Nullable(String), `mobile_picture_url` Nullable(String), `tablet_picture_url` Nullable(String), `web_picture_url` Nullable(String), `background_color` Nullable(String), `expiration_at` Nullable(DateTime64(6, 'UTC')), `ogon_creation_date` Nullable(DateTime64(6, 'UTC')), `ogon_modification_date` Nullable(DateTime64(6, 'UTC')), `creation_date` Nullable(DateTime64(6, 'UTC')), `modification_date` Nullable(DateTime64(6, 'UTC')), `link_type` Nullable(String), `author_id` Nullable(UUID), `editor_id` Nullable(UUID), `ogon_promotion_view_id` Nullable(String), `full_picture_url_path` Nullable(String), `partner_id` Nullable(UUID), `name` Nullable(String), `promotion_id` Nullable(UUID), `is_published` Nullable(Bool), `picture_id` Nullable(UUID)) ENGINE = PostgreSQL(bonus_db, `table` = 'banner', schema = 'public') COMMENT 'FOREIGN TABLE. Описание актуальных баннеров. https://confluence.gid.team/x/e-qhBQ'. (NAMED_COLLECTION_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

Indeed, I do not see such config file:

$ ll etc/clickhouse-server/config.d
total 40
lrwxrwxrwx. 1 pasha pasha 34 Jul 23 15:25 01-clickhouse-01-listen.xml -> ..data/01-clickhouse-01-listen.xml
lrwxrwxrwx. 1 pasha pasha 34 Jul 23 15:25 01-clickhouse-02-logger.xml -> ..data/01-clickhouse-02-logger.xml
lrwxrwxrwx. 1 pasha pasha 37 Jul 23 15:25 01-clickhouse-03-query_log.xml -> ..data/01-clickhouse-03-query_log.xml
lrwxrwxrwx. 1 pasha pasha 36 Jul 23 15:25 01-clickhouse-04-part_log.xml -> ..data/01-clickhouse-04-part_log.xml
lrwxrwxrwx. 1 pasha pasha 37 Jul 23 15:25 01-clickhouse-05-trace_log.xml -> ..data/01-clickhouse-05-trace_log.xml
lrwxrwxrwx. 1 pasha pasha 40 Jul 23 15:25 chop-generated-remote_servers.xml -> ..data/chop-generated-remote_servers.xml
lrwxrwxrwx. 1 pasha pasha 17 Jul 23 15:25 config.xml -> ..data/config.xml
lrwxrwxrwx. 1 pasha pasha 17 Jul 23 15:25 macros.xml -> ..data/macros.xml
lrwxrwxrwx. 1 pasha pasha 27 Jul 23 15:25 operator_metrics.xml -> ..data/operator_metrics.xml
lrwxrwxrwx. 1 pasha pasha 24 Jul 23 15:25 z_log_disable.xml -> ..data/z_log_disable.xml

share your latest manifest kubectl get chi -n gidplatform-dev gid -o yaml

chi_gidplatform-dev.yaml.gz (slightly truncated and obfuscated to conform our security engineers requirements)

[Slach]

Should it be removed?

yes

remove

      config.d/named-collections.xml:
        valueFrom:
          secretKeyRef:
            key: custom-named-collections.xml
            name: clickhouse-secret-named-collections-xml

replace

          - mountPath: /etc/clickhouse-server/config.d/
            name: named-collections

to

          - mountPath: /etc/clickhouse-server/config.d/
            name: merged-config

[Hubbitus]

Thanks.

Still the configuration error like (looks more dramatic):

2024.07.23 16:12:45.876594 [ 47 ] {} <Debug> HTTP-Session: 61acc357-35dc-4ea6-ad71-e1ee7834a6b3 Logout, user_id: b17d2bea-57c0-071d-d96d-87afe40b887
2024.07.23 16:12:45.900653 [ 691 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/users.xml'
2024.07.23 16:12:45.900729 [ 691 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/users.xml'.
2024.07.23 16:12:45.901970 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-hostname-ports.xml'.
2024.07.23 16:12:45.902238 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-macros.xml'.
2024.07.23 16:12:45.902500 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-zookeeper.xml'.
2024.07.23 16:12:45.902797 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/01-clickhouse-operator-profile.xml'.
2024.07.23 16:12:45.903089 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/02-clickhouse-default-profile.xml'.
2024.07.23 16:12:45.903484 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/chop-generated-users.xml'.
2024.07.23 16:12:45.904021 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/features.xml'.
2024.07.23 16:12:45.904296 [ 691 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/users.d/users.xml'.
2024.07.23 16:12:45.907082 [ 691 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.
2024.07.23 16:12:45.907210 [ 691 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performing update on configuration
2024.07.23 16:12:45.909285 [ 691 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performed update on configuration
2024.07.23 16:12:45.952867 [ 692 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/config.xml'
2024.07.23 16:12:45.952930 [ 692 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-server/config.xml'.
2024.07.23 16:12:45.955847 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-hostname-ports.xml'.
2024.07.23 16:12:45.956263 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-macros.xml'.
2024.07.23 16:12:45.956583 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/conf.d/chop-generated-zookeeper.xml'.
2024.07.23 16:12:45.956963 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-01-listen.xml'.
2024.07.23 16:12:45.957279 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-02-logger.xml'.
2024.07.23 16:12:45.957582 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-03-query_log.xml'.
2024.07.23 16:12:45.957941 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-04-part_log.xml'.
2024.07.23 16:12:45.958287 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-05-trace_log.xml'.
2024.07.23 16:12:45.958717 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/chop-generated-remote_servers.xml'.
2024.07.23 16:12:45.959326 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/config.xml'.
2024.07.23 16:12:45.959630 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/macros.xml'.
2024.07.23 16:12:45.959947 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/operator_metrics.xml'.
2024.07.23 16:12:45.960408 [ 692 ] {} <Debug> ConfigProcessor: Merging configuration file '/etc/clickhouse-server/config.d/z_log_disable.xml'.
2024.07.23 16:12:45.972532 [ 692 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
2024.07.23 16:12:45.972880 [ 692 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/config.xml', performing update on configuration
2024.07.23 16:12:45.985132 [ 692 ] {} <Information> Application: Setting max_server_memory_usage was set to 6.71 GiB (7.45 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2024.07.23 16:12:45.985171 [ 692 ] {} <Information> Application: Setting merges_mutations_memory_usage_soft_limit was set to 3.73 GiB (7.45 GiB available * 0.50 merges_mutations_memory_usage_to_ram_ratio)
2024.07.23 16:12:45.985190 [ 692 ] {} <Information> Application: Merges and mutations memory limit is set to 3.73 GiB
2024.07.23 16:12:45.989233 [ 692 ] {} <Information> MergeTreeBackgroundExecutor: Loaded new threads count (16) and max tasks count (32) for MergeMutateExecutor
2024.07.23 16:12:45.989284 [ 692 ] {} <Information> MergeTreeBackgroundExecutor: Loaded new threads count (8) and max tasks count (8) for MoveExecutor
2024.07.23 16:12:45.989303 [ 692 ] {} <Information> MergeTreeBackgroundExecutor: Loaded new threads count (16) and max tasks count (16) for FetchExecutor
2024.07.23 16:12:45.989321 [ 692 ] {} <Information> MergeTreeBackgroundExecutor: Loaded new threads count (8) and max tasks count (8) for CommonExecutor
2024.07.23 16:12:45.991295 [ 692 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.2.1.2248 (official build)). If it is an IPv6 or IPv4 address and
 your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</
listen_host>
2024.07.23 16:12:45.991566 [ 692 ] {} <Warning> Application: Listen [::]:0 failed: Poco::Exception. Code: 1000, e.code() = 0, SSL Exception: Configuration error: no certificate file has been specified (version 24.2.1.2248 (official build)). If it is an IPv6 or IP
v4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <li
sten_host>::</listen_host>
2024.07.23 16:12:45.991887 [ 692 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.2.1.2248 (official build)). If it is an IPv6 or IPv4 address and
 your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</
listen_host>
2024.07.23 16:12:45.992138 [ 692 ] {} <Warning> Application: Listen [::]:0 failed: Poco::Exception. Code: 1000, e.code() = 0, SSL Exception: Configuration error: no certificate file has been specified (version 24.2.1.2248 (official build)). If it is an IPv6 or IP
v4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <li
sten_host>::</listen_host>
2024.07.23 16:12:45.992420 [ 692 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.2.1.2248 (official build)). If it is an IPv6 or IPv4 address and
 your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</
listen_host>
2024.07.23 16:12:45.992784 [ 692 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.2.1.2248 (official build)). If it is an IPv6 or IPv4 address and
 your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</
listen_host>
2024.07.23 16:12:45.993050 [ 692 ] {} <Warning> Application: Listen [0.0.0.0]:0 failed: Poco::Exception. Code: 1000, e.code() = 0, SSL Exception: Configuration error: no certificate file has been specified (version 24.2.1.2248 (official build)). If it is an IPv6
or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4
: <listen_host>::</listen_host>
2024.07.23 16:12:45.993304 [ 692 ] {} <Warning> Application: Listen [0.0.0.0]:0 failed: Poco::Exception. Code: 1000, e.code() = 0, SSL Exception: Configuration error: no certificate file has been specified (version 24.2.1.2248 (official build)). If it is an IPv6
or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4
: <listen_host>::</listen_host>
2024.07.23 16:12:45.993651 [ 692 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.2.1.2248 (official build)). If it is an IPv6 or IPv4 address and
 your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</
listen_host

Configuration files:

$ ll etc/clickhouse-server/config.d/
total 40
lrwxrwxrwx. 1 pasha pasha 34 Jul 23 19:12 01-clickhouse-01-listen.xml -> ..data/01-clickhouse-01-listen.xml
lrwxrwxrwx. 1 pasha pasha 34 Jul 23 19:12 01-clickhouse-02-logger.xml -> ..data/01-clickhouse-02-logger.xml
lrwxrwxrwx. 1 pasha pasha 37 Jul 23 19:12 01-clickhouse-03-query_log.xml -> ..data/01-clickhouse-03-query_log.xml
lrwxrwxrwx. 1 pasha pasha 36 Jul 23 19:12 01-clickhouse-04-part_log.xml -> ..data/01-clickhouse-04-part_log.xml
lrwxrwxrwx. 1 pasha pasha 37 Jul 23 19:12 01-clickhouse-05-trace_log.xml -> ..data/01-clickhouse-05-trace_log.xml
lrwxrwxrwx. 1 pasha pasha 40 Jul 23 19:12 chop-generated-remote_servers.xml -> ..data/chop-generated-remote_servers.xml
lrwxrwxrwx. 1 pasha pasha 17 Jul 23 19:12 config.xml -> ..data/config.xml
lrwxrwxrwx. 1 pasha pasha 17 Jul 23 19:12 macros.xml -> ..data/macros.xml
lrwxrwxrwx. 1 pasha pasha 27 Jul 23 19:12 operator_metrics.xml -> ..data/operator_metrics.xml
lrwxrwxrwx. 1 pasha pasha 24 Jul 23 19:12 z_log_disable.xml -> ..data/z_log_disable.xml

[Slach]

let's check generated pod manifest

kubectl get pod -n gidplatform-dev chi-gid-gid-0-0-0 -o yaml

without sensitive credentials

[Hubbitus]

Hello!

kubectl get pod -n gidplatform-dev chi-gid-gid-0-0-0 -o yaml output: chi_gidplatform-dev.yaml.gz

[Slach]

this wrong output you shared kind: ClickHouseInstallation the same i requested kind: Pod

shared yaml for ClickHouseInstallatin doesn't contains changes which i recommends in https://github.com/Altinity/clickhouse-operator/issues/1456#issuecomment-2245503010

open your manifests and replace

          volumeMounts:
          - mountPath: /var/private/ssl
            name: kafkassl

to

          volumeMounts:
          - mountPath: /var/private/ssl
            name: kafkassl
          - mountPath: /etc/clickhouse-server/config.d/
            name: merged-config

and replace

        volumes:
        - name: kafkassl
          secret:
            secretName: bi-clickhouse

to

        volumes:
        - name: kafkassl
          secret:
            secretName: bi-clickhouse
        - name: merged-config
          projected:
            sources:
            - configMap:
                name: chi-git-common-configd
            - secret:
                secretName: clickhouse-secret-named-collections-xml

our communication looks like someone else manage your manifest and you don't understand what exacly happens in your infrastucture, it looks like broken phone, twice

[Hubbitus]

@Slach, very sorry. You are right. Mistake happened. The error provided before is from branch with the configuration as you provided. But YAML dump was made after state was rolled back to the main branch.

With that (current) configuration ArgoCD can't create pod with error:

FailedCreate
create Pod chi-gid-gid-0-0-0 in StatefulSet chi-gid-gid-0-0 failed error: Pod "chi-gid-gid-0-0-0" is invalid: [spec.volumes[2].projected.sources[1].secret.name: Required value, spec.containers[0].volumeMounts[1].name: Not found: "merged-config"]

Am I right and output of kubectl get chi -n gidplatform-dev gid -o yaml will be needed?

[Slach]

spec.volumes[2].projected.sources[1].secret.name: Required value, spec.containers[0].volumeMounts[1].name: Not found: "merged-config"]

let's replace secretName to name in projected apply

        volumes:
        - name: kafkassl
          secret:
            secretName: bi-clickhouse
        - name: merged-config
          projected:
            sources:
            - configMap:
                name: chi-git-common-configd
            - secret:
                name: clickhouse-secret-named-collections-xml

look documentation

kubectl explain pod.spec.volumes.projected.sources
kubectl explain pod.spec.volumes.projected.sources.secret

[Hubbitus]

Awesome. @Slach, thank you very much!