Clickhouse fails parsing users.xml with 0.23.0

[derbauer97]

Hey,

I noticed that 0.23.0 was released and wanted to test the feature reading password directly from Secrets. But i get the following error:

2024.02.02 09:05:54.562549 [ 1 ] {} Application: DB::Exception: Either 'password' or 'password_sha256_hex' or 'password_double_sha1_hex' or 'no_password' or 'ldap' or 'kerberos' or 'ssl_certificates' must be specified for user networks.: while parsing user 'networks' in users configuration file: while loading configuration file '/etc/clickhouse-server/users.xml'

i thought ok maybe my Clickhouse Version 23.3.19.32 is not supporting reading password from env so i rolled the configuration back to the old secrets syntax. But the issue still persists so i had to roll back to 0.22.2 to fix my Installation.

I could not find any info in the Clickhouse Documentation when password from env is supported or any hint on this configuration at all.

[Slach]

you should need to use:

spec:
  configuration:
    users:
      user_name/k8s_secret_password: secret_name/key_in_secret 

or

spec:
  configuration:
    users:
      user_name/k8s_secret_password_sha256_hex: secret_name/key_in_secret 

[derbauer97]

My Configuration looks like these:

     spec:
       configuration:
            users:
                user1/k8s_secret_password: clickhouse-user/user-1
                clickhouse_operator/k8s_secret_password: clickhouse-user/clickhouse_operator

But i get the same error with 0.23.0. It is working with 0.22.2

[alex-zaitsev]

Hi @derbauer97 , this is weird indeed. We have deprecated this syntax in favour of a better one (see release notes with examples), but old syntax should work. It is covered by regression tests, and all tests pass.

[hueiyuan]

@alex-zaitsev We have encountered the same problem in 0.23.0

[derbauer97]

@alex-zaitsev

i tested both. And both settings throw the same error. With the new syntax i can at least understand the error message. But why the the old deprecated syntax is not working i can't explain.

[salimidruide]

Same here: Please check my configurations: https://github.com/Altinity/clickhouse-operator/issues/1332 I had no issues on operator 0.22.2

[alex-zaitsev]

@derbauer97 , could you check generated config map? It should have 'usersd' in the name, so should be easy to locate. We still do not understand where it breaks.

[derbauer97]

@alex-zaitsev

i think i found the problematic configuration:

43a44,45
>           <prefer_localhost_replica>0</prefer_localhost_replica>
>           <!-- materialize_ttl_recalculate_only>1</materialize_ttl_recalculate_only> 21.10 and above -->
106c108
<                     <ip>192.168.5.99</ip>
---
>                     <ip>192.168.5.24</ip>
125a128,131
>             <networks>
>                 <ip>192.168.4.186</ip>
>             </networks>
>             <profile>clickhouse_operator</profile>
140c146

that is the diff between the config map of 0.22.2 and 0.23.0. Somehow the operator creates a user named networks which obviously does not have a user configuration:

2024.02.06 09:05:24.113345 [ 1 ] {} Application: DB::Exception: Either 'password' or 'password_sha256_hex' or 'password_double_sha1_hex' or 'no_password' or 'ldap' or 'kerberos' or 'ssl_certificates' must be specified for user networks.: while parsing user 'networks' in users configuration file: while loading configuration file '/etc/clickhouse-server/users.xml'

this is also what the log states.

        <users>
            <clickhouse_operator>
                <networks>
                    <host_regexp>(chi-instana-clickhouse-[^.]+\d+-\d+|clickhouse\-instana-clickhouse)\.clickhouse\.svc\.cluster\.local$</host_regexp>
                    <ip>::1</ip>
                    <ip>127.0.0.1</ip>
                    <ip>192.168.0.0/16</ip>
                </networks>
                <password_sha256_hex>test123</password_sha256_hex>
                <profile>default</profile>
                <quota>default</quota>
            </clickhouse_operator>
            <default>
                <networks>
                    <host_regexp>(chi-instana-clickhouse-[^.]+\d+-\d+|clickhouse\-instana-clickhouse)\.clickhouse\.svc\.cluster\.local$</host_regexp>
                    <ip>::1</ip>
                    <ip>127.0.0.1</ip>
                    <ip>192.168.5.24</ip>
                    <ip>192.168.12.100</ip>
                    <ip>192.168.17.90</ip>
                    <ip>192.168.18.75</ip>
                    <ip>192.168.0.0/16</ip>
                </networks>
                <profile>default</profile>
                <quota>default</quota>
            </default>
            <instana>
                <networks>
                    <host_regexp>(chi-instana-clickhouse-[^.]+\d+-\d+|clickhouse\-instana-clickhouse)\.clickhouse\.svc\.cluster\.local$</host_regexp>
                    <ip>::1</ip>
                    <ip>127.0.0.1</ip>
                    <ip>192.168.0.0/16</ip>
                </networks>
                <password_sha256_hex>test123</password_sha256_hex>
                <profile>default</profile>
                <quota>default</quota>
            </instana>
            <networks>
                <ip>192.168.4.186</ip>
            </networks>
            <profile>clickhouse_operator</profile>
        </users>

Here also the ActionPlan:

I0206 09:02:40.135588       1 worker.go:568] ActionPlan start---------------------------------------------:
Diff item start -------------------------
removed spec items: 2
diff item path [0]:'.Configuration.Users.m["default/networks/ip"].vector[7]'
diff item value[0]:'"192.168.4.232"'
diff item path [1]:'.Configuration.Users.m["default/networks/ip"].vector[8]'
diff item value[1]:'"192.168.0.0/16"'
Diff item end -------------------------
Diff item start -------------------------
modified spec items: 2
diff item path [0]:'.TaskID'
diff item value[0]:'"d666bbfa-d6d3-40ec-9fb9-22c281d5e55d"'
diff item path [1]:'.Configuration.Users.m["default/networks/ip"].vector[6]'
diff item value[1]:'"192.168.0.0/16"'
Diff item end -------------------------
modified labels

[sunsingerus]

Very similar issue is reproducible with OLM setup, which is described in here #1332 Just to understand the background, couple of questions

1. Do you use OLM setup?
2. Do you have operator configuration specified with the custom resource?

[derbauer97]

Hey @sunsingerus,

1. No we are using the offical helm chart to install the operator
2. We are not using custom configuration

[sunsingerus]

@derbauer97 we are working on 0.23.1 version which has fixes of this issue. Please check. It is not released yet, but image is available on docker hub and Helm chart is in the 0.23.1 branch as well

[alex-zaitsev]

Fixed in https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.23.1