Duplicated security config in opensearch.yml

[pawelw1]

Describe the bug

Bug Fix for configMap Read-only file system error in statefulset.yml https://github.com/opensearch-project/helm-charts/pull/516 creates duplicated security config in opensearch.yml

https://github.com/opensearch-project/helm-charts/commit/a1c5b8f301d68649f0534b63bf545a61844ce651#diff-08885706cd45ad445d2696cd7e39cf7d4b5e82cb51b0ee6bd86e6ee9f818e158

To Reproduce

Steps to reproduce the behaviour:

1. Deploy charts from the latest back to 2.11.1.
2. OpenSearch nodes will get deployed but opensearch.yml will contain duplicated security config.
3. The OpenSearch node will ignore the custom security config from values.yml (.Values.config.opensearch.yml) and will use a duplicated config as per the example.

Example:

pablo@kube-1:~$ kubectl exec -it opensearch-cluster-master-0 -- cat config/opensearch.yml
Defaulted container "opensearch" out of: opensearch, fsgroup-volume (init), configfile (init)
cluster.name: opensearch-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# Implicitly done if ".singleNode" is set to "true".
# discovery.type: single-node

# Start OpenSearch Security Demo Configuration
# WARNING: revise all the lines below before you go into production
plugins:
  security:
    ssl:
      transport:
        pemcert_filepath: admin.pem
        pemkey_filepath: admin-key.pem
        pemtrustedcas_filepath: root-ca.pem
        enforce_hostname_verification: false
      http:
        enabled: true
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
    allow_unsafe_democertificates: true
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=kirk,OU=client,O=client,L=test,C=de
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]
######## End OpenSearch Security Demo Configuration ########

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-config", ".plugins-ml-connector", ".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".plugins-ml-conversation-meta", ".plugins-ml-conversation-interactions", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models", ".geospatial-ip2geo-data*"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

Expected behavior opensearch.yml must contain only a single security configuration provided through values.yaml.

Chart Name Specify the Chart which is affected? All charts from OpenSearch version 2.11.1 to the latest.

[aitchjoe]

same problem when use chart 2.21.0, if we remove default plugins.security setting (check values.yaml), then no duplicated security config.

but if we want to disable "OpenSearch Security Demo Installer" (seen in log), we have not found a method in Setting up a demo configuration. search source and got SecuritySettingsConfigurer.java:

    void checkIfSecurityPluginIsAlreadyConfigured() {
        // Check if the configuration file contains the 'plugins.security' string
        if (installer.OPENSEARCH_CONF_FILE != null && new File(installer.OPENSEARCH_CONF_FILE).exists()) {
            try (BufferedReader br = new BufferedReader(new FileReader(installer.OPENSEARCH_CONF_FILE, StandardCharsets.UTF_8))) {
                String line;
                while ((line = br.readLine()) != null) {
                    if (line.toLowerCase().contains("plugins.security")) {
                        System.out.println(installer.OPENSEARCH_CONF_FILE + " seems to be already configured for Security. Quit.");
                        System.exit(installer.skip_updates);
                    }
                }

but default values are:

config:
  opensearch.yml: |
    cluster.name: opensearch-cluster
    network.host: 0.0.0.0
    plugins:
      security:
        ......

not plugins.security.... so we tried:

config:
  opensearch.yml: |
    cluster.name: opensearch-cluster
    network.host: 0.0.0.0
    plugins.security.disabled: true

check /usr/share/opensearch/config/opensearch.yml there is no OpenSearch Security Demo Configuration added, if we changed back:

config:
  opensearch.yml: |
    cluster.name: opensearch-cluster
    network.host: 0.0.0.0
    plugins:
      security:
        disabled: true

OpenSearch Security Demo Configuration added.

[dblock]

[Catch All Triage - 1, 2, 3]

[dancristiancecoi]

The issue appears to be due to demo configuration script being run by default on Helm charts even if you made changes to the security config.

To get past this issue you can disable the demo security configuration by setting DISABLE_INSTALL_DEMO_CONFIG to "true" in the extraEnvs section of your values.yaml file.

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "true"

[pawelw1]

It looks like the custom configuration is added to opensearch.yml after the OpenSearch service is up. Otherwise, the service would fail to start due to duplicated entries.

[arunbabumm]

Any update on this bug?. we are unable to use latest version due to this.