[HELM] Unable to load a custom `internal_users.yml`

dotdc commented 3 years ago

Describe the bug

Hi, I've been unable to load a custom internal_users.yml to my kubernetes helm chart deployment using both presented method in the values file (issue refer to the elasticsearch.securityConfig.config.data.internal_users.yml method).

Is this a bug or is there something wrong in my configuration?

Thank you in advance.

To Reproduce Steps to reproduce the behavior:

Build opendistro helm chart (https://opendistro.github.io/for-elasticsearch-docs/docs/install/helm/)
Deploy with custom values (see custom.values.yaml below)
Check users on Kibana or in /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml on the master pod

Expected behavior

This should load the custom internal_users.yml set in custom.values.yaml

Configuration (please complete the following information):

ODFE/Kibana version : 1.13.1
Distribution : Helm Chart
Kubernetes v1.18.16
Host Machine : Flatcar Container Linux by Kinvolk 2765.2.1

Aditionnal question

In the default internal_users.yml there is several users other than admin:

kibanaserver
kibanaro
logstash
readall
snapshotrestore

Are they used in any way or just here as an example?

Relevant information

custom.values.yaml :

---
elasticsearch:
  securityConfig:
    config:
      data:
        # config.yml: |-
        # roles.yml: |-
        # rolesMapping.yml: |-
        # tenants.yml: |-
        internal_users.yml: |-
          _meta:
            type: "internalusers"
            config_version: 2

          fluentbit:
            hash: "$2y$12$4uM.APaTAjDXKGGjmyCZYexXFFXUnZ4sXSv2nNrF0bmCJgY/cXmKW"
            reserved: false
            hidden: false

          admin:
            hash: "$2y$12$BrMTXYvq04gV3vMxAON7I.4/cXsxsf.nXqIur.gLGYlpydze3kRzq"
            reserved: true
            backend_roles:
            - "admin"

cmanzur commented 3 years ago

Exactly the same problem using k8s v1.20 My custom internal_users.yml file is mounted in the pod but the users are not loaded. Is maybe mandatory to add an elasticsearch.config to make it work? But I don't want to do that, I only want to change the users.

sreekarjami commented 3 years ago

Hi @dotdc , Thanks for opening the issue, we will try reproducing the issue and inform if we need additional information.

leonzo commented 3 years ago

I have the same issue from the default helm configuration https://opendistro.github.io/for-elasticsearch-docs/docs/install/helm/ to which I want to update configuration file using custome value

This is actually a config update recommendation from here https://github.com/opendistro-for-elasticsearch/opendistro-build/tree/main/helm#security-configuration

I am expecting the custom.values.yml to show up in /usr/share/elasticsearch/plugins/opendistro_security/securityconfig

But it is the default file which appears without the added custom config which is no where to be found.

For me it looks another config maybe in values.yaml is needed.

Looking forward for any help

dotdc commented 3 years ago

I saw some indent changes in values.yaml in the new 1.13.2 but the problem is still here. \ Does anyone use this feature? If yes, could they share they values/configs?

I would like to know if there is another setting needed to make this work or if it's just ignored by the templating (I didn't find the usage in the templating)

Let me know if you need more information or if I can help solve this problem.

drewmarshburn commented 3 years ago

Hey, I had the same problem you have described and found a solution.

At a high level, my goal was to change only the internal_users.yml and avoid touching the other default security configurations.

There's a few moving parts so I just want to capture a brief overview (partially for my own notes/team):

The opendistro_security plugin manages configuration as an index within Elasticsearch itself (more info here). The index is .opendistro_security.
Each elasticsearch node also has file-based opendisto_security plugin configuration. By default this is stored in /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/. The internal_users.yml file is one component of this configuration. From what I can tell, the defaults come from the opendistro_security plugin itself (see here). I cannot find any documentation of the default users, including their credentials or their purposes. (The only one I could find was admin/admin creds of the admin user, and I found that in various forum posts.)
The Helm chart options allow you to specify values that are mounted as files in place of the default configuration.

There are two options for overwriting internal_users.yml with the Helm chart values:

elasticsearch.securityConfig.config.data: This allows you to overwrite all of the default configuration in one place and has been suggested in another issue as a solution. I did not get this approach working, I'll give more details below.
elasticsearch.securityConfig.internalUsersSecret: This allows you to reference the internal_users.yml file contents provided within a secret. I succeeded by using this option.

Both of the available settings are applied only to the configuration of the Elasticsearch master node, the client and data nodes will have the default configuration files present and they will never be changed. (You can see both getting applied in helm/opendistro-es/templates/elasticsearch/es-master-sts.yaml.)

I'll give a little more detail on both methods before describing the problem that (from what I can tell) affects them both.

Using `elasticsearch.securityConfig.config.data`

Contrary to what was suggested above, this setting does appear in the Helm chart: https://github.com/opendistro-for-elasticsearch/opendistro-build/blob/9549021030d2ff5ee506996b37f97731fce41970/helm/opendistro-es/templates/elasticsearch/es-master-sts.yaml#L284. The gotcha with this approach is that, for some reason, you have to provide a value for elasticsearch.securityConfig.config.securityConfigSecret. Not sure why this is the case because the data itself can be provided directly in the values file.

The real reason this method did not work for me is that it overwrites all of the default configuration files, so I found that it broke things when I was just trying to provide internal_users.yml.

Using `elasticsearch.securityConfig.internalUsersSecret`

This also gets mounted as configuration for the master node: https://github.com/opendistro-for-elasticsearch/opendistro-build/blob/9549021030d2ff5ee506996b37f97731fce41970/helm/opendistro-es/templates/elasticsearch/es-master-sts.yaml#L299. For extra clarity, here are some example files:

values.yaml:

elasticsearch:
  securityConfig:
    internalUsersSecret: es-internal-users

es-internal-users.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: es-internal-users
stringData:
  internal_users.yml: |-
    _meta:
      type: "internalusers"
      config_version: 2
    admin:
      hash: YOURHASH
      reserved: true
      backend_roles:
      - "admin"

This worked for me because it only overwrites the internal_users.yml without modifying the rest of the sample config.

Warning: This will break the Kibana connection because it relies on the default users to connect to Elasticsearch.

You should provide a new Kibana connection account as a Secret that you reference in your values.yaml like so:

values.yaml:

kibana:
  elasticsearchAccount:
    secret: kibana-user

kibana-user.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: kibana-user
stringData:
  username: admin
  password: SECRET
  cookie: SECRET

The Big Problem

Regardless of the approach that you use, I think the problem here is how the .opendistro_security index gets initialized. Seems like there are two subproblems:

Configuration only winds up on the master node, so if the client or data node start up first, they will set the index with the default configuration and the conf from the master node gets ignored.
It is very unclear what settings affect the index creation. The README mentioned opendistro_security.allow_default_init_securityindex: true but I was not able to see any results from using this.

Ultimately, once all the elasticsearch nodes started, I had to get a shell into the master node and use the securityadmin.sh command to ensure that the configuration was applied to the index properly. I found it convenient to rerun the /usr/share/elasticsearch/securityadmin_demo.sh since that takes the default configuration directory and you don't have to delve into all of the securityadmin.sh options.

Solution Steps

Just to summarize:

Write a new internal_users.yml file into a Secret. (Use /usr/share/elasticsearch/plugins/opendistro_security/tools/hash.sh to create the password hashes.)
Provide the secret name in elasticsearch.securityConfig.internalUsersSecret
Take one of your users and provide them as the Kibana connection user as described above.
Redeploy the Helm chart, depending on the node startup interleaving, you could get lucky and have your new config.
If you have the default, exec into the master node and run /usr/share/elasticsearch/securityadmin_demo.sh

Moving Forward

Few comments for collaborators.

The big one: configuration should be deterministically applied. Not sure if the node startup could be reordered or if the configuration files should just get mounted in all nodes, but it's definitely broken that we can provide config that is not reflected in the running system after a restart.

Secondly, the README of the Helm chart is very confusing. I think part of the issue (and this is inevitable) is that you're configuring so many things at once: you're giving the deployment settings for the Helm chart itself, you're configuring the Elasticsearch nodes, you're configuring the opendistro_security plugin for Elasticsearch, you're configuring Kibana, and there's probably more. It seems like a lot of time is spent on documenting the "sub-settings" for all of the individual components, and that may be better suited to doc in the sub-projects themselves. The commercial Elastic products do a pretty good job of making references to other places, the ECK documentation is a good example.

Finally, it may be nice to reflect the default settings in the Helm chart itself. It is not clear how the demo settings are applied or invalidated, or how you can change some (like only modifying the internal_users.yml) without changing others. (As an example, if I provide a single value in elasticsearch.config, will it completely scrap the demo configuration?)

I'm happy to jump in and get my hands dirty if you can provide guidance/feedback. Thanks!

dotdc commented 3 years ago

Hi drewmarshburn, thank you for this detailed reply!

I didn't had time to update this issue but I managed to get this working yesterday after finding out the need to set elasticsearch.securityConfig.config.securityConfigSecret in the PR : https://github.com/opendistro-for-elasticsearch/opendistro-build/pull/250). I will make a PR to set a value on the default values.yml to make this easier for others on a future release.

I also faced the same issue, once internal_users.yml was finally loaded, I had to copy/pasted all the default files in my custom values files to make it work. I also needed to add nodes_dn.yml and whitelist.yml (not present in the default values.yml) in order to use securityadmin_demo.sh without errors.

I'm now facing a strange behavior, I used hash.sh to generate hashes for several users and some of them don't work... I tried to create a new hash several times but I always get a password error on kibana (only on a few users, the others are working fine). Will try to change usernames to see if their is something wrong with them (too long, too many "-" or something else...)

I agree with you, values.yml should contain all the default settings/files to make it easier all users. I'm also willing to help but I'm just starting with Opendistro so... :)

Thanks again, this helped me understand a bit more Opendistro and will probably help others too.

drewmarshburn commented 3 years ago

Glad to hear you were also able to work this out! I haven't had any issues with hash.sh but so far I have only reset the password of the admin user, so I'll keep an eye out as I do more.

I'm an ODFE noob too, so looking forward to some guidance as it's available!

dotdc commented 3 years ago

PR : https://github.com/opendistro-for-elasticsearch/opendistro-build/pull/750

shibumi commented 3 years ago

@drewmarshburn thanks a lot! you just made my day with your walkthrough.

EDIT: as you suggested I had to run securityadmin_demo.sh, because the configuration did not applied correctly. This is a security risk.There should be no manual intervention needed when logging into the pod.

EDIT: Ok another observation. I configured everything correctly, but kibana throws errors like this now:

{"type":"log","@timestamp":"2021-05-03T16:15:03Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[security_exception]: no permissions for [cluster:monitor/nodes/info] and User [name=admin, backend_roles=[], requestedTenant=null]"}

The secret has definitely the backend_roles: ["admin"] entry...

dotdc commented 2 years ago

PR for this issue moved to Opensearch : https://github.com/opensearch-project/helm-charts/pull/142

opendistro-for-elasticsearch / opendistro-build