Support for nested groups in LDAP authentication realm

[robin13]

Currently Elasticsearch does not support searching nested groups in LDAP, but it does with AD (see docs). Would it be possible to enable nested groups for the LDAP authentication realm as well?

[elasticmachine]

Pinging @elastic/es-security

[tvernum]

The information in the docs is heavily summarised, and the reality is slightly more nuanced that the docs imply. We may want to update the doc to try and reflect reality a little more closely.

TL;DR Our LDAP realm supports nested groups if-and-only-if the directory resolves them transparently. Active Directory does resolve nested groups, and it is my understanding that Red Hat Directory Server / 389DS does as well. OpenLDAP does not. Other commercial LDAP directories might.

Technical Details In any user store where there is a concept of Users and Groups, you have two options about how to model & store that relationship:

1. Groups contain a list of their members.
2. Members (typically Users) hold a list of the groups to which they belong.

Standard LDAP schemas implement the former option. For example the GroupOfNames ObjectClass contains a member attribute which holds the DN for each object that is a member of that group. That DN could refer to a User or another GroupOfNames.

This has some advantages:

• Only those object classes that specifically deal with membership (i.e. Groups) need to know the details of how membership is handled. From an encapsulation point of view this is preferable, compared with scattering membership information into all object classes that can potentially be a member of a group.
• It is optimized for answering questions in the form of "who is a member of this group?", which (see point below) is often a very close proxy for "who has access to this data/file/server/etc?".
• For access control models that are built directly on top of groups - e.g "Users in the ServerAdmin group may log on to this server" - it is quite efficient at answering those questions at runtime (you expand ths ServerAdmin group to get its members, and then see whether the current user is in that list).

However, it is not a good fit for the type of model Elasticsearch Security uses where groups are essentially just metadata about users. When a user authenticates to Elasticsearch using LDAP, we resolve their metadata, including their group membership, and turn that into a list of roles.
We are never interested in the question "Who is a member of MI6?", we always want to ask "To which groups does James.Bond belong?

Thankfully many LDAP directory have support for tracking the inverted relationship from User to Group via memberOf functionality. Active Directory and Red Hat/389 DS do this by default. OpenLDAP requires the optional MemberOfOverlay.

You can see the difference between these two between these two possible LDAP models in the configuration options for the LDAP realm.

• The user_group_attribute setting is used to configure the name of the memberOf attribute, in which case we simply ask the directory server to return the user's group membership when we retrieve the rest of their data.
• For directories without memberOf support, we need to use the group_search.* settings, which tell us how to search the directory for any groups that list the user as a member.

When you're doing the top-down group resolution that LDAP schemas are typically designed around, nested group expansion is a little cumbersome, but manageable. The approach is roughly:
To find all the (nested) members of group CN=Elasticsearch Admins, OU=groups, DC=example, DC=com

1. Access the CN=Elasticsearch Admins, OU=groups, DC=example, DC=com entry, and retrieve the member attribute.
2. Access the entries for each of those "members" (it is usually possible to get all of them in a single request), and retrieve the dn, cn, objectClass and member attributes.
3. For any entries where the objectClass is a User, add the DN/CN to the list of "users-who-belong-to-this-group".
4. For any entries where the objectClass is a group, add their members to the list of "members-to-retrieve" and go to step 2.
5. If there were no groups in step 4, you're finished.

However, working from the bottom-up approach (using group_search) is horribly inefficient.
To find all the (nested) groups to which CN=Keyser Söze, OU=users DC=example, DC=com belongs

1. Search the directory for any groups that list CN=Keyser Söze, OU=users DC=example, DC=com as a member.
2. Add the DN of each result to the "group-membership" set.
3. For each DN that was new in step 2 (that is, not already in the set), go to step 1. It is sometimes possible to search for several groups at once, but it is not usually possible to do them all in one request.

Logically, that is quite easy, but the search in step 1 (find entries with these attributes) is slow when compared with the get-entry-by-DN lookup that is used in the top-down approach.

We have chosen not implement that behaviour because it can lead to very slow authentication processing and holds on to pooled LDAP connections and internal ES locks while it does that. It is possible that we might chose to do implement this in the future, but our estimation is that given the performance profile of that sort of implementation, it would not actually be very useful in the real world.

We would be more willing to do nested memberOf support because that has a performance profile more like the top-down approach.
However, for Acitve Directory and Red Hat/389 DS we don't need to, because both of those directories automatically do it for us. When we ask which groups a user belongs to, they expand nested groups for us (technically the keep the inverted references up to date whenever a group is modified, rather than building them when we search so it is faster). The only directory that I know of that has memberOf support but does not automatically expand nested-groups is OpenLDAP.

Net Result

1. If you are using Active Directory, then use our active_directory realm. It will automatically handle nested groups, as well as a bunch of other AD specific things.
2. If you are using RedHat DS or 389DS, then you should already have nested group support using user_group_attribute / memberOf. If that isn't working correctly, then please let us know because we can probably fix it.
3. If you are using OpenLDAP with the memberOf overlay (or any other LDAP directory that supports user_group_attribute / memberOf, but doesn't automatically expand nested groups) and want support for nested groups, please tell us, because we can probably build it (but even better - tell your directory vendor that they should automatically expand nested groups in their memberOf attribute)
4. If you are using a directory that requires group resolution using group_search (and not user_group_attribute) then we are unlikely to support nested groups, because it is not technically possible for us to implement it in an efficient manner.

[maltewhiite]

Hello @tvernum - TLDR: How do I make nested groups work in Elastic, with a 389 LDAP server?

Removed original post, and moved it here, to save space: https://pastebin.com/raw/3HU9sZc9

[jkakavas]

Hi @maltewhiite, this issue is about discussing generic support for nested groups in Elasticsearch ldap authentication realm. Would you mind opening a topic with your user question in our forums ? Someone will try and assist you there. Thanks!

[maltewhiite]

EDIT: I got it to work in my local vagrant setup. You're welcome to delete my two comments if you want this thread to be clean.

I used the following documentation:

• https://www.elastic.co/guide/en/elasticsearch/reference/7.0/security-settings.html#ref-ldap-settings
• https://www.elastic.co/guide/en/elasticsearch/reference/7.0/configuring-ldap-realm.html
• https://www.elastic.co/guide/en/elasticsearch/reference/7.0/ldap-realm.html
• https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-clear-cache.html - Used for testing. So I can set IDM groups, clear cache, and immediately test, without having to wait for the server to update its cache.

If group_search.base_dn isn't set, then it enables user_group_attribute and user_search.scope. user_group_attribute has default memberOf and user_search.scope has default sub_tree. sub_tree searches all objects contained under base_dn.

Basically all I ended up doing to make it work, was removing this following line from the /etc/elasticsearch/elasticsearch.yml file, and then restarting the service: xpack.security.authc.realms.ldap.dap389.group_search.base_dn: cn=groups,cn=accounts,dc=sanitized

This wasn't easy to figure out for me. Spent 16 hours to figure this out. I basically knew nothing about LDAP and Elastic before this, though.