LAPS lookup plugin - Githubissues

no-12 commented 7 months ago

SUMMARY

Adds a new LAPS lookup plugin

Because of this comment ansible-collections/community.windows#527 by @jborean93 it looks like community.windows.laps_password is no longer in active development. So I decided to implement of a new lookup plugin that lives inside microsoft.ad, because the code of this lookup plugin heavily relies on the same implementations used by the microsoft.ad.ldap inventory plugin.

This is only a first draft. No automated tests added and I haven't done any manual testing yet because I have no working LAPS setup by hand.

Please let me no if this is something that has the potential to get merged, so I can add some tests

ISSUE TYPE

New Module Pull Request

COMPONENT NAME

laps

github-actions[bot] commented 7 months ago

Docs Build 📝

Thank you for contribution!✨

The docsite for this PR is available for download as an artifact from this run: https://github.com/ansible-collections/microsoft.ad/actions/runs/8118582971

You can compare to the docs for the main branch here: https://ansible-collections.github.io/microsoft.ad/branch/main

File changes:

A collections/index_lookup.html

A collections/microsoft/ad/laps_lookup.html

M collections/environment_variables.html

M collections/index.html

M collections/index_filter.html

M collections/index_inventory.html

M collections/index_module.html

M collections/microsoft/ad/as_datetime_filter.html

M collections/microsoft/ad/as_guid_filter.html

M collections/microsoft/ad/as_sid_filter.html

M collections/microsoft/ad/computer_module.html

M collections/microsoft/ad/debug_ldap_client_module.html

M collections/microsoft/ad/dn_escape_filter.html

M collections/microsoft/ad/docsite/guide_attributes.html

M collections/microsoft/ad/docsite/guide_ldap_connection.html

M collections/microsoft/ad/docsite/guide_ldap_inventory.html

M collections/microsoft/ad/docsite/guide_list_values.html

M collections/microsoft/ad/docsite/guide_migration.html

M collections/microsoft/ad/domain_controller_module.html

M collections/microsoft/ad/domain_module.html

M collections/microsoft/ad/group_module.html

M collections/microsoft/ad/index.html

M collections/microsoft/ad/ldap_inventory.html

M collections/microsoft/ad/membership_module.html

M collections/microsoft/ad/object_info_module.html

M collections/microsoft/ad/object_module.html

M collections/microsoft/ad/offline_join_module.html

M collections/microsoft/ad/ou_module.html

M collections/microsoft/ad/parse_dn_filter.html

M collections/microsoft/ad/user_module.html

M collections/microsoft/index.html

M index.html

M search.html

Click to see the diff comparison.

**NOTE:** only file modifications are shown here. New and deleted files are excluded. See the file list and check the published docs to see those files. ```diff diff --git a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/collections/environment_variables.html b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/collections/environment_variables.html index 2319e92..a76a271 100644 --- a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/collections/environment_variables.html +++ b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/collections/environment_variables.html @@ -82,6 +82,7 @@

Reference indexes:

@@ -141,6 +142,7 @@ Environment variables used by the ansible-core configuration are documented in <

Kerberos support requires the pyspnego[kerberos] extras to be installed.

See LDAP authentication for more information.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -150,6 +152,7 @@ Environment variables used by the ansible-core configuration are documented in <

Can be the path to a CA certificate PEM or DER file, directory of PEM certificates, or the CA certificate PEM string that is used for certificate validation.

If omitted, the default CA store used for validation is dependent on the current Python settings.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -163,6 +166,7 @@ Environment variables used by the ansible-core configuration are documented in <

ignore_hostname will validate the CA trust chain but will ignore any hostname checks performed by TLS.

See Certificate validation for more information.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -175,6 +179,7 @@ Environment variables used by the ansible-core configuration are documented in <

Use certificate_key if the certificate specified does not contain the key.

Use certificate_password if the key is encrypted with a password.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -185,6 +190,7 @@ Environment variables used by the ansible-core configuration are documented in <

The value can either be a path to a file containing the key in the PEM or DER encoded form, or it can be the string of a PEM encoded key.

Use certificate_password if the key is encrypted with a password.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -193,6 +199,7 @@ Environment variables used by the ansible-core configuration are documented in < MICROSOFT_AD_LDAP_CERTIFICATE_PASSWORD

The password used to decrypt the certificate key specified by certificate or certificate_key.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -201,6 +208,7 @@ Environment variables used by the ansible-core configuration are documented in < MICROSOFT_AD_LDAP_CONNECTION_TIMEOUT

The timeout in seconds to wait until the connection is established before failing.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -212,6 +220,7 @@ Environment variables used by the ansible-core configuration are documented in <

The auth_protocol negotiate, kerberos, and ntlm all support encryption over LDAP whereas simple does not.

If using auth_protocol=simple over LDAP without TLS then this must be set to False. As no encryption is used, all traffic will be in plaintext and should be avoided.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -222,6 +231,7 @@ Environment variables used by the ansible-core configuration are documented in <

If auth_protocol is simple and no password is specified, the bind will be performed as an unauthenticated bind.

If auth_protocol is negotiate, kerberos, or ntlm and no password is specified, it will attempt to use the local cached credential specified by username if available.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -232,6 +242,7 @@ Environment variables used by the ansible-core configuration are documented in <

Port 389 is used for LDAP and port 686 is used for LDAPS.

Defaults to port 636 if tls_mode=ldaps otherwise 389.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -242,6 +253,7 @@ Environment variables used by the ansible-core configuration are documented in <

If not specified the server will be derived from the current krb5.conf default_realm setting and with an SRV DNS lookup.

See Server lookup for more information.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -254,6 +266,7 @@ Environment variables used by the ansible-core configuration are documented in <

start_tls will connect over LDAP (port 389) and perform the StartTLS operation before the authentication bind.

It is recommended to use ldaps over start_tls if TLS is going to be used.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

@@ -264,6 +277,7 @@ Environment variables used by the ansible-core configuration are documented in <

If auth_protocol is simple and no username is specified, anonymous authentication is used.

If auth_protocol is negotiate, kerberos, or ntlm and no username is specified, it will attempt to use the local cached credential if available, for example one retrieved by kinit.

Used by: +microsoft.ad.laps lookup plugin, microsoft.ad.ldap inventory plugin

diff --git a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/collections/index.html b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/collections/index.html index 5e93894..ce08a79 100644 --- a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/collections/index.html +++ b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/collections/index.html @@ -86,6 +86,7 @@

Reference indexes:

diff --git a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/collections/index_filter.html b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/collections/index_filter.html index 9e61e9c..e788af7 100644 --- a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/collections/index_filter.html +++ b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/collections/index_filter.html @@ -25,7 +25,7 @@ - + @@ -86,6 +86,7 @@

Index of all Inventory Plugins

+

Index of all Lookup Plugins

Index of all Modules

Reference indexes:

@@ -139,7 +140,7 @@

diff --git a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/search.html b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/search.html index e8e80f0..665694e 100644 --- a/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/base/search.html +++ b/home/runner/work/microsoft.ad/microsoft.ad/docsbuild/head/search.html @@ -82,6 +82,7 @@

Reference indexes:

```

softwarefactory-project-zuul[bot] commented 7 months ago

Build succeeded. https://ansible.softwarefactory-project.io/zuul/buildset/4e5a16db6db24e01ad3e1afd4d5fcc3e

:heavy_check_mark: ansible-galaxy-importer SUCCESS in 5m 11s :heavy_check_mark: build-ansible-collection SUCCESS in 8m 26s

softwarefactory-project-zuul[bot] commented 7 months ago

Build succeeded. https://ansible.softwarefactory-project.io/zuul/buildset/092535e979e740caaac53166f4fff63b

:heavy_check_mark: ansible-galaxy-importer SUCCESS in 4m 44s :heavy_check_mark: build-ansible-collection SUCCESS in 8m 25s

jborean93 commented 7 months ago

I was originally going to create this lookup plugin but decided not to in favour of the inventory plugin. The primary reason behind this is due to how templates works in Ansible. When you set the variable ansible_password: '{{ lookup("microsoft.ad.laps", ...) }}' in your inventory/group_vars then the literal value will be the template string. Only when that variable is used will the lookup be run which means that each task in your playbook will require this LDAP query to run slowing down your tasks. It's even worse when you are using the encrypted LAPS password because not only does it need to open a connection to the LDAP server to get the raw value it also then needs to open an RPC connection to decrypt the value. This is also done per user even if they are targeting the same domain environment as the lookup is run per var and there is no shared LDAP connection it can utilise together. So altogether using a lookup template as the value for ansible_password means you have not only the extra connection plugin connection but potentially 1-2 connections for LAPS plus the network hops needed for the authentication attempt per task per user.

The LDAP inventory plugin avoids all this because it can retrieve the data for all hosts in one invocation, cache the actual value in the var so it doesn't need to run a task. I would be more keen on potentially documenting this a bit better an potentially showing how it is possible to replicate how to use LAPS password with the LDAP inventory plugin.

I am happy to be convinced otherwise but the above is what made me hold off from creating an explicit LAPS lookup to replace the one in community.windows.

no-12 commented 7 months ago

As far as I know, if you use a lookup plugin inside the set_facts module, then it will only run once. per playbook.

There are some issues with the inventory approach:

When there is a huge inventory and you only want to target a very small sub-set with dynamic "hosts" field or limit. In this scenario there is a huge overhead of fetching all LAPS credentials.
Inventories in AWX are strictly separated from the playbook. You can use constructed Inventories like the microsoft.ad.ldap plugin. But this will store all LAPS credentials in clear text in the AWX DB #100 and reevaluate the constructed inventory on every playbook run can take several minutes on huge inventories.

So I think adding a second possiblity to this collection gives the user the chance to choose the "right" approach for his problem.

@jborean93 Thanks to your work regarding the LDAP communication and especially the rpc to decrypt the LAPS password, it is quite easy to add this lookup plugin the collection. The ldap plugin utils are marked as "internal" so it would be ver hard to maintain the lookup plugin outside of this collection.

jborean93 commented 7 months ago

As far as I know, if you use a lookup plugin inside the set_facts module, then it will only run once. per playbook.

That is the exception to the rule yes as the templating actually happens during the set_fact run and it will set the templated result as the variable. Unfortunately this means to use the lookup properly means having your playbook with this explicit step resulting in some hard ties between the inventory and your play which is never nice. Plus now people need to know how to "safely" use the lookup which isn't very good UX and by far I would say they would expect that setting it in the inventory/host vars wouldn't result in the issue it does today.

When there is a huge inventory and you only want to target a very small sub-set with dynamic "hosts" field or limit. In this scenario there is a huge overhead of fetching all LAPS credentials.

That's what the search_base and filteroptions are for. You can use it to restrict to a specific OU/container or filter the objects further by name/some other attribute. This can result in the same type of query the lookup would do but more efficient as you send it only once.

Inventories in AWX are strictly separated from the playbook

I will concede I don't know too much about how AWX works but inventories being separated from the playbook sounds like the right thing to do. Your playbooks should really be host agnostic (outside of it being Windows vs something else). All connection vars should be sourced from the inventory IMO.

no-12 commented 7 months ago

Unfortunately this means to use the lookup properly means having your playbook with this explicit step resulting in some hard ties between the inventory and your play which is never nice.

I do not understand why using a lookup plugin in a set_facts module would tie the inventory to the play. I think it's the opposite. In my perspective the inventory and the ansible user/password are two completely separate things. And yes the ldap inventory plugin ties this two things nicely together because it useses LDAP to get both them from the same source. But you can have two different sources to these two things. For example a static inventory file and LDAP for the LAPS credentials.

Inventories in AWX are strictly separated from the playbook

With this i mean the inventory sync and the playbook run are separated. You can force them to run one after another, but this will result in a increased job run time if the inventory is not cached. And with the LAPS password changing quite often it is hard to refresh the cache in a timely manner.

All connection vars should be sourced from the inventory IMO.

This is a good point, but the thing with AWX is that it stores the host vars of each hosts it's DB. This is why I initially opened #100

The inventory and hosts data stored in the AWX DB are very useful for everyone to read except the ansible credential. We have 10k+ hosts in our AWX DB using a constructed inventory with filters would give us no ongoing Job history pf each host because on every new job with a different filter the constructed inventory would have other hosts.

The other problem with constructed inventories and so many hosts in AWX is performance. Storing so many hosts takes quite some time.

jborean93 commented 7 months ago

I suppose in the end it does hurt to have this functionality I just think we should try and point people towards the more efficient solution which is the inventory plugin. I've added a few comments in the review, please let me know what you think.

I do not understand why using a lookup plugin in a set_facts module would tie the inventory to the play.

My main point here is that your play now needs to have something like the below at the start of the playbook.

- hosts: windows
  gather_facts: false
  tasks:
  - set_fact:
      ansible_password: '{{ lookup("microsoft.ad.laps", ...) }}'

This now means the play is essentially set to only work with hosts that use LAPS or to use the hosts in question you have to use this playbook. Connection vars should IMO always be independent from a play itself and unfortunately that means with this lookup it will run on every task.

jborean93 commented 6 months ago

@no-12 are you still interested in pursuing this? I'm hoping to push out a new release soon and was checking to see if I should wait or just keep this for the release after.

no-12 commented 6 months ago

Yes I'm interested in getting this merged, but I'm not sure if I can get this in a "mergable" state in the next few days. So please do not wait for this pull request to get ready.

I wanted to work through your comments and add at least some rudimentary unit tests.

And I was also thinking about adding integration tests, but I'm not sure if this is doable or desirable. I have to look into the existing implementation of the ldap inventory plugin integration tests, but right now I don't fully understand the setup.

jborean93 commented 6 months ago

No worries, please let me know if you need help with the testing side, it is quite complex unfortunately but there is not too much we can do about that.

If the Windows images aren't new enough to test with you could potentially extend the schema manually and just set the values as a test. The encrypted ones will be harder to pull off but technically doable. If all else fails we could look at what the membership tests do in that it doesn't run in CI but provides a Vagrant setup that can allow us to run a test suite locally.

ansible-collections / microsoft.ad

LAPS lookup plugin #102

SUMMARY

ISSUE TYPE

COMPONENT NAME

Docs Build 📝