ipaclient_setup_automount module breaks funcitioning automounts in repair mode

[akors]

Hi there!

Using

Controller:

• Ansible 2.9.1
• Python 3.7.5
• Fedora 31
• freeipa-ansible 0.1.6-2

Target:

• Python 3.6.8
• CentOS 8

I have a shell snippet in /etc/profile.d that contains an error that should be unrelated to ansible-freeipa. For reference, this is the error message that I can see every time I'm starting a shell: ERROR: Directory '/opt/mystuff/modulefiles.

Now to the actual issue: When I run the ipaclient role like this:

- name: Playbook to configure IPA clients with username/password
  hosts: all
  become: true
  roles:
  - role: ipaclient
    state: present

With an inventory where ipaautomount_location=default, I expect my automounts to be configured. The play runs, the task "ipaclient : Install - Configure automount" claims to be changed with no errros, but the target host does not have automounts.

When I supply -vvv to ansible, finally I get what's wrong:

TASK [ipaclient : Install - Configure automount] *********************************************************************************************************************************************************************************************
task path: /usr/share/ansible/roles/ipaclient/tasks/install.yml:321
Using module file /usr/share/ansible/roles/ipaclient/library/ipaclient_setup_automount.py
Pipelining is enabled.
<mypc00608.local> ESTABLISH SSH CONNECTION FOR USER: root
<mypc00608.local> SSH: EXEC sshpass -d9 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/home/akorsunsky/.ansible/cp/afe9f6e7f9 mypc00608.local '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=wwyswneqhoxqwvbtlyvweckrjsnjkusr] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-wwyswneqhoxqwvbtlyvweckrjsnjkusr ; /usr/libexec/platform-python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<mypc00608.local> (0, b'\n{"changed": true, "invocation": {"module_args": {"servers": ["infrasrv.local", "mypc00115.local"], "sssd": true, "automount_location": "default"}}}\n', b"ERROR: Directory '/opt/mystuff/modulefiles' not found\nAutomount configuration failed: CalledProcessError(Command ['/usr/sbin/ipa-client-automount', '--debug', '-U', '--location', 'default', '--server', 'infrasrv.local'] returned non-zero exit status 3: '')\n")
changed: [mypc00608] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "automount_location": "default",
            "servers": [
                "infrasrv.local",
                "mypc00115.local"
            ],
            "sssd": true
        }
    }
}

Note how the script notices that Automount configuration failed, the return status is non-zero but the module does not throw an error. First of all, this should definitely be a failure.

Second, the reason why it failed is further in: ERROR: Directory '/opt/mystuff/modulefiles' not found, which is the issue with broken /etc/profile.d snippet. I can fix that, but I do wonder why that's an issue in the first place, since it shouldn't affect the script itself, only the shell it runs in.

[akors]

Fixing my shell error, the module still fails and still claims it worked. Running /usr/sbin/ipa-client-automount manually on the target host shows:

An automount location is already configured

After /usr/sbin/ipa-client-automount --uninstall, the client install play works again. I don't know why the host was left in a state where automounts were "alread configured" according, but not actually working.

[akors]

So this seems to have nothing to do with my broken shell.

I can reliably break functioning automounts by passing the ipaclient_allow_repair=on to the role. This can be fixed by running /usr/sbin/ipa-client-automount --uninstall on the target host and then running the role again.

[t-woerner]

Do you have a difference in the automount configuration files?

[akors]

@t-woerner

Do you have a difference in the automount configuration files?

What do you mean by "difference"? I have other automount locations configured via files, but that should be irrelevant I hope.

The automounts from FreeIPA are default configuration at the "default" location.

[t-woerner]

The use of the module with repair mode seems to either corrupt something like changing a configuration file in a bad way or stop or deactivate the service. It would be good to have some more information to understand what is happening.

[akors]

Here is the change that happens when running the role with ipaclient_allow_repair=on

diff -u3 -r before-setup-automount/sssd/sssd.conf after-setup-automount/sssd/sssd.conf
--- before-setup-automount/sssd/sssd.conf   2019-12-02 15:23:19.889905218 +0100
+++ after-setup-automount/sssd/sssd.conf    2019-12-02 15:28:16.939990528 +0100
@@ -11,10 +11,8 @@
 cache_credentials = True
 ldap_tls_cacert = /etc/ipa/ca.crt
 krb5_store_password_if_offline = True
-autofs_provider = ipa
-ipa_automount_location = default
 [sssd]
-services = nss, pam, ssh, sudo, autofs
+services = nss, pam, ssh, sudo

 domains = rcpe.pharm.local
 [nss]

Output with -vvv:

ASK [ipaclient : Install - Configure automount] *********************************************************************************************************************************************************************************************
task path: /usr/share/ansible/roles/ipaclient/tasks/install.yml:321
Using module file /usr/share/ansible/roles/ipaclient/library/ipaclient_setup_automount.py
Pipelining is enabled.
<rcpepc00608.local> ESTABLISH SSH CONNECTION FOR USER: rcpe-root
<rcpepc00608.local> SSH: EXEC sshpass -d9 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="rcpe-root"' -o ConnectTimeout=10 -o ControlPath=/home/akorsunsky/.ansible/cp/afe9f6e7f9 rcpepc00608.local '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=gddxdrqihzycpdiroqoliypzoqfllhtr] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-gddxdrqihzycpdiroqoliypzoqfllhtr ; /usr/libexec/platform-python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<rcpepc00608.local> (0, b'\n{"changed": true, "invocation": {"module_args": {"servers": ["infrasrv.local", "rcpepc00115.local"], "sssd": true, "automount_location": "default"}}}\n', b"Automount configuration failed: CalledProcessError(Command ['/usr/sbin/ipa-client-automount', '--debug', '-U', '--location', 'default', '--server', 'infrasrv.local'] returned non-zero exit status 3: '')\n")
changed: [rcpepc00608] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "automount_location": "default",
            "servers": [
                "infrasrv.local",
                "rcpepc00115.local"
            ],
            "sssd": true
        }
    }
}

First of all, the automounts seem to get unconfigured with repair mode.

Second, the install script actually fails but the task is reported to succeed. Is it possible that something gets unconfigured during error handling or something?

[akors]

This looks highly suspicious:

https://github.com/freeipa/freeipa/blob/master/ipaclient/install/ipa_client_automount.py#L586

def configure_automount():
    try:
    # ...
    except Exception as e:
        logger.debug('Raised exception %s', e)
        print("Installation failed. Rolling back changes.")
        uninstall(fstore, statestore)
        return 1

    return 0

My guess is that it fails because it is already configured, but then runs "uninstall" and destroys the previous configuration.

[chr15p]

I think we have hit the same (or very similar problem), we run a client install via the role, it sets up automount and all works correctly, we then run the same playbook a second time and automount stops working entirely, and nothing gets mounted at all.

Some poking around in the code it appears that the second time the role is run it resets the sssd.conf file to a default configuration without the automount settings, and then runs the ipaclient_setup_automount module to reconfigure automounting. The ipaclient_setup_automount module runs /usr/sbin/ipa-client-automount behind the scenes. /usr/sbin/ipa-client-automount is desgined so if it detects it has been run before then it exits with the message "An automount location is already configured" without making any changes.

This means the automount configuration (e.g. autofs_provider = ipa and ipa_automount_location = default, and adding autofs to the services list) does not get added back in to to sssd.conf and so sssd stops providing automount information to clients deamons such as autofs.

The workaround for us was to add those 3 settings back into sssd.conf and restart sssd.service and autofs.service

[ewenmcneill]

I believe I also ran into this, trying to convert some enrolled IPA/IdM clients from not using automount to using automount, by rerunning the client enrolment playbook with:

ansible ...  HOSTNAME ... -e ipaclient_allow_repair=on

after ensuring ipaautomount_location was being set for that host by the ansible variables. That run did change from "skipping: automount" to "changed: automount", and /etc/nsswitch.conf seemed to get updated, but /etc/sssd/sssd.conf didn't have the right entries in it so the autofs part wasn't started.

Based on https://github.com/freeipa/ansible-freeipa/issues/151#issuecomment-704395124 and https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/identity_management_guide/configuring-automount, I ended up creating an /etc/sssd/conf.d/52-sssd-idm-autofs.conf from a template (below) that explicitly populated the required settings for the autofs part of sssd to start. That was enough to have the automounts visible from IdM.

It'd be nice if the ipaclient enrollment (especially in repair mode) was more idempotent -- ie, rather than trying to remove/re-add things like automounting it just went through ensuring everything was actually present where/how it should be, and just added things that had never been added previously. Because the "remove things and try to put them back" approach seems unnecessarily fragile, and less likely to "converge on the desired settings".

Ewen

# Configure autofs service of sssd
#
# This *may* also be configured in sssd.conf, by the idm-client enrollment,
# but due to a bug:
#
# https://github.com/freeipa/ansible-freeipa/issues/151
#
# it does not get properly configured if the client is ever updated/repaired
# so we also explicitly set it up.  (Unfortunately this means that we are
# hard coding the list of sssd services here, due to the clumsy
# configuration file syntax sssd uses; but the list IdM configures is
# fairly standard, so this should be fairly safe.)

[domain/{{ipaclient_domain|default(ipaserver_domain)}}]
autofs_provider = ipa
ipa_automount_location = {{automount_location}}

[sssd]
services = nss, pam, ssh, sudo, autofs

[okapia]

I'm also hitting this same problem. /usr/sbin/ipa-client-automount does return a status of 3 along with the "An automount location is already configured" message so the situation should be detectable. What that doesn't do is handle changes such as the automount location changing. For my own deployment playbook, it's tempting to add a task which runs /usr/sbin/ipa-client-automount --uninstall first.

Not being in a position to run DNS and have servers auto-detected, I need to rerun the role on systems where the list of IPA servers/replicas is out-of-date.