Incorrect procedure for replica uninstallation

The correct procedure to uninstall IPA replica is (cited from documentation):

To uninstall server.example.com:

On another server, use the ipa server-del command to delete server.example.com from the topology:
```
[root@another_server ~]# ipa server-del server.example.com
```
On server.example.com, use the ipa-server-install --uninstall command:
```
[root@server ~]# ipa-server-install --uninstall
```
Make sure all name server (NS) DNS records pointing to server.example.com are deleted from your DNS zones. This applies regardless of whether you use integrated DNS managed by IdM or external DNS.

Documentation for reference: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/uninstalling_ipa_servers

Looking at https://github.com/freeipa/ansible-freeipa/blob/master/roles/ipareplica/tasks/uninstall.yml it seems that only step 2 is done.

This may lead to race-condition type of issues. E.g. my recent CI run: With playbook matching https://github.com/freeipa/ansible-freeipa/blob/master/playbooks/uninstall-cluster.yml

PLAY [Uninstall IPA clients] ***************************************************
skipping: no hosts matched

PLAY [Uninstall IPA replicas] **************************************************

TASK [Gathering Facts] *********************************************************
ok: [replica1.ipadomain.test]
ok: [replica2.ipadomain.test]

TASK [ipareplica : Import variables specific to distribution] ******************
ok: [replica1.ipadomain.test] => (item=/usr/share/ansible/roles/ipareplica/vars/RedHat-8.yml)
ok: [replica2.ipadomain.test] => (item=/usr/share/ansible/roles/ipareplica/vars/RedHat-8.yml)

TASK [ipareplica : Install IPA replica] ****************************************
skipping: [replica1.ipadomain.test]
skipping: [replica2.ipadomain.test]

TASK [ipareplica : Uninstall IPA replica] **************************************
included: /usr/share/ansible/roles/ipareplica/tasks/uninstall.yml for replica1.ipadomain.test, replica2.ipadomain.test

TASK [ipareplica : Uninstall - Uninstall IPA replica] **************************
changed: [replica1.ipadomain.test]
changed: [replica2.ipadomain.test]

PLAY [Uninstall IPA server] ****************************************************

TASK [Gathering Facts] *********************************************************
ok: [master.ipadomain.test]

TASK [ipaserver : Import variables specific to distribution] *******************
ok: [master.ipadomain.test] => (item=/usr/share/ansible/roles/ipaserver/vars/RedHat-8.yml)

TASK [ipaserver : Install IPA server] ******************************************
skipping: [master.ipadomain.test]

TASK [ipaserver : Uninstall IPA server] ****************************************
included: /usr/share/ansible/roles/ipaserver/tasks/uninstall.yml for master.ipadomain.test

TASK [ipaserver : Uninstall - Uninstall IPA server] ****************************
ok: [master.ipadomain.test]

PLAY RECAP *********************************************************************
master.ipadomain.test      : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
replica1.ipadomain.test    : ok=4    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
replica2.ipadomain.test    : ok=4    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

I.e. it seems that whole topology was removed. But CI failed on checking if master was removed and indeed, ipactl status was saying that it is running. Looking into ipaserver-uninstall.log, I see:

2019-09-17T20:32:26Z DEBUG raw: config_show(version='2.233')
2019-09-17T20:32:26Z DEBUG config_show(rights=False, all=False, raw=False, version='2.233')
2019-09-17T20:32:26Z DEBUG   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 179, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 340, in run
    return cfgr.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 358, in run
    self.validate()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 368, in validate
    for _nothing in self._validator():
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 633, in _configure
    next(validator)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 73, in _uninstall
    for unused in self._uninstaller(self.parent):
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py", line 561, in main
    uninstall_check(self)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/install.py", line 255, in decorated
    func(installer)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/install.py", line 1076, in uninstall_check
    ca.uninstall_check(options)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ca.py", line 146, in uninstall_check
    raise ScriptError("Aborting uninstall operation.")

2019-09-17T20:32:26Z DEBUG The ipa-server-install command failed, exception: ScriptError: Aborting uninstall operation.
2019-09-17T20:32:26Z ERROR Aborting uninstall operation.
2019-09-17T20:32:26Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information

Which in essence means that master uninstallation was stopped on topology check. It doesn't say what is the issue. So here I'll be guessing saying that ipa server-del result (which is also part of uninstallation on replica) was not replicated to master before the replication agreements were destroyed (this is the reason why the correct procedure is to run ipa server-del on another host first). Thus master is thinking that there is still some IPA server, but that IPA server is neither CA renewal master nor CRL master or it doesn't have CA at all and thus it will fail on topology check.

Used inventory in the test (with variables replaced):

[ipaserver]
{{ ipa_domain.master.name }}

[ipareplicas]
{% for replica in ipa_domain.replicas %}
{{ replica.name }}
{% endfor %}

[ipaserver:vars]
ipaserver_setup_dns=yes
ipaserver_no_firewalld=no
ipaserver_forwarders={{ dns_forwarder }}
ipaserver_no_dnssec_validation=yes

ipaadmin_password={{ admin_password }}
ipadm_password={{ dirman_password }}
ipaserver_domain={{ ipa_domain.name }}
ipaserver_realm={{ ipa_domain.realm }}

[ipareplicas:vars]
ipaadmin_password={{ admin_password }}
ipadm_password={{ dirman_password }}
ipareplica_domain={{ ipa_domain.name }}
ipaserver_no_dnssec_validation=yes

Expected behavior: Given that Ansible is operating on a cluster, I'd expect that it will remove the replicas in the documented way (this is something which standard uninstaller cannot do).

freeipa / ansible-freeipa

Incorrect procedure for replica uninstallation #118