search

abbra / freeipa

Mirror of FreeIPA, an integrated security information management solution
http://www.freeipa.org
GNU General Public License v3.0
2 stars 1 forks source link

gc-wip: ipaserver uninstallation failed with GC installed #63

Closed wladich closed 3 years ago

wladich commented 4 years ago

The issue was seen only once so far.

Test results: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/dbd7866a-17cf-11eb-be89-fa163e4ad552/

Test output:

DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:513 RUN ['ipa-server-install', '--uninstall', '-U', '--ignore-topology-disconnect', '--ignore-last-of-role']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Updating DNS system records
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Forcing removal of master.ipa.test
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 ------------------------------------
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Ignoring topology connectivity errors.
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Deleted IPA server "master.ipa.test"
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 ------------------------------------
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Removing Global Catalog
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Unconfiguring ipa-gcsyncd
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Unconfiguring global catalog
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 Operations error: 
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:557 The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd704:transport.py:217 Exit code: 1
ERROR    ipatests.pytest_ipa.integration.host.Host.master.cmd704:host.py:199 stderr: Forcing removal of master.ipa.test
Ignoring topology connectivity errors.
Operations error: 
The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information

http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/dbd7866a-17cf-11eb-be89-fa163e4ad552/test_integration-test_global_catalog.py-TestGlobalCatalogInstallation-uninstall/master.ipa.test/var/log/ipaserver-uninstall.log.gz:

2020-10-26T22:26:36Z DEBUG Stop of ipa-gcsyncd.service complete
2020-10-26T22:26:36Z DEBUG Starting external process
2020-10-26T22:26:36Z DEBUG args=['/bin/systemctl', 'disable', 'ipa-gcsyncd.service']
2020-10-26T22:26:37Z DEBUG Process finished, return code=0
2020-10-26T22:26:37Z DEBUG stdout=
2020-10-26T22:26:37Z DEBUG stderr=
2020-10-26T22:26:37Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2020-10-26T22:26:37Z DEBUG Unconfiguring global catalog
2020-10-26T22:26:37Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2020-10-26T22:26:37Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2020-10-26T22:26:37Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2020-10-26T22:26:37Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2020-10-26T22:26:37Z DEBUG raw: dnsrecord_del(<DNS name ipa.test.>, '_ldap._tcp.Default-First-Site-Name._sites.gc._msdcs', del_all=True, version='2.239')
2020-10-26T22:26:37Z DEBUG dnsrecord_del(<DNS name ipa.test.>, <DNS name _ldap._tcp.Default-First-Site-Name._sites.gc._msdcs>, del_all=True, structured=False, raw=False, version='2.239')
2020-10-26T22:26:37Z DEBUG raw: dnsrecord_delentry(<DNS name ipa.test.>, (<DNS name _ldap._tcp.Default-First-Site-Name._sites.gc._msdcs>,), version='2.239')
2020-10-26T22:26:37Z DEBUG dnsrecord_delentry(<DNS name ipa.test.>, (<DNS name _ldap._tcp.Default-First-Site-Name._sites.gc._msdcs>,), continue=False, version='2.239')
2020-10-26T22:26:37Z DEBUG flushing ldapi://%2Frun%2Fslapd-IPA-TEST.socket from SchemaCache
2020-10-26T22:26:37Z DEBUG retrieving schema for SchemaCache url=ldapi://%2Frun%2Fslapd-IPA-TEST.socket conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7f5869e24370>
2020-10-26T22:26:37Z DEBUG raw: dnsrecord_del(<DNS name ipa.test.>, '_ldap._tcp.gc._msdcs', del_all=True, version='2.239')
2020-10-26T22:26:37Z DEBUG dnsrecord_del(<DNS name ipa.test.>, <DNS name _ldap._tcp.gc._msdcs>, del_all=True, structured=False, raw=False, version='2.239')
2020-10-26T22:26:37Z DEBUG raw: dnsrecord_delentry(<DNS name ipa.test.>, (<DNS name _ldap._tcp.gc._msdcs>,), version='2.239')
2020-10-26T22:26:37Z DEBUG dnsrecord_delentry(<DNS name ipa.test.>, (<DNS name _ldap._tcp.gc._msdcs>,), continue=False, version='2.239')
2020-10-26T22:26:37Z DEBUG raw: dnsrecord_del(<DNS name ipa.test.>, '_gc._tcp.Default-First-Site-Name._sites', del_all=True, version='2.239')
2020-10-26T22:26:37Z DEBUG dnsrecord_del(<DNS name ipa.test.>, <DNS name _gc._tcp.Default-First-Site-Name._sites>, del_all=True, structured=False, raw=False, version='2.239')
2020-10-26T22:26:37Z DEBUG raw: dnsrecord_delentry(<DNS name ipa.test.>, (<DNS name _gc._tcp.Default-First-Site-Name._sites>,), version='2.239')
2020-10-26T22:26:37Z DEBUG dnsrecord_delentry(<DNS name ipa.test.>, (<DNS name _gc._tcp.Default-First-Site-Name._sites>,), continue=False, version='2.239')
2020-10-26T22:28:17Z DEBUG Unhandled LDAPError: OPERATIONS_ERROR: {'msgtype': 107, 'msgid': 26, 'result': 1, 'desc': 'Operations error', 'ctrls': []}
2020-10-26T22:28:17Z DEBUG   File "/usr/lib/python3.8/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.8/site-packages/ipapython/install/cli.py", line 340, in run
    return cfgr.run()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 655, in _configure
    next(executor)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.8/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.8/site-packages/ipapython/install/common.py", line 73, in _uninstall
    for unused in self._uninstaller(self.parent):
  File "/usr/lib/python3.8/site-packages/ipaserver/install/server/__init__.py", line 592, in main
    uninstall(self)
  File "/usr/lib/python3.8/site-packages/ipaserver/install/server/install.py", line 275, in decorated
    func(installer)
  File "/usr/lib/python3.8/site-packages/ipaserver/install/server/install.py", line 1173, in uninstall
    gc.uninstall(fstore)
  File "/usr/lib/python3.8/site-packages/ipaserver/install/gc.py", line 149, in uninstall
    gcinstance.GCInstance(fstore=fstore).uninstall()
  File "/usr/lib/python3.8/site-packages/ipaserver/install/gcinstance.py", line 796, in uninstall
    self.__remove_gc_dns_records()
  File "/usr/lib/python3.8/site-packages/ipaserver/install/gcinstance.py", line 750, in __remove_gc_dns_records
    api.Command.dnsrecord_del(
  File "/usr/lib/python3.8/site-packages/ipalib/frontend.py", line 471, in __call__
    return self.__do_call(*args, **options)
  File "/usr/lib/python3.8/site-packages/ipalib/frontend.py", line 499, in __do_call
    ret = self.run(*args, **options)
  File "/usr/lib/python3.8/site-packages/ipalib/frontend.py", line 821, in run
    return self.execute(*args, **options)
  File "/usr/lib/python3.8/site-packages/ipaserver/plugins/dns.py", line 3932, in execute
    result = self.obj.methods.delentry(*keys,
  File "/usr/lib/python3.8/site-packages/ipalib/frontend.py", line 471, in __call__
    return self.__do_call(*args, **options)
  File "/usr/lib/python3.8/site-packages/ipalib/frontend.py", line 499, in __do_call
    ret = self.run(*args, **options)
  File "/usr/lib/python3.8/site-packages/ipalib/frontend.py", line 821, in run
    return self.execute(*args, **options)
  File "/usr/lib/python3.8/site-packages/ipaserver/plugins/baseldap.py", line 1603, in execute
    delete_entry(pkey)
  File "/usr/lib/python3.8/site-packages/ipaserver/plugins/baseldap.py", line 1578, in delete_entry
    self._exc_wrapper(nkeys, options, ldap.delete_entry)(dn)
  File "/usr/lib/python3.8/site-packages/ipaserver/plugins/baseldap.py", line 1120, in wrapped
    return func(*call_args, **call_kwargs)
  File "/usr/lib/python3.8/site-packages/ipaserver/plugins/baseldap.py", line 1128, in exc_func
    return callback(
  File "/usr/lib/python3.8/site-packages/ipaserver/plugins/baseldap.py", line 1624, in exc_callback
    raise exc
  File "/usr/lib/python3.8/site-packages/ipaserver/plugins/baseldap.py", line 1120, in wrapped
    return func(*call_args, **call_kwargs)
  File "/usr/lib/python3.8/site-packages/ipapython/ipaldap.py", line 1723, in delete_entry
    self.conn.delete_s(str(dn))
  File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3.8/site-packages/ipapython/ipaldap.py", line 1161, in error_handler
    raise errors.DatabaseError(desc=desc, info=info)

2020-10-26T22:28:17Z DEBUG The ipa-server-install command failed, exception: DatabaseError: Operations error: 
2020-10-26T22:28:17Z ERROR Operations error: 
2020-10-26T22:28:17Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
flo-renaud commented 3 years ago

I wonder whether it's happening because there is a replica. The uninstallation of one of the servers removes the DNS records for the global catalog and it should be done only when the last GC instance is removed.

wladich commented 3 years ago

As I do not know a reliable way of reproducing the issue, let's close it.