search

freeipa / freeipa-container

FreeIPA server in containers — images at https://quay.io/repository/freeipa/freeipa-server?tab=tags
https://quay.io/repository/freeipa/freeipa-server?tab=tags
Apache License 2.0
615 stars 259 forks source link

setting up a new replica with latest freeipa-server: failed in "retrieving schema for SchemaCache" #150

Closed kforner closed 6 years ago

kforner commented 7 years ago

I'm still stuck with my master and its replica running an old version of the freeipa-server docker (adelton/freeipa-server:latest-systemd). I was trying to setup a new replica using the latest freeipa/freeipa-server.

I had to add the --skip-conncheck option to ipa-data/ipa-replica-install-options because there is no ssh installed in the docker.

But it still fails at some point. Here are the IMHO relevant lines from /var/log/ipareplica-install.log, out of ~ 2000 lines:

2017-08-02T11:54:20Z DEBUG Successfully updated nsDS5ReplicaId.
2017-08-02T11:54:20Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-QUARTZBIO-COM.socket from SchemaCache
2017-08-02T11:54:20Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-QUARTZBIO-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject inst
ance at 0x7fdb699aed88>
2017-08-02T11:54:37Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 449, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 439, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 431, in __setup_replica
    r_bindpw=self.dm_password)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 1068, in setup_replication
    raise RuntimeError("Failed to start replication")
RuntimeError: Failed to start replication

2017-08-02T11:54:37Z DEBUG   [error] RuntimeError: Failed to start replication
2017-08-02T11:54:37Z DEBUG Destroyed connection context.ldap2_140580366919440
2017-08-02T11:54:37Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 318, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 310, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 334, in execute
    for nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 376, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 405, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 363, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 597, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 376, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 405, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 457, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 363, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1741, in main
    install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 375, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 818, in install
    ds = install_replica_ds(config, options, ca_enabled, remote_api)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 139, in install_replica_ds
    api=remote_api,
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 412, in create_replica
    self.start_creation(runtime=60)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 449, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 439, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 431, in __setup_replica
    r_bindpw=self.dm_password)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 1068, in setup_replication
    raise RuntimeError("Failed to start replication")

2017-08-02T11:54:37Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: Failed to start replication
2017-08-02T11:54:37Z ERROR Failed to start replication
2017-08-02T11:54:37Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
germanparente commented 7 years ago

Replication failed. So, probably the database in iparig is not populated but the service should be running on port 389. Is daemon ns-slapd running ?

kforner commented 7 years ago

Is daemon ns-slapd running ?

No.

# ps -fA
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 18:08 ?        00:00:00 /usr/sbin/init --show-status=false
root        41     1  0 18:08 ?        00:00:00 tail --silent -n 0 -f --retry /var/log/ipa-server-configure-first.log /var/log/ipa-server-run.log
root        54     1  0 18:08 ?        00:00:00 /usr/lib/systemd/systemd-journald
dbus        59     1  0 18:08 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root        64     0  0 18:08 ?        00:00:00 bash
root        81    64  0 18:08 ?        00:00:00 ps -fA
# ipactl status
Directory Service: STOPPED
ntpd Service: STOPPED
ipa: INFO: The ipactl command was successful
# ipactl start 
Upgrade required: please run ipa-server-upgrade command
Aborting ipactl

Shoud I run ipa-server-upgrade ??

germanparente commented 7 years ago

Since we are troubleshooting the port access, we could just start directory service:

systemctl start dirsrv@QUARTZBIO-COM.service

kforner commented 7 years ago

ok: on rig (inside the docker):

systemctl start dirsrv@QUARTZBIO-COM.service

on master (inside the docker):

# openssl s_client -connect 10.9.70.8:389
CONNECTED(00000003)
140314875082616:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 201 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1502296212
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
germanparente commented 7 years ago

Let's use hostname. Not ip address.

And particularly from ipa to rig.

When starting replication, it's ipa node that connects to rig to send the data.

For instance, in ipa:

ldapsearch -p 389 -h rig -D "cn=directory manager" -W -b "" -s base

kforner commented 7 years ago

on ipa (docker):

# ldapsearch -p 389 -h rig -D "cn=directory manager" -W -b "" -s base
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

#
dn:
objectClass: top
defaultnamingcontext: dc=quartzbio,dc=com
dataversion: 020170809162814
netscapemdsuffix: cn=ldap://dc=iparig,dc=quartzbio,dc=com:389
lastusn: -1
ipatopologypluginversion: 1.0
ipatopologyismanaged: off
ipaDomainLevel: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

Is it right ?

kforner commented 7 years ago

Some news: after a bit of a struggle, I managed to setup a new replica, but not on a docker, on a fresh fedora 26 laptop. In the process I learned taht a potential problem could be missing DNS entries (from freeIPA) DNS. So I created a new DNS entry for iparig, then relaunched the replica install, and it worked better. Still crashed, but much farther:

2017-08-15T13:36:52Z DEBUG   [39/42]: activating sidgen plugin
2017-08-15T13:36:52Z DEBUG   [40/42]: activating extdom plugin
2017-08-15T13:36:52Z DEBUG   [41/42]: tuning directory server
2017-08-15T13:36:56Z DEBUG wait_for_open_ports: localhost [389] timeout 300
2017-08-15T13:36:57Z DEBUG   [42/42]: configuring directory to start on boot
2017-08-15T13:36:58Z DEBUG   [1/27]: creating certificate server user
2017-08-15T13:36:58Z DEBUG   [2/27]: configuring certificate server instance
[CA]
2017-08-15T13:37:33Z DEBUG   [error] RuntimeError: CA configuration failed.

and 

Loading deployment configuration from /tmp/tmpDrGyti.
Installing CA into /var/lib/pki/pki-tomcat.
Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.
Importing certificates from /tmp/ca.p12:
---------------
5 entries found
---------------
  Certificate ID: e12228b23df70b90f2ea082de903908a93f3fba4
  Serial Number: 0x2
  Nickname: ocspSigningCert cert-pki-ca
  Subject DN: CN=OCSP Subsystem,O=QUARTZBIO.COM
  Issuer DN: CN=Certificate Authority,O=QUARTZBIO.COM
  Has Key: true

  Certificate ID: 8777903969dce3806a014d746748b4f144947b4e
  Serial Number: 0x4
  Nickname: subsystemCert cert-pki-ca
  Subject DN: CN=CA Subsystem,O=QUARTZBIO.COM
  Issuer DN: CN=Certificate Authority,O=QUARTZBIO.COM
  Has Key: true

  Certificate ID: 3ee52926a3371a627ad3da893bee4d8966d04131
  Serial Number: 0x1
  Nickname: caSigningCert cert-pki-ca
  Subject DN: CN=Certificate Authority,O=QUARTZBIO.COM
  Issuer DN: CN=Certificate Authority,O=QUARTZBIO.COM
  Has Key: true

  Certificate ID: 8a7c017f1b47027d4ac9d16992f5385d203f06d3
  Serial Number: 0x3
  Nickname: Server-Cert cert-pki-ca
  Subject DN: CN=ipa.quartzbio.com,O=QUARTZBIO.COM
  Issuer DN: CN=Certificate Authority,O=QUARTZBIO.COM
  Has Key: true

  Certificate ID: 833075eac0ed3dac5932823ccc29b5b4ffc074b
  Serial Number: 0x5
  Nickname: auditSigningCert cert-pki-ca
  Subject DN: CN=CA Audit,O=QUARTZBIO.COM
  Issuer DN: CN=Certificate Authority,O=QUARTZBIO.COM
  Has Key: true
---------------
Import complete
---------------
Imported certificates in /etc/pki/pki-tomcat/alias:

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
Server-Cert cert-pki-ca                                      u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu

Installation failed:
<!DOCTYPE html><html><head><title>Apache Tomcat/8.0.43 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-siz
e:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-
size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:T
ahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style> </head
><body><h1>HTTP Status 500 - java.lang.Exception: Missing system certificate: ocspSigningCert cert-pki-ca</h1><div class="line"></div><p><b>type</b> Exception report</p><p><b>message</b
> <u>java.lang.Exception: Missing system certificate: ocspSigningCert cert-pki-ca</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfill
ing this request.</u></p><p><b>exception</b></p><pre>org.jboss.resteasy.spi.UnhandledException: java.lang.Exception: Missing system certificate: ocspSigningCert cert-pki-ca
    org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:77)
    org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:220)
    org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:175)

and from /var/log/pki/pki-tomcat/ca/debug:

[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: ConfigurationUtils.importKeyCert()
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: Deleting existing certificates:
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Certificate caSigningCert cert-pki-ca
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Certificate ocspSigningCert cert-pki-ca
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Certificate subsystemCert cert-pki-ca
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Certificate auditSigningCert cert-pki-ca
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: Master certs:
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:  - preop.cert.signing.dn: cn=Certificate Authority,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:  - preop.cert.ocsp_signing.dn: cn=OCSP Subsystem,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:  - preop.cert.subsystem.dn: cn=CA Subsystem,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:  - preop.cert.audit_signing.dn: cn=CA Audit,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: Importing new keys:
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Key: CN=OCSP Subsystem,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Find cert with subject DN CN=OCSP Subsystem,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Certificate is missing/removed, ignore key
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Key: CN=CA Subsystem,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Find cert with subject DN CN=CA Subsystem,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Certificate is missing/removed, ignore key
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Key: CN=Certificate Authority,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Find cert with subject DN CN=Certificate Authority,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Imported cert 1
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Key: CN=ipa.quartzbio.com,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Key not in master list, ignore key
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Key: CN=CA Audit,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Find cert with subject DN CN=CA Audit,O=QUARTZBIO.COM
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Certificate is missing/removed, ignore key
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: Importing new certificates:
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: - Certificate: caSigningCert cert-pki-ca
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: Property preop.master.signing.nickname: caSigningCert cert-pki-ca
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   CA signing cert: true
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]:   Imported cert 1
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: SystemConfigService: verify certificates
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: ConfigurationUtils.verifySystemCertificates(): checking certificate caSigningCert cert-pki-ca
[15/Aug/2017:13:37:33][http-bio-8443-exec-3]: ConfigurationUtils.verifySystemCertificates(): checking certificate ocspSigningCert cert-pki-ca
java.lang.Exception: Missing system certificate: ocspSigningCert cert-pki-ca
    at com.netscape.cms.servlet.csadmin.ConfigurationUtils.verifySystemCertificates(ConfigurationUtils.java:1014)
    at org.dogtagpki.server.rest.SystemConfigService.configureClone(SystemConfigService.java:898)
    at org.dogtagpki.server.rest.SystemConfigService.configureSubsystem(SystemConfigService.java:1019)
    at org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:164)
felipevolpone commented 7 years ago

hi @kforner, since this doesn't seem an issue related directly to ipa container, could you post it on freeipa-users list: freeipa-users@lists.fedorahosted.org? There, there are more people able to help you.

adelton commented 6 years ago

Hello folks who have followed this issue -- is there anything that might be needed specifically for the container case?

adelton commented 6 years ago

We might have lost traction here, closing.

germanparente commented 6 years ago

Hi,

Sorry, I have no idea from my side.

Regards,

German

On Fri, Jul 20, 2018, 1:54 PM Jan Pazdziora notifications@github.com wrote:

We might have lost traction here, closing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/freeipa/freeipa-container/issues/150#issuecomment-406578548, or mute the thread https://github.com/notifications/unsubscribe-auth/AXnZa2s3T8qr8EUBik6SWHRQFPUq6mDkks5uIcTqgaJpZM4Oq12U .