search

freeipa / freeipa-container

FreeIPA server in containers — images at https://quay.io/repository/freeipa/freeipa-server?tab=tags
https://quay.io/repository/freeipa/freeipa-server?tab=tags
Apache License 2.0
615 stars 259 forks source link

FreeIPA Replica Server Container Exit #92

Closed maesterX closed 5 years ago

maesterX commented 8 years ago

Hi,

I am currently playing with a multi-master deployment of FreeIPA version 4.3.1 (CentOS-7 upstream) on separate Docker hosts. The first master container spins up just fine (Host1) and using a OTP to spin up the replica master container (Host2), the process executes then exits while trying to restart named.

This is a snippet of the install process:

...
Created symlink from /etc/systemd/system/named.service to /dev/null.

ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG      [8/8]: changing resolv.conf to point to ourselves
  [8/8]: changing resolv.conf to point to ourselves
ipa         : DEBUG    Backing up system configuration file '/etc/resolv.conf'
ipa         : DEBUG    Saving Index File to '/var/lib/ipa/sysrestore/sysrestore.index'
ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG    Done configuring DNS (named).
Done configuring DNS (named).
ipa         : DEBUG    Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl stop ipa-dnskeysyncd.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=
ipa         : DEBUG    stderr=Failed to open /dev/tty: No such device or address
Failed to open /dev/tty: No such device or address

ipa         : DEBUG    Configuring DNS key synchronization service (ipa-dnskeysyncd)
Configuring DNS key synchronization service (ipa-dnskeysyncd)
ipa         : DEBUG      [1/7]: checking status
  [1/7]: checking status
ipa.ipapython.ipaldap.SchemaCache: DEBUG    flushing ldapi://%2fvar%2frun%2fslapd-CENGNLOCAL-CA.socket from SchemaCache
ipa.ipapython.ipaldap.SchemaCache: DEBUG    retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-CENGNLOCAL-CA.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0xe553cf                                           8>
ipa         : DEBUG    Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
ipa         : DEBUG    Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG      [2/7]: setting up bind-dyndb-ldap working directory
  [2/7]: setting up bind-dyndb-ldap working directory
ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG      [3/7]: setting up kerberos principal
  [3/7]: setting up kerberos principal
ipa         : DEBUG    Removing service keytab: /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=kadmin.local -q addprinc -randkey ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA -x ipa-setup-override-restrictions
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=Authenticating as principal host/admin@CENGNLOCAL.CA with password.
Principal "ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA" created.

ipa         : DEBUG    stderr=WARNING: no policy specified for ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA; defaulting to no policy

ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=kadmin.local -q ktadd -k /etc/ipa/dnssec/ipa-dnskeysyncd.keytab ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA -x ipa-setup-override-restrictions
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=Authenticating as principal host/admin@CENGNLOCAL.CA with password.
Entry for principal ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/ipa/dnssec/ipa-dnskeysyncd.keytab.
Entry for principal ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/ipa/dnssec/ipa-dnskeysyncd.keytab.
Entry for principal ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/ipa/dnssec/ipa-dnskeysyncd.keytab.
Entry for principal ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/ipa/dnssec/ipa-dnskeysyncd.keytab.
Entry for principal ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/etc/ipa/dnssec/ipa-dnskeysyncd.keytab.
Entry for principal ipa-dnskeysyncd/ipa2.cengnlocal.ca@CENGNLOCAL.CA with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/etc/ipa/dnssec/ipa-dnskeysyncd.keytab.

ipa         : DEBUG    stderr=
ipa.ipapython.ipaldap.SchemaCache: DEBUG    flushing ldapi://%2fvar%2frun%2fslapd-CENGNLOCAL-CA.socket from SchemaCache
ipa.ipapython.ipaldap.SchemaCache: DEBUG    retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-CENGNLOCAL-CA.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0xe553e6                                           0>
ipa         : DEBUG      duration: 1 seconds
ipa         : DEBUG      [4/7]: setting up SoftHSM
  [4/7]: setting up SoftHSM
ipa         : DEBUG    Creating /var/lib/ipa/dnssec directory
ipa         : DEBUG    Creating new softhsm config file
ipa         : DEBUG    Backing up system configuration file '/etc/sysconfig/named'
ipa         : DEBUG    Saving Index File to '/var/lib/ipa/sysrestore/sysrestore.index'
ipa         : DEBUG    Creating tokens /var/lib/ipa/dnssec/tokens directory
ipa         : DEBUG    Saving user PIN to /var/lib/ipa/dnssec/softhsm_pin
ipa         : DEBUG    Saving SO PIN to /etc/ipa/dnssec/softhsm_pin_so
ipa         : DEBUG    Initializing tokens
ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/usr/bin/softhsm2-util --init-token --slot 0 --label ipaDNSSEC --pin XXXXXXXX --so-pin XXXXXXXX
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=The token has been initialized.

ipa         : DEBUG    stderr=
ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG      [5/7]: adding DNSSEC containers
  [5/7]: adding DNSSEC containers
ipa.ipapython.ipaldap.SchemaCache: DEBUG    flushing ldapi://%2fvar%2frun%2fslapd-CENGNLOCAL-CA.socket from SchemaCache
ipa.ipapython.ipaldap.SchemaCache: DEBUG    retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-CENGNLOCAL-CA.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x12e041                                           70>
ipa         : INFO     DNSSEC container exists (step skipped)
ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG      [6/7]: creating replica keys
  [6/7]: creating replica keys
ipa         : DEBUG    Creating replica's key pair
ipa         : DEBUG    Storing replica public key to LDAP, ipk11UniqueId=autogenerate,cn=keys,cn=sec,cn=dns,dc=cengnlocal,dc=ca
ipa         : DEBUG    Replica public key stored
ipa         : DEBUG    Setting CKA_WRAP=False for old replica keys
ipa         : DEBUG    Changing ownership of token files
ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG      [7/7]: configuring ipa-dnskeysyncd to start on boot
  [7/7]: configuring ipa-dnskeysyncd to start on boot
ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl disable ipa-dnskeysyncd.service
ipa         : DEBUG    Process finished, return code=1
ipa         : DEBUG    stdout=
ipa         : DEBUG    stderr=Failed to open /dev/tty: No such device or address
Failed to execute operation: Too many levels of symbolic links

ipa         : DEBUG      duration: 0 seconds
ipa         : DEBUG    Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl restart ipa-dnskeysyncd.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=
ipa         : DEBUG    stderr=Failed to open /dev/tty: No such device or address
Failed to open /dev/tty: No such device or address

ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl is-active ipa-dnskeysyncd.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=active

ipa         : DEBUG    stderr=
ipa         : DEBUG    Restarting named
Restarting named
ipa         : DEBUG    Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl is-active named-pkcs11.service
ipa         : DEBUG    Process finished, return code=3
ipa         : DEBUG    stdout=unknown

ipa         : DEBUG    stderr=
ipa         : DEBUG    Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
ipa         : DEBUG    Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl restart named-pkcs11.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=
ipa         : DEBUG    stderr=Failed to open /dev/tty: No such device or address
Failed to open /dev/tty: No such device or address

ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl is-active named-pkcs11.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=active

ipa         : DEBUG    stderr=

ipa.ipalib.plugins.dns.dnsconfig_show: DEBUG    raw: dnsconfig_show(version=u'2.164')
ipa.ipalib.plugins.dns.dnsconfig_show: DEBUG    dnsconfig_show(rights=False, all=False, raw=False, version=u'2.164')
Global DNS configuration in LDAP server is empty
You can use 'dnsconfig-mod' command to set global DNS options that
would override settings in local named.conf files

ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl enable ipa.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=
ipa         : DEBUG    stderr=Failed to open /dev/tty: No such device or address
Created symlink from /etc/systemd/system/multi-user.target.wants/ipa.service to /usr/lib/systemd/system/ipa.service.

ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl restart ipa.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=
ipa         : DEBUG    stderr=Failed to open /dev/tty: No such device or address
Failed to open /dev/tty: No such device or address

ipa         : DEBUG    Starting external process
ipa         : DEBUG    args=/bin/systemctl is-active ipa.service
ipa         : DEBUG    Process finished, return code=0
ipa         : DEBUG    stdout=active

ipa         : DEBUG    stderr=
ipa.ipapython.install.cli.install_tool(Replica): INFO     The ipa-replica-install command was successful
incorrect section name: 172.17.0.2
syntax error
cat: /run/ipa/exit_code: No such file or directory

I noticed however that the docker container IP addresses on both hosts are the same but I still encountered the same issue with the replica container on a custom docker bridge network.

I manually started the exited container and checked the FreeIPA services. Snippet below:

[root@ipa2 /]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful

I then restarted all the services and recalled the command and they were all running. Snippet below:

[root@ipa2 /]# ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting ipa_memcached Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@ipa2 /]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

FreeIPA basic operations (login, replication, etc) worked fine but I am still struggling to figure out why the process exited. I would like to think it is docker specific due to this line:

ipa.ipapython.install.cli.install_tool(Replica): INFO     The ipa-replica-install command was successful

Any help to shed more light on this would be very much appreciated.

Thanks

adelton commented 8 years ago

May we assume that you use

-e IPA_SERVER_IP=<public-ip-of-the-host>

plus a series of -p options to docker run?

adelton commented 8 years ago

I have hard time figuring out where that

incorrect section name: 172.17.0.2

message comes from.

maesterX commented 8 years ago

Yes you may.

The docker run command on host 1 was

docker run --name freeipa-master1 --privileged -ti \
  -e 'IPA_SERVER_IP=10.100.0.54' -p 53:53/udp -p 53:53 \
  -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 -p 464:464 \
  -p 88:88/udp -p 464:464/udp -p 123:123/udp -p 7389:7389 \
  -p 9443:9443 -p 9444:9444 -p 9445:9445 \
  -h 'ipa1.cengnlocal.ca' \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --tmpfs /run --tmpfs /tmp \
  -v /var/lib/ipa-data:/data:Z freeipa-server

and on host 2:

docker run --name freeipa-server-container --dns 10.100.0.54 --privileged -ti \
  -e 'IPA_SERVER_IP=10.100.0.193' -p 53:53/udp -p 53:53 \
  -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 -p 464:464 \
  -p 88:88/udp -p 464:464/udp -p 123:123/udp -p 7389:7389 \
  -p 9443:9443 -p 9444:9444 -p 9445:9445 \
  -h ipa2.cengnlocal.ca \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --tmpfs /run --tmpfs /tmp \
  -v /var/lib/ipa-data:/data:Z \
  freeipa-server ipa-replica-install
adelton commented 8 years ago

Can you try without the --privileged? Primarily because that's not how I test it here. I was not able to reproduce the issue without --privileged.

maesterX commented 8 years ago

Ok. Will do and let you know how it goes. Thank you

jslatten commented 8 years ago

Same issue for me...Any insight?

adelton commented 8 years ago

Same issue for me...Any insight?

Do you run --privileged or not privileged? Do you see the incorrect section name message?

nfuentes commented 8 years ago

Hi!

i'm trying to setup an ipa replica on amazon AWS, but i'm having the following error:

[27/43]: restarting directory server
ipa         : CRITICAL Failed to restart the directory server (Command '/bin/systemctl restart dirsrv@WATEA-COM-AR.service' returned non-zero exit status 1). See the installation log for details.

This is an extract of the logfile:

2016-11-23T15:45:22Z DEBUG certmonger request is in state dbus.String(u'NEWLY_ADDED_READING_KEYINFO', variant_level=1)
2016-11-23T15:45:27Z DEBUG certmonger request is in state dbus.String(u'CA_UNCONFIGURED', variant_level=1)
2016-11-23T15:45:27Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-WATEA-COM-AR.socket from SchemaCache
2016-11-23T15:45:27Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-WATEA-COM-AR.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f6aafc176c8>
2016-11-23T15:45:28Z DEBUG   duration: 5 seconds
2016-11-23T15:45:28Z DEBUG   [27/43]: restarting directory server
2016-11-23T15:45:28Z DEBUG Starting external process
2016-11-23T15:45:28Z DEBUG args=/bin/systemctl --system daemon-reload
2016-11-23T15:45:28Z DEBUG Process finished, return code=0
2016-11-23T15:45:28Z DEBUG stdout=
2016-11-23T15:45:28Z DEBUG stderr=Failed to open /dev/tty: No such device or address

2016-11-23T15:45:28Z DEBUG Starting external process
2016-11-23T15:45:28Z DEBUG args=/bin/systemctl restart dirsrv@WATEA-COM-AR.service
2016-11-23T15:45:28Z DEBUG Process finished, return code=1
2016-11-23T15:45:28Z DEBUG stdout=
2016-11-23T15:45:28Z DEBUG stderr=Failed to open /dev/tty: No such device or address
Failed to open /dev/tty: No such device or address
Job for dirsrv@WATEA-COM-AR.service failed because the control process exited with error code. See "systemctl status dirsrv@WATEA-COM-AR.service" and "journalctl -xe" for details.

2016-11-23T15:45:28Z CRITICAL Failed to restart the directory server (Command '/bin/systemctl restart dirsrv@WATEA-COM-AR.service' returned non-zero exit status 1). See the installation log for details.
2016-11-23T15:45:29Z DEBUG   duration: 0 seconds
2016-11-23T15:45:29Z DEBUG   [28/43]: setting up initial replication
2016-11-23T15:45:39Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 447, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 437, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 405, in __setup_replica
    self.dm_password)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 114, in enable_replication_version_checking
    conn.do_simple_bind(bindpw=dirman_passwd)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1621, in do_simple_bind
    self.__bind_with_wait(self.simple_bind, timeout, binddn, bindpw)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1616, in __bind_with_wait
    self.__wait_for_connection(timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1599, in __wait_for_connection
    wait_for_open_socket(lurl.hostport, timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1291, in wait_for_open_socket
    raise e
error: [Errno 111] Connection refused

2016-11-23T15:45:39Z DEBUG   [error] error: [Errno 111] Connection refused
2016-11-23T15:45:39Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 318, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 310, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 332, in execute
    for nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 372, in __runner
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 394, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 362, in __runner
 step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 359, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1687, in main
    promote(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 377, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1393, in promote
    promote=True, pkcs12_info=dirsrv_pkcs12_info)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 125, in install_replica_ds
    promote=promote,
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 399, in create_replica
    self.start_creation(runtime=60)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 447, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 437, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 405, in __setup_replica
    self.dm_password)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 114, in enable_replication_version_checking
    conn.do_simple_bind(bindpw=dirman_passwd)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1621, in do_simple_bind
    self.__bind_with_wait(self.simple_bind, timeout, binddn, bindpw)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1616, in __bind_with_wait
    self.__wait_for_connection(timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1599, in __wait_for_connection
    wait_for_open_socket(lurl.hostport, timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1291, in wait_for_open_socket
    raise e

2016-11-23T15:45:39Z DEBUG The ipa-replica-install command failed, exception: error: [Errno 111] Connection refused
2016-11-23T15:45:39Z ERROR [Errno 111] Connection refused
2016-11-23T15:45:39Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

i'm launching the container with the following docker command:

sudo docker run --privileged --name freeipa-server-container -ti -h heracles.watea.com.ar --dns=192.168.10.64 --dns=192.168.10.28 -e IPA_SERVER_IP=192.168.10.64 -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /etc/hosts:/etc/hosts --tmpfs /run --tmpfs /tmp -p 53:53/udp -p 53:53 -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 -p 464:464 -p 88:88/udp -p 464:464/udp -p 123:123/udp -p 7389:7389 -p 9443:9443 -p 9444:9444 -p 9445:9445 --network host -v /var/lib/ipa-data:/data freeipa-server ipa-replica-install --no-host-dns --skip-conncheck --admin-password=Dx90puns --allow-zone-overlap

I've read that it's not suggested to run it with privileged mode, but if I remove that parameter, I can't launch it. Docker is running on a centos 7 host

Any ideas?

Thanks!

adelton commented 8 years ago

i'm trying to setup an ipa replica on amazon AWS, but i'm having the following error:

  1. Could we use different issue for this report, so that it is properly tracked?
  2. Can you be more specific about the "can't launch it"?
  3. What does systemctl status dirsrv@WATEA-COM-AR.service show?
nfuentes commented 8 years ago

Could we use different issue for this report, so that it is properly tracked? yes, I've just created this issue: https://github.com/adelton/docker-freeipa/issues/95

adelton commented 7 years ago

I still was not able to reproduce the issue.

zultron commented 7 years ago

I'm seeing it, too. incorrect section name and syntax error are strings in /usr/bin/nsupdate.

zultron commented 7 years ago

I switched the install to a container with Internet access, and the problem disappeared. I'm not positive that restricted network was causing the problem, but there's something to look at.

adelton commented 7 years ago

Thank you for the pointer to nsupdate. That would lead us to https://github.com/freeipa/freeipa-container/blob/master/ipa-server-configure-first#L29-L47. Could you perhaps add set -x to the start of that update_server_ip_address function to see what IP address values we are dealing with here? Or change that pipe that leads to nsupdate -g to store those echo output in file and cat is so that we know exactly what we pass to nsupdate?

zultron commented 7 years ago

Just for the record, I ran into this one more time yesterday. However, after enabling set -x and resetting/restarting the install, it wouldn't reproduce. Hrm.

PR #156 means to make it easy for others to turn on script tracing by adding docker run -e DEBUG_TRACE=1. Currently, one must check out this repo, edit the script and build a new container to collect that debugging information.

It looks like @adelton has been working with upstream on the related BZ1377973 to get a fix into v. 4.5 for ipa-server-install --ip-address=$IP when $IP isn't a configured interface address inside the container. Nice work! In use cases where the external IP is fixed, that fix will make the update_server_ip_address function will be obsolete once the container is updated to FreeIPA 4.5 (see #157), and this issue won't apply. Am I correct about that?

jtyr commented 6 years ago

I'm having problems to create replica instance because it doesn't respect the host IP address which I define via IPA_SERVER_IP and --ip-address. It always uses internal docker IP for the replica DNS which causes the installation to fail:

### Docker log
...
Starting replication, please wait until this has completed.
Update in progress, 15 seconds elapsed
[ldap://ipa1.example.test:389] reports: Update failed! Status: [49  - LDAP error: Invalid credentials]

  [error] RuntimeError: Failed to start replication
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipapython.admintool: ERROR    Failed to start replication
ipapython.admintool: ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
FreeIPA server configuration failed.

### /var/log/ipareplica-install.log
...
2018-04-06T10:58:42Z DEBUG Fetching nsDS5ReplicaId from master [attempt 1/5]
2018-04-06T10:58:42Z DEBUG retrieving schema for SchemaCache url=ldap://ipa1.example.test:389 conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7f2df156eac8>
2018-04-06T10:58:42Z DEBUG Successfully updated nsDS5ReplicaId.
2018-04-06T10:58:59Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py", line 506, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py", line 496, in run_step
    method()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line 441, in __setup_replica
    cacert=self.ca_file)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/replication.py", line 1677, in setup_promote_replication
    raise RuntimeError("Failed to start replication")
RuntimeError: Failed to start replication

2018-04-06T10:58:59Z DEBUG   [error] RuntimeError: Failed to start replication
2018-04-06T10:58:59Z DEBUG Destroyed connection context.ldap2_139835351925368
2018-04-06T10:58:59Z DEBUG Backing up system configuration file '/etc/ipa/default.conf'
2018-04-06T10:58:59Z DEBUG Saving Index File to '/var/lib/ipa/sysrestore/sysrestore.index'
2018-04-06T10:58:59Z DEBUG   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 336, in run
    cfgr.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 364, in run
    self.execute()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 388, in execute
    for _nothing in self._executor():
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 430, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 459, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 420, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 417, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 654, in _configure
    next(executor)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 430, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 459, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 517, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 514, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 420, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 417, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 66, in _install
    for _nothing in self._installer(self.parent):
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py", line 617, in main
    replica_install(self)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 388, in decorated
    func(installer)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 1404, in install
    pkcs12_info=dirsrv_pkcs12_info)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 110, in install_replica_ds
    setup_pkinit=not options.no_pkinit,
  File "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line 406, in create_replica
    self.start_creation(runtime=30)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py", line 506, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py", line 496, in run_step
    method()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line 441, in __setup_replica
    cacert=self.ca_file)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/replication.py", line 1677, in setup_promote_replication
    raise RuntimeError("Failed to start replication")

2018-04-06T10:58:59Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: Failed to start replication
2018-04-06T10:58:59Z ERROR Failed to start replication
2018-04-06T10:58:59Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

If I change the DNS record of the replica to the host IP as soon as the client registers it (change A record for ipa2 from 172.17.0.2 to 192.168.233.11), then the installation process finishes successfully. I have tried several image tags (latest ~ IPA v4.4.x, centos-7 ~ IPA 4.5.x, fedora-27 ~ IPA v4.6.x) but the result is always the same.

Here is how I run the replica container:

# For the latest tag
docker run \
  --name freeipa-master2 \
  -e IPA_SERVER_IP=192.168.233.11 \
  -tid \
  -h ipa2.example.test \
  --dns 192.168.233.10 \
  -v /var/lib/freeipa-data:/data:Z \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --tmpfs /run --tmpfs /tmp \
  -p 53:53/udp \
  -p 53:53 \
  -p 80:80 \
  -p 88:88/udp \
  -p 88:88 \
  -p 123:123/udp \
  -p 389:389 \
  -p 443:443 \
  -p 464:464/udp \
  -p 464:464 \
  -p 636:636 \
  -p 7389:7389 \
  -p 9443:9443 \
  -p 9444:9444 \
  -p 9445:9445 \
  freeipa/freeipa-server \
  ipa-replica-install \
    --admin-password=password \
    --setup-dns \
    --ip-address 192.168.233.11 \
    --forwarder 8.8.8.8 --forwarder 8.8.4.4 \
    --setup-ca \
    --server ipa1.example.test \
    --domain example.test \
    --no-host-dns

# For the centos-7 and the fedore-27 tags
docker run \
  --name freeipa-master2 \
  -e IPA_SERVER_IP=192.168.233.11 \
  -tid \
  -h ipa2.example.test \
  --dns 192.168.233.10 \
  --sysctl net.ipv6.conf.lo.disable_ipv6=0 \
  -v /var/lib/freeipa-data:/data:Z \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --tmpfs /run --tmpfs /tmp \
  -p 53:53/udp \
  -p 53:53 \
  -p 80:80 \
  -p 88:88/udp \
  -p 88:88 \
  -p 123:123/udp \
  -p 389:389 \
  -p 443:443 \
  -p 464:464/udp \
  -p 464:464 \
  -p 636:636 \
  -p 7389:7389 \
  -p 9443:9443 \
  -p 9444:9444 \
  -p 9445:9445 \
  freeipa/freeipa-server:fedora-27 \
  ipa-replica-install \
    --admin-password=password \
    --setup-dns \
    --ip-address 192.168.233.11 \
    --forwarder 8.8.8.8 --forwarder 8.8.4.4 \
    --setup-ca \
    --server ipa1.example.test \
    --domain example.test \
    --no-host-dns

Here is how I run the container for the ipa1.example.test server:

docker run \
  --name freeipa-master1 \
  -tid \
  -e IPA_SERVER_IP=192.168.233.10 \
  -h ipa1.example.test \
  -v /var/lib/freeipa-data:/data:Z \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --tmpfs /run --tmpfs /tmp \
  -p 53:53/udp \
  -p 53:53 \
  -p 80:80 \
  -p 88:88/udp \
  -p 88:88 \
  -p 123:123/udp \
  -p 389:389 \
  -p 443:443 \
  -p 464:464/udp \
  -p 464:464 \
  -p 636:636 \
  -p 7389:7389 \
  -p 9443:9443 \
  -p 9444:9444 \
  -p 9445:9445 \
  freeipa/freeipa-server \
    --setup-dns \
    --forwarder 8.8.8.8 --forwarder 8.8.4.4 \
    --realm=EXAMPLE.TEST \
    --ds-password=password \
    --admin-password=password \
    --hostname=ipa1.example.test \
    --domain example.test \
    --auto-reverse \
    --unattended
adelton commented 6 years ago

@jtyr, sorry for late reply. I'm a bit confused -- what is the IPA master hostname and IP in your example (is IPA master running in container as well or not) and what is the replica hostname in your setup?

jtyr commented 6 years ago

@adelton The master is running in the container names freeipa-master1 (hostname ipa1.example.test, IP 192.168.233.10) and the replica is running in the container named freeipa-master2 (hostname ipa2.example.test, IP 192.168.233.11).

adelton commented 6 years ago

I wonder -- can you create the DNS record for the replica on the master (to point to replica's host IP address) even before you start the replica container?

jtyr commented 6 years ago

That would work as well. It's the same like I change the DNS record of the replica to the host IP as soon as the client registers as I described above.

adelton commented 6 years ago

The IPA_SERVER_IP is really only used to put in some specific value to DNS when the IPA server is running DNS server, which is only after the replica was established.

We'd need someone from the FreeIPA team to figure out if it's correct that the --ip-address 192.168.233.11 option that you use on the replica to define "its" IP address to be the IP address of the host does not seem to be used during replication setup. @Tiboris, would you please check what is the behaviour of FreeIPA replicas on the host (no containers) for example in Amazon's AWS? If you have master outside of AWS and want to setup replica in AWS where the host obviously onyl sees its own IP addresses and you pass --ip-address ... with the public address of the AWS machine, will the replication work?

maxozerov commented 6 years ago

ohh.. for me its mystical problems: same issue

Update failed! Status: [49 - LDAP error: Invalid credentials]

And finally, start container, run cmd ipa-server-install --uninstall then remove my host (replica-host) from current ipa hosts, and start container again -- it's work, no problem after ipa-replica-install worked - all fine and no error's.

Maybe problem not in DNS or ip-address. Container - replica 

  docker run -p 53:53/udp \
       -p 53:53 -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 \
       -p 464:464 -p 88:88/udp -p 464:464/udp -p 749:749 -p 123:123/udp \
       -e IPA_SERVER_IP=${IPA_SERVER_REPL_ADDR} --name ${IPA_SERVER_REPL_NAME} -ti \
       -h ${IPA_SERVER_REPL_NAME} \
       -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
       --tmpfs /run --tmpfs /tmp \
       -v ${IPA_DATA_DIR}:/data:Z ${IPA_DOCKER_REPO} ipa-replica-install

Ubuntu 16.04.4 LTS Docker version 18.03.1-ce, build 9ee9f40 Dockerfile.centos-7

adelton commented 5 years ago

I assume that by using manual DNS records, it is possible to setup the replica reasonably well.