can not setup replica: please run ipa-server-upgrade command

[kforner]

Both freeIPA use the same docker: adelton/freeipa-server:latest-systemd (25543e7d8c07) from 16 months ago containing freeIPA v 4.2.3

I generated replica-infos on the master using:

docker exec -ti $(MASTER_NAME) ipa-replica-prepare $(REPLICA) --ip-address $(REPLICA_IP) -v

I then run the replica after decrypting and putting the ipa-replica-install-options in ipa-data/ then

        docker run --name $(REPLICA_NAME) -d  \
        --restart=always \
         -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
        -e IPA_SERVER_IP=$(REPLICA_IP) \
        --dns $(MASTER_IP) \
        -h $(REPLICA_HOST).$(DOMAIN) \
        -v $(PWD)/ipa-data:/data \
        -v /etc/localtime:/etc/localtime:ro \
        -p 53:53  -p 53:53/udp \
        -p 80:80 -p 443:443 \
        -p 389:389 -p 636:636 -p 88:88 -p 464:464 \
        -p 88:88/udp -p 464:464/udp -p 7389:7389 \
        -p 9443:9443 -p 9444:9444 -p 9445:9445  $(IMAGE)

If I docker exec into the container, and run

# ipactl start
Upgrade required: please run ipa-server-upgrade command
Aborting ipactl

# ipa-server-upgrade
session memcached servers not running
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Certmonger is not running. Start certmonger and run upgrade again.

What I don't understand is that both freeIPA are running the exact same docker image, hence the same freeIPA version.

What should I do ? Thanks.

[MartinBasti]

Hello, could you please try more recent images from official docker hub?

https://hub.docker.com/r/freeipa/freeipa-server/

adelton/freeipa-server:latest-systemd doesn't sound like stable to me, but more like an experimental branch.

[kforner]

The problem is that my master freeIPA runs on that version, and I cannot afford to migrate it without first successfully installing a replica. Could only the replica use the newest image ? thanks.

[MartinBasti]

It is common migrate path to use newer replica, it should work with containers too. 4.2 reached end of life so you should update to get security updates. (in case you are using CA, install replica with CA --setup-ca)

[kforner]

I tried with latest (freeipa/freeipa-server:latest), and it seemed to went well until:

...
Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Check RPC connection to remote master
Retrying using SSH...
Check SSH connection to remote master
WARNING: ssh not installed, skipping ssh test
ipa         : DEBUG    stderr=Traceback (most recent call last):
  File "/usr/sbin/ipa-replica-conncheck", line 557, in <module>
    sys.exit(main())
  File "/usr/sbin/ipa-replica-conncheck", line 530, in main
    if result.returncode != 0:
AttributeError: 'tuple' object has no attribute 'returncode'

ipa.ipapython.install.cli.install_tool(Replica): DEBUG      File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 318, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 308, in run
    self.validate()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 317, in validate
    for nothing in self._validator():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 376, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 400, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 363, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 575, in _configure
    next(validator)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 376, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 400, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 457, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 395, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 363, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1739, in main
    install_check(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 375, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 777, in install_check
    ca_cert_file=cafile)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 106, in replica_conn_check
    "Connection check failed!"

ipa.ipapython.install.cli.install_tool(Replica): DEBUG    The ipa-replica-install command failed, exception: ScriptError: Connection check failed!
Please fix your network settings according to error messages above.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
ipa.ipapython.install.cli.install_tool(Replica): ERROR    Connection check failed!
Please fix your network settings according to error messages above.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
FreeIPA server configuration failed.

[kforner]

Any help ? I cannot use the same docker image that I'm using for the other servers, and the newest do not work either...

[MartinBasti]

Could you try with --skip-conncheck ?

[martbab]

Actually this is a bug in the conncheck code that only appears when there is no SSH installed on the replica (like in a container). I would go with @MartinBasti's suggestion to use --skip-conncheck to workaround the issue while the upstream produces a fix.

[martbab]

Link to the issue: https://pagure.io/freeipa/issue/6935

[kforner]

Sure, sorry for the delay. I retried with --skip-conncheck :

2017-05-09T15:11:10Z DEBUG Logging to /var/log/ipareplica-install.log
2017-05-09T15:11:10Z DEBUG ipa-replica-install was invoked with arguments [] and options: {'no_dns_sshfp': None, 'skip_schema_check': None, 'setup_kra': None, 'ip_addresses': None, 'mkhomedir': None, 'no_pkinit': None, 'http_cert_files': None, 'no_ntp': None, 'verbose': True, 'no_forwarders': None, 'keytab': None, 'ssh_trust_dns': None, 'domain_name': None, 'http_cert_name': None, 'dirsrv_cert_files': None, 'no_dnssec_validation': None, 'no_reverse': None, 'pkinit_cert_files': None, 'unattended': False, 'auto_reverse': None, 'auto_forwarders': None, 'no_host_dns': True, 'no_sshd': None, 'no_ui_redirect': None, 'dirsrv_config_file': None, 'forwarders': None, 'pkinit_cert_name': None, 'setup_ca': True, 'realm_name': None, 'skip_conncheck': True, 'no_ssh': None, 'forward_policy': None, 'dirsrv_cert_name': None, 'quiet': False, 'server': None, 'setup_dns': None, 'host_name': None, 'log_file': None, 'reverse_zones': None, 'allow_zone_overlap': None}
2017-05-09T15:11:10Z DEBUG IPA version 4.4.4-1.fc25
2017-05-09T15:11:10Z DEBUG Starting external process
2017-05-09T15:11:10Z DEBUG args=/usr/sbin/selinuxenabled
2017-05-09T15:11:10Z DEBUG Process finished, return code=1
2017-05-09T15:11:10Z DEBUG stdout=
2017-05-09T15:11:10Z DEBUG stderr=
2017-05-09T15:11:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-09T15:11:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2017-05-09T15:11:10Z DEBUG httpd is not configured
2017-05-09T15:11:10Z DEBUG kadmin is not configured
2017-05-09T15:11:10Z DEBUG dirsrv is not configured
2017-05-09T15:11:10Z DEBUG pki-tomcatd is not configured
2017-05-09T15:11:10Z DEBUG install is not configured
2017-05-09T15:11:10Z DEBUG krb5kdc is not configured
2017-05-09T15:11:10Z DEBUG ntpd is not configured
2017-05-09T15:11:10Z DEBUG named is not configured
2017-05-09T15:11:10Z DEBUG ipa_memcached is not configured
2017-05-09T15:11:10Z DEBUG filestore is tracking no files
2017-05-09T15:11:10Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2017-05-09T15:11:10Z DEBUG Configuring client side components
2017-05-09T15:11:10Z DEBUG Starting external process
2017-05-09T15:11:10Z DEBUG args=/usr/sbin/ipa-client-install --unattended --no-ntp --password xxxxxxx
2017-05-09T15:11:11Z DEBUG Process finished, return code=1
2017-05-09T15:11:11Z DEBUG Starting external process
2017-05-09T15:11:11Z DEBUG args=/usr/sbin/ipa-client-install --unattended --uninstall
2017-05-09T15:11:11Z DEBUG Process finished, return code=2
2017-05-09T15:11:11Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 318, in run
    cfgr.run()
...
    raise ScriptError("Configuration of client side components failed!")

2017-05-09T15:11:11Z DEBUG The ipa-replica-install command failed, exception: ScriptError: Configuration of client side components failed!
2017-05-09T15:11:11Z ERROR Configuration of client side components failed!
2017-05-09T15:11:11Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

I also looked at the ipa-data/var/log/ipaclient-install.log :

...
2017-05-09T15:11:11Z DEBUG will use discovered server: ipa.quartzbio.com
2017-05-09T15:11:11Z INFO Discovery was successful!
2017-05-09T15:11:11Z DEBUG will use discovered realm: QUARTZBIO.COM
2017-05-09T15:11:11Z DEBUG will use discovered basedn: dc=quartzbio,dc=com
2017-05-09T15:11:11Z INFO Client hostname: iparig.quartzbio.com
...
2017-05-09T15:11:11Z INFO Successfully retrieved CA cert
...
2017-05-09T15:11:11Z DEBUG Starting external process
2017-05-09T15:11:11Z DEBUG args=/usr/sbin/ipa-join -s ipa.quartzbio.com -b dc=quartzbio,dc=com -h iparig.quartzbio.com -w XXXXXXXX
2017-05-09T15:11:11Z DEBUG Process finished, return code=15

/usr/sbin/ipa-join -s ipa.quartzbio.com -b dc=quartzbio,dc=com -h iparig.quartzbio.com ...

I guess this is the crux of the problem. The process discovered ipa.quartzbio.com as master server hostname. The problem is that it actually is an apache proxy server that redirects to the appropriate ipa web ui, e.g.. ipa.quartzbio.com -> 186.127.207.13 while the freeIPA server runs on 10.95.72.6 How can I tell the replica install process to use the appropriate IP for the freeIPA master server ?

Thanks a lot.

[MartinBasti]

There is no option how to provide IP address of remote server, your records must resolve properly to right hosts.

[kforner]

I must add that everything works all right for years for all our infrastructure, that I could setup the replica with the former docker version. I did not setup the record myself (I have no idea how to do it). I always followed the docs, and instructions on how to setup an apache proxy on top of the freeIPA web ui.

[adelton]

If you have some Apache (reverse) proxy, it only proxies the http(s) traffic while for clients and replicas you also need LDAP and other ports. You really should be pointing the replica to the real master, or to IP address which exposes or forwards all the necessary ports.

I have no idea why with previous docker (docker, or FreeIPA server container image?) you were able to setup replicas in the past.

But the nice thing about the containerized FreeIPA setup is that all the configuration and data are isolated in that single data volume that you mount to /data. So you should be able to copy content of that data volume for one of your existing replicas to some different machine, isolated from the rest of the setup, and run fresh container image with that data and let it upgrade and test it and verify that it works. And then ditch it and run the same on one of the real-life replica, having of course backup and the original container image ready in case something goes wrong.

[kforner]

But the nice thing about the containerized FreeIPA setup is that all the configuration and data are isolated in that single data volume that you mount to /data. So you should be able to copy content of that data volume for one of your existing replicas to some different machine, isolated from the rest of the setup, and run fresh container image with that data and let it upgrade and test it and verify that it works.

My first naive attempt at it:

docker run --name replica -t -d \
        --privileged \
        -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
        -v $(pwd)/ipa-data:/data \
        -v /etc/localtime:/etc/localtime:ro \
        -p 53:53  -p 53:53/udp \
        -p 80:80 -p 443:443 \
        -p 389:389 -p 636:636 -p 88:88 -p 464:464 \
        -p 88:88/udp -p 464:464/udp -p 7389:7389 \
        -p 9443:9443 -p 9444:9444 -p 9445:9445 $IMAGE`

Failed to insert module 'autofs4': Function not implemented
systemd 222 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Fedora 23 (Twenty Three)!

Set hostname to <72c12a7c5802>.
Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Configuration file /usr/lib/systemd/system/pki-tomcatd.target is marked executable. Please remove executable permission bits. Proceeding anyway.
Configuration file /lib/systemd/system/pki-tomcatd@.service is marked executable. Please remove executable permission bits. Proceeding anyway.
local-fs.target: Cannot add dependency job, ignoring: Unit local-fs.target is masked.
ldconfig.service: Cannot add dependency job, ignoring: Unit ldconfig.service is masked.
dev-hugepages.mount: Cannot add dependency job, ignoring: Unit dev-hugepages.mount is masked.
systemd-update-done.service: Cannot add dependency job, ignoring: Unit systemd-update-done.service is masked.
sys-fs-fuse-connections.mount: Cannot add dependency job, ignoring: Unit sys-fs-fuse-connections.mount is masked.
systemd-hwdb-update.service: Cannot add dependency job, ignoring: Unit systemd-hwdb-update.service is masked.
swap.target: Cannot add dependency job, ignoring: Unit swap.target is masked.
fedora-autorelabel-mark.service: Cannot add dependency job, ignoring: Unit fedora-autorelabel-mark.service is masked.
slices.target: Cannot add dependency job, ignoring: Unit slices.target is masked.
rpcbind.socket: Cannot add dependency job, ignoring: Unit rpcbind.socket is masked.
dnf-makecache.timer: Cannot add dependency job, ignoring: Unit dnf-makecache.timer is masked.
getty.target: Cannot add dependency job, ignoring: Unit getty.target is masked.
nfs-client.target: Cannot add dependency job, ignoring: Unit nfs-client.target is masked.
systemd-user-sessions.service: Cannot add dependency job, ignoring: Unit systemd-user-sessions.service is masked.
nfs-client.target: Cannot add dependency job, ignoring: Unit nfs-client.target is masked.
systemd-logind.service: Cannot add dependency job, ignoring: Unit systemd-logind.service is masked.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Network.
[  OK  ] Reached target Paths.
[  OK  ] Created slice Root Slice.
[  OK  ] Listening on Journal Audit Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Created slice system-dirsrv.slice.
[  OK  ] Listening on Journal Socket.
         Starting Read and set NIS domainname from /etc/sysconfig/network...
         Starting Create System Users...
         Starting Rebuild Journal Catalog...
         Starting Journal Service...
         Starting Setup Virtual Console...
         Starting Apply Kernel Variables...
         Mounting Debug File System...
         Starting Load/Save Random Seed...
[  OK  ] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
[  OK  ] Reached target Remote File Systems.
[  OK  ] Created slice system-pki\x2dtomcatd.slice.
[  OK  ] Set up automount Arbitrary Executab...ats File System Automount Point.
[  OK  ] Mounted Debug File System.
[  OK  ] Started Read and set NIS domainname from /etc/sysconfig/network.
[  OK  ] Started Create System Users.
systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE
[FAILED] Failed to start Setup Virtual Console.
See 'systemctl status systemd-vconsole-setup.service' for details.
systemd-vconsole-setup.service: Unit entered failed state.
systemd-vconsole-setup.service: Failed with result 'exit-code'.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Started Rebuild Journal Catalog.
[  OK  ] Started Create Static Device Nodes in /dev.
         Starting udev Kernel Device Manager...
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
         Starting Security Auditing Service...
[  OK  ] Started Security Auditing Service.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Network Time Service...
         Starting PKI Tomcat Server pki-tomcat...
         Starting Upgrade IPA server upon subsequent runs...
         Starting 389 Directory Server QUARTZBIO-COM....
         Starting System Security Services Daemon...
[  OK  ] Started D-Bus System Message Bus.

FreeIPA server is already configured, starting the services.
         Starting D-Bus System Message Bus...
         Starting Certificate monitoring and PKI enrollment...
[  OK  ] Started Network Time Service.
[FAILED] Failed to start PKI Tomcat Server pki-tomcat.
See 'systemctl status pki-tomcatd@pki-tomcat.service' for details.
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Upgrade IPA server upon subsequent runs.
         Starting Identity, Policy, Audit...
[FAILED] Failed to start System Security Services Daemon.
See 'systemctl status sssd.service' for details.
[  OK  ] Reached target User and Group Name Lookups.
[  OK  ] Started Certificate monitoring and PKI enrollment.
[  OK  ] Started 389 Directory Server QUARTZBIO-COM..
[  OK  ] Reached target 389 Directory Server.
[  OK  ] Reached target PKI Tomcat Server.
         Starting 389 Directory Server QUARTZBIO-COM....
[  OK  ] Started 389 Directory Server QUARTZBIO-COM..
[FAILED] Failed to start Identity, Policy, Audit.
See 'systemctl status ipa.service' for details.
         Starting Update self IP address of the IPA server...

[kforner]

And then ditch it and run the same on one of the real-life replica, having of course backup and the >original container image ready in case something goes wrong.

But I also want to setup an additional replica. Using this technique, is-it possible to modify the replica info (hostname, IP) and establish new replica relations using the command-line ?

[adelton]

You want to get to the latest versions of FreeIPA, no matter if they are run in container, in VM, or on bare metal. After you've upgraded, you should be able to make another replica more easily, with up-to-date versions of IPA / container images.

[felipevolpone]

Closing this due lack of activity.