search

freeipa / freeipa-container

FreeIPA server in containers — images at https://quay.io/repository/freeipa/freeipa-server?tab=tags
https://quay.io/repository/freeipa/freeipa-server?tab=tags
Apache License 2.0
614 stars 259 forks source link

ipa replica install fails with org.freedesktop.DBus.Error.NoReply: Did not receive a reply #605

Closed dmitry-shiryaev closed 7 months ago

dmitry-shiryaev commented 8 months ago

Hello, I am trying to setup three ipa replicas against a single master node. All three replicas are running on Debian 12 kvm vms, inside almalinux-8-4.9.12 docker (version 25.0.3, build 4debf41) container. Two replicas are running smoothly. However, when I am starting my third one I get:

On replica:

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Check RPC connection to remote master
Execute check on remote master
ERROR: Remote master check failed with following error message(s):
an internal error has occurred

2024-03-28T14:48:58Z DEBUG Starting external process
2024-03-28T14:48:58Z DEBUG args=['/usr/sbin/ipa-client-install', '--unattended', '--uninstall']
2024-03-28T14:49:00Z DEBUG Process finished, return code=0
2024-03-28T14:49:00Z DEBUG   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.9/site-packages/ipapython/install/cli.py", line 344, in run
    return cfgr.run()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 358, in run
    self.validate()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 368, in validate
    for _nothing in self._validator():
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 435, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 463, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 425, in __runner
    step()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 419, in step_next
    return next(self.__gen)
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 641, in _configure
    next(validator)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 435, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 463, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 526, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 523, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 425, in __runner
    step()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 419, in step_next
    return next(self.__gen)
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.9/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python3.9/site-packages/ipaserver/install/server/__init__.py", line 597, in main
    replica_promote_check(self)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/server/replicainstall.py", line 401, in decorated
    func(installer)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/server/replicainstall.py", line 423, in decorated
    func(installer)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/server/replicainstall.py", line 1192, in promote_check
    replica_conn_check(
  File "/usr/lib/python3.9/site-packages/ipaserver/install/replication.py", line 128, in replica_conn_check
    raise ScriptError(

2024-03-28T14:49:00Z DEBUG The ipa-replica-install command failed, exception: ScriptError: Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
2024-03-28T14:49:00Z ERROR Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
2024-03-28T14:49:00Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

On master:

[Thu Mar 28 14:48:58.208310 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462] ipa: ERROR: non-public: DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
[Thu Mar 28 14:48:58.208340 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462] Traceback (most recent call last):
[Thu Mar 28 14:48:58.208342 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib/python3.6/site-packages/ipaserver/rpcserver.py", line 417, in wsgi_execute
[Thu Mar 28 14:48:58.208344 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     result = command(*args, **options)
[Thu Mar 28 14:48:58.208346 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 471, in __call__
[Thu Mar 28 14:48:58.208348 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     return self.__do_call(*args, **options)
[Thu Mar 28 14:48:58.208350 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 499, in __do_call
[Thu Mar 28 14:48:58.208351 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     ret = self.run(*args, **options)
[Thu Mar 28 14:48:58.208353 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 816, in run
[Thu Mar 28 14:48:58.208355 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     return self.execute(*args, **options)
[Thu Mar 28 14:48:58.208356 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib/python3.6/site-packages/ipaserver/plugins/server.py", line 948, in execute
[Thu Mar 28 14:48:58.208358 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     ret, stdout, _stderr = server.conncheck(keys[-1])
[Thu Mar 28 14:48:58.208367 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib64/python3.6/site-packages/dbus/proxies.py", line 70, in __call__
[Thu Mar 28 14:48:58.208369 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     return self._proxy_method(*args, **keywords)
[Thu Mar 28 14:48:58.208370 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib64/python3.6/site-packages/dbus/proxies.py", line 145, in __call__
[Thu Mar 28 14:48:58.208372 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     **keywords)
[Thu Mar 28 14:48:58.208374 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]   File "/usr/lib64/python3.6/site-packages/dbus/connection.py", line 651, in call_blocking
[Thu Mar 28 14:48:58.208376 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462]     message, timeout)
[Thu Mar 28 14:48:58.208378 2024] [wsgi:error] [pid 4939:tid 139657344001792] [remote 10.88.0.10:38462] dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

It is similar to https://github.com/freeipa/freeipa-container/issues/187 but I think it was fixed.

I also try installing my 3rd ipa replica against any of the two successful installs with the same error. I checked and this is not a firewall or networking issue. What am I missing here?

adelton commented 8 months ago

The message says

See /var/log/ipareplica-conncheck.log for more information.

so I'd start investigation there.

dmitry-shiryaev commented 8 months ago

There is nothing really useful there imo

2024-03-29T07:21:21Z DEBUG [try 1]: Forwarding 'ping/1' to json server 'https://master.example.com/ipa/json'
2024-03-29T07:21:21Z DEBUG New HTTP connection (master.example.com)
2024-03-29T07:21:22Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=io4Eewukte9AOGjrWKnwx4BuloJAaSF7A9HRxcbxl85wEoOqVnBBQlvcO%2fjS0EIzhQg6xCrTWwkr8x%2fdT56iY7ofjgd%2f6T0dtGE8KlsJF0GH303l06BXDqUZSNwZtXMCDli3d1Vj31o2DN%2bDxk9CXIisynCb%2bVrSiQ%2bLFWZuBwTZ7%2bbKlVhpJqC58nxL18jQZce5M2XiK3kJ1kzPmsRpQs4b14E1hHRfPR8bKm3mS2A%3d;path=/ipa;httponly;secure;']'
2024-03-29T07:21:22Z DEBUG storing cookie 'ipa_session=MagBearerToken=io4Eewukte9AOGjrWKnwx4BuloJAaSF7A9HRxcbxl85wEoOqVnBBQlvcO%2fjS0EIzhQg6xCrTWwkr8x%2fdT56iY7ofjgd%2f6T0dtGE8KlsJF0GH303l06BXDqUZSNwZtXMCDli3d1Vj31o2DN%2bDxk9CXIisynCb%2bVrSiQ%2bLFWZuBwTZ7%2bbKlVhpJqC58nxL18jQZce5M2XiK3kJ1kzPmsRpQs4b14E1hHRfPR8bKm3mS2A%3d;' for principal admin@MASTER.COM
2024-03-29T07:21:22Z INFO Execute check on remote master
2024-03-29T07:21:22Z DEBUG [try 1]: Forwarding 'server_conncheck' to json server 'https://master.example.com/ipa/json'
2024-03-29T07:21:22Z DEBUG HTTP connection keep-alive (master.example.com)
2024-03-29T07:21:48Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=PrM9tsNshtNaLreS01%2fgaTlY2UUdp1axh3B39lLs2K8G7%2bkNRlWMGriSfbsNJhM9dCbDSmbyYD3NBUAtZ6oTXigil5x32HAAuZAIHv9%2fzaKxpGU71ZlXEW9MXHu9qqDFywwGLNFRfvWo87Fqs5j3jr9487%2btRg8qI8pXY8OLf13nBJzgFI3tt0TtvaEuyT2mWnFbJz%2b22mNYv9vFUc%2biE%2fPFBBGlurJ%2fQjsxB3F%2beX4%3d;path=/ipa;httponly;secure;']'
2024-03-29T07:21:48Z DEBUG storing cookie 'ipa_session=MagBearerToken=PrM9tsNshtNaLreS01%2fgaTlY2UUdp1axh3B39lLs2K8G7%2bkNRlWMGriSfbsNJhM9dCbDSmbyYD3NBUAtZ6oTXigil5x32HAAuZAIHv9%2fzaKxpGU71ZlXEW9MXHu9qqDFywwGLNFRfvWo87Fqs5j3jr9487%2btRg8qI8pXY8OLf13nBJzgFI3tt0TtvaEuyT2mWnFbJz%2b22mNYv9vFUc%2biE%2fPFBBGlurJ%2fQjsxB3F%2beX4%3d;' for principal local_admin@EXAMPLE.COM
2024-03-29T07:21:48Z DEBUG Destroyed connection context.rpcclient_139641227937496
2024-03-29T07:21:48Z ERROR ERROR: Remote master check failed with following error message(s):
an internal error has occurred
2024-03-29T07:21:48Z DEBUG Stopping listening thread.
2024-03-29T07:21:48Z DEBUG 389 tcp: Stopped listening
2024-03-29T07:21:48Z DEBUG 636 tcp: Stopped listening
2024-03-29T07:21:48Z DEBUG 88 tcp: Stopped listening
2024-03-29T07:21:48Z DEBUG 88 udp: Stopped listening
2024-03-29T07:21:48Z DEBUG 464 tcp: Stopped listening
2024-03-29T07:21:48Z DEBUG 464 udp: Stopped listening
2024-03-29T07:21:48Z DEBUG 80 tcp: Stopped listening
2024-03-29T07:21:48Z DEBUG 443 tcp: Stopped listening
adelton commented 8 months ago

So if I understand it correctly, you have an non-containerized FreeIPA master and you managed to create two containerized replicas and the third one fails. And it fails with that an internal error has occurred error on the master.

In that case I'd simplify the situation by trying to add the third replica not as a containerized one but one one a host / VM. In other words, take the containerization completely out of the picture and see what you get on the master.

dmitry-shiryaev commented 8 months ago

The master is also running inside a container. And as I mentioned above, third replica having an issue with any instance of ipa being a master: master, relical01, replica02. Not just master. So all points to the problem being with the third replica.

Take the containerization completely out of the picture and see what you get on the master.

Is it possible to install freeipa on Debian?

adelton commented 8 months ago

The master is also running inside a container. And as I mentioned above, third replica having an issue with any instance of ipa being a master: master, relical01, replica02. Not just master. So all points to the problem being with the third replica.

At the same time you say the replicas are the same, and the error is reported on the master.

Is it possible to install freeipa on Debian?

I can see https://packages.debian.org/experimental/freeipa-server. I never tried it.

dmitry-shiryaev commented 8 months ago

I provided two logs: one from replica installation the other is httpd log on master

on master when replica install fails I see this in /var/log/httpd/error_log non-public: DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

on replica, when it fails

ERROR: Remote master check failed with following error message(s):
an internal error has occurred

2024-03-28T14:48:58Z DEBUG Starting external process
2024-03-28T14:48:58Z DEBUG args=['/usr/sbin/ipa-client-install', '--unattended', '--uninstall']
2024-03-28T14:49:00Z DEBUG Process finished, return code=0
2024-03-28T14:49:00Z DEBUG   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()

I don't understand, what are you confused about?

adelton commented 7 months ago

I try to steer you towards checking if the third replica is really the same as the previous one ... well of course except its IP address and possibly host name need to be different and properly configured and propagated.

I'd recommend to try creating a test replica in a VM, not in a container. That would help you to make a bit more solid conclusion about the source of the problem -- is it the FreeIPA itself, the networking setup, or the fact that the FreeIPA replica was running containerized.

adelton commented 7 months ago

We seem to have lost traction here, closing.