spantaleev / matrix-docker-ansible-deploy

🐳 Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker
GNU Affero General Public License v3.0
4.84k stars 1.04k forks source link

On CentOS 8 Docker internal network needs to be set as trusted network for firewalld #300

Closed stfl closed 8 months ago

stfl commented 4 years ago

I just set up on CentOS 8 amongst other minor problems like docker-ce install, chronyd instead of ntpd, user 991 not beeing available on a plain VM I discovered that synapse cannot connect to the postgres container on the docker internal network 172.18.0.0

This server is configured to use 'matrix.org' as its trusted key server via the
'trusted_key_servers' config option. 'matrix.org' is a good choice for a key
server since it is long-lived, stable and trusted. However, some admins may
wish to use another server for this purpose.

To suppress this warning and continue using 'matrix.org', admins should set
'suppress_key_server_warning' to 'true' in homeserver.yaml.
--------------------------------------------------------------------------------
2019-11-07 23:04:56,352 - root - 237 - WARNING - None - ***** STARTING SERVER *****
2019-11-07 23:04:56,352 - root - 238 - WARNING - None - Server /usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py version 1.5.1
2019-11-07 23:04:57,380 - twisted - 172 - ERROR -  - Traceback (most recent call last):
2019-11-07 23:04:57,380 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
2019-11-07 23:04:57,380 - twisted - 172 - ERROR -  -     "__main__", mod_spec)
2019-11-07 23:04:57,381 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
2019-11-07 23:04:57,381 - twisted - 172 - ERROR -  -     exec(code, run_globals)
2019-11-07 23:04:57,381 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 659, in <module>
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -     main()
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 654, in main
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -     hs = setup(sys.argv[1:])
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 362, in setup
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -     with hs.get_db_conn(run_new_connection=False) as db_conn:
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/server.py", line 433, in get_db_conn
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -     db_conn = self.database_engine.module.connect(**db_params)
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  - psycopg2.OperationalError: could not connect to server: Host is unreachable
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -    Is the server running on host "matrix-postgres" (172.18.0.3) and accepting
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -    TCP/IP connections on port 5432?
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -

According to https://github.com/moby/moby/issues/32138 and stackoverflow I had to configure interface docker0 for firewalld's trused zone

firewall-cmd --permanent --zone=trusted --add-interface=docker0
firewall-cmd --permanent --zone=trusted --add-interface=br-308912342a48
firewall-cmd --permanent --zone=trusted --add-interface=br-91ad7930f0d8
firewall-cmd --reload
service firewalld restart
spantaleev commented 4 years ago

Thanks for doing this research! It would surely be helpful for adding CentOS 8 support!

For now, I'm worried about Docker CE not being available for CentOS 8 and needing to resort to hacks to install it. I guess that's how you did it?

I'm guessing that once this gets solved upstream in a proper manner, it may integrate with firewalld more nicely and we may not need to mark Docker interfaces as trusted.. Let's see..

Until the Docker installation issue is solved upstream, I'm refraining from doing any playbook fixes. If someone would like to patch up the playbook for the current situation though, please go ahead..

stfl commented 4 years ago

I used the docker-ce package for el7 from the docker repo and dnf needs the --nobest flag to chose a container.d version.

https://linuxconfig.org/how-to-install-docker-in-rhel-8

dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
dnf install --nobest docker-ce

Furthermore I noticed that docker-python is not available in the repos and needs to be added with pip.

hitchhooker commented 4 years ago

TASK [matrix-base : Ensure yum packages are installed] *************************** fatal: [matrix.rotko.net]: FAILED! => {"changed": false, "failures": ["No package docker-python available.", "No package ntp available."], "msg": ["Failed to install some of the specified packages"], "rc": 1, "results": []} how did you install these with pip?

stfl commented 4 years ago

I am not sure anymore.

Probably sth like pip install docker and removing the dependency from the ansible file

teutat3s commented 4 years ago

Just writing this down in case anybody gets redirected here:

I came across this issue too, but on CentOS 7 and with the new ma1sd fork of mxisd. It seems that server validation was added in ma1sd and it tries to get the .well-known records from https://your.domain.

For this to work and not get the Host unreachable error, I also had to add the matrix docker network interface to the trusted zone with firewall-cmd

pemontto commented 4 years ago

Had similar issues with docker-python on CentOS 7 with docker-python, there are no packages available. I needed to install python-pip from epel then pip install docker in roles/matrix-base/tasks/server_base/setup_centos.yml

- name: Ensure yum packages are installed
  yum:
    name:
      - "{{ 'python3-pip' if ansible_python_version is version('3', '>=') else 'python2-pip'}}"
      - "{{ matrix_ntpd_package }}"
      - fuse
    state: latest
    update_cache: yes
    enablerepo: epel

- name: Ensure pip docker module is installed
  pip:
    name: "{{ 'docker' if ansible_python_version is version('2.7', '>=') else 'docker-py'}}"
spantaleev commented 4 years ago

It's probably better for people installing on CentOS 8 to:

firewall-cmd --permanent --zone=trusted --add-interface=docker0

# Add the id of the `matrix` Docker network below
firewall-cmd --permanent --zone=trusted --add-interface=br-<SOME_ID_HERE>

# Add the id of the `matrix-coturn` Docker network below
firewall-cmd --permanent --zone=trusted --add-interface=br-<SOME_ID_HERE>

firewall-cmd --reload
systemctl restart firewalld
# Disable Docker installation. We've done it manually.
matrix_docker_installation_enabled: false

# Switch from ntpd to Chrony
matrix_ntpd_package: chrony
matrix_ntpd_service: chronyd
hitchhooker commented 4 years ago

has anyone tried to replace docker in centos8 with better supported podman & podman-compose? alias podman=docker; alias podman-compose=docker-compose do not seem to do the trick here, but receiving error: fatal: [matrix.rotko.net]: FAILED! => {"changed": false, "msg": "Could not find the requested service docker: host"} anybody more familiar with podman and this script, is this something easily done or should I just go with the docker instead of figuring it out?

E: Seems like issue I was looking for was Issue 64

spantaleev commented 4 years ago

podman's networking is completely different.

alias podman=docker works in the very simplest of use-cases, which don't involve multiple containers and cross-container networking.

spantaleev commented 3 years ago

Since d556aa943fbd7, our docker-ce.repo no longer hardcodes 7, but uses the $releasever variable, so installing docker-ce on CentOS 8 should work better now.

My limited testing with Docker 19.03.04 on CentOS 8 (for other things, not the playbook) shows that networking works fine without changing firewalld configuration. But I may be testing on a distro with some other defaults.

docker-python still seems to be unavailable. This may be the biggest showstopper for us right now, assuming networking really works well out of the box now and firewalld changes are not necessary.

I'm guessing the other things (ntpd package) are still an issue, but those are easy to fix.

blaztinn commented 3 years ago

I've managed to install docker-py and ntp (chrony) on CentOS 8 Stream with the following draft PR: https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1105

derhagen commented 8 months ago

Can this issue be closed then? And should CentOS 8 be marked as working in https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/prerequisites.md ?