On CentOS 8 Docker internal network needs to be set as trusted network for firewalld

[stfl]

I just set up on CentOS 8 amongst other minor problems like docker-ce install, chronyd instead of ntpd, user 991 not beeing available on a plain VM I discovered that synapse cannot connect to the postgres container on the docker internal network 172.18.0.0

This server is configured to use 'matrix.org' as its trusted key server via the
'trusted_key_servers' config option. 'matrix.org' is a good choice for a key
server since it is long-lived, stable and trusted. However, some admins may
wish to use another server for this purpose.

To suppress this warning and continue using 'matrix.org', admins should set
'suppress_key_server_warning' to 'true' in homeserver.yaml.
--------------------------------------------------------------------------------
2019-11-07 23:04:56,352 - root - 237 - WARNING - None - ***** STARTING SERVER *****
2019-11-07 23:04:56,352 - root - 238 - WARNING - None - Server /usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py version 1.5.1
2019-11-07 23:04:57,380 - twisted - 172 - ERROR -  - Traceback (most recent call last):
2019-11-07 23:04:57,380 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
2019-11-07 23:04:57,380 - twisted - 172 - ERROR -  -     "__main__", mod_spec)
2019-11-07 23:04:57,381 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
2019-11-07 23:04:57,381 - twisted - 172 - ERROR -  -     exec(code, run_globals)
2019-11-07 23:04:57,381 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 659, in <module>
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -     main()
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 654, in main
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -     hs = setup(sys.argv[1:])
2019-11-07 23:04:57,382 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 362, in setup
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -     with hs.get_db_conn(run_new_connection=False) as db_conn:
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/server.py", line 433, in get_db_conn
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -     db_conn = self.database_engine.module.connect(**db_params)
2019-11-07 23:04:57,383 - twisted - 172 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  - psycopg2.OperationalError: could not connect to server: Host is unreachable
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -    Is the server running on host "matrix-postgres" (172.18.0.3) and accepting
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -    TCP/IP connections on port 5432?
2019-11-07 23:04:57,384 - twisted - 172 - ERROR -  -

According to https://github.com/moby/moby/issues/32138 and stackoverflow I had to configure interface docker0 for firewalld's trused zone

firewall-cmd --permanent --zone=trusted --add-interface=docker0
firewall-cmd --permanent --zone=trusted --add-interface=br-308912342a48
firewall-cmd --permanent --zone=trusted --add-interface=br-91ad7930f0d8
firewall-cmd --reload
service firewalld restart

[spantaleev]

Thanks for doing this research! It would surely be helpful for adding CentOS 8 support!

For now, I'm worried about Docker CE not being available for CentOS 8 and needing to resort to hacks to install it. I guess that's how you did it?

I'm guessing that once this gets solved upstream in a proper manner, it may integrate with firewalld more nicely and we may not need to mark Docker interfaces as trusted.. Let's see..

Until the Docker installation issue is solved upstream, I'm refraining from doing any playbook fixes. If someone would like to patch up the playbook for the current situation though, please go ahead..

[stfl]

I used the docker-ce package for el7 from the docker repo and dnf needs the --nobest flag to chose a container.d version.

https://linuxconfig.org/how-to-install-docker-in-rhel-8

dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
dnf install --nobest docker-ce

Furthermore I noticed that docker-python is not available in the repos and needs to be added with pip.

[hitchhooker]

TASK [matrix-base : Ensure yum packages are installed] *************************** fatal: [matrix.rotko.net]: FAILED! => {"changed": false, "failures": ["No package docker-python available.", "No package ntp available."], "msg": ["Failed to install some of the specified packages"], "rc": 1, "results": []} how did you install these with pip?

[stfl]

I am not sure anymore.

Probably sth like pip install docker and removing the dependency from the ansible file

[teutat3s]

Just writing this down in case anybody gets redirected here:

I came across this issue too, but on CentOS 7 and with the new ma1sd fork of mxisd. It seems that server validation was added in ma1sd and it tries to get the .well-known records from https://your.domain.

For this to work and not get the Host unreachable error, I also had to add the matrix docker network interface to the trusted zone with firewall-cmd

[pemontto]

Had similar issues with docker-python on CentOS 7 with docker-python, there are no packages available. I needed to install python-pip from epel then pip install docker in roles/matrix-base/tasks/server_base/setup_centos.yml

- name: Ensure yum packages are installed
  yum:
    name:
      - "{{ 'python3-pip' if ansible_python_version is version('3', '>=') else 'python2-pip'}}"
      - "{{ matrix_ntpd_package }}"
      - fuse
    state: latest
    update_cache: yes
    enablerepo: epel

- name: Ensure pip docker module is installed
  pip:
    name: "{{ 'docker' if ansible_python_version is version('2.7', '>=') else 'docker-py'}}"

[spantaleev]

It's probably better for people installing on CentOS 8 to:

• ensure SELinux is disabled (applies to CentOS 7 as well)

• install Docker manually following some guide, like this. Note: it's not necessary to completely disable firewalld, as long as you mark the Docker network interfaces as trusted. We do it in a step below.

• install docker-python manually: dnf install python3-pip && pip3 install docker

• create the Docker networks manually (docker network create matrix && docker network create matrix-coturn). We do this in order to mark them as trusted in firewalld

• find the Docker network ids (we'll need them below): docker network ls

• mark the Docker networks as trusted:

firewall-cmd --permanent --zone=trusted --add-interface=docker0

# Add the id of the `matrix` Docker network below
firewall-cmd --permanent --zone=trusted --add-interface=br-<SOME_ID_HERE>

# Add the id of the `matrix-coturn` Docker network below
firewall-cmd --permanent --zone=trusted --add-interface=br-<SOME_ID_HERE>

firewall-cmd --reload
systemctl restart firewalld

• use the following vars.yml configuration:

# Disable Docker installation. We've done it manually.
matrix_docker_installation_enabled: false

# Switch from ntpd to Chrony
matrix_ntpd_package: chrony
matrix_ntpd_service: chronyd

• run the playbook installation normally

[hitchhooker]

has anyone tried to replace docker in centos8 with better supported podman & podman-compose? alias podman=docker; alias podman-compose=docker-compose do not seem to do the trick here, but receiving error: fatal: [matrix.rotko.net]: FAILED! => {"changed": false, "msg": "Could not find the requested service docker: host"} anybody more familiar with podman and this script, is this something easily done or should I just go with the docker instead of figuring it out?

E: Seems like issue I was looking for was Issue 64

[spantaleev]

podman's networking is completely different.

alias podman=docker works in the very simplest of use-cases, which don't involve multiple containers and cross-container networking.

[spantaleev]

Since d556aa943fbd7, our docker-ce.repo no longer hardcodes 7, but uses the $releasever variable, so installing docker-ce on CentOS 8 should work better now.

My limited testing with Docker 19.03.04 on CentOS 8 (for other things, not the playbook) shows that networking works fine without changing firewalld configuration. But I may be testing on a distro with some other defaults.

docker-python still seems to be unavailable. This may be the biggest showstopper for us right now, assuming networking really works well out of the box now and firewalld changes are not necessary.

I'm guessing the other things (ntpd package) are still an issue, but those are easy to fix.

[blaztinn]

I've managed to install docker-py and ntp (chrony) on CentOS 8 Stream with the following draft PR: https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1105

[derhagen]

Can this issue be closed then? And should CentOS 8 be marked as working in https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/prerequisites.md ?