search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.79k stars 2.13k forks source link

Matrix service won't start on clean install of Ubuntu server 20.04.2 #9683

Closed MasterJubei closed 3 years ago

MasterJubei commented 3 years ago

EDIT: The answer for me appears to be yes. Using the manual install it seems to work fine via sudo apt install -y lsb-release wget apt-transport-https The default package is broken? Is this even maintained? Or is it just broken on my end. Original post:

Installed via apt install synapse-server I ran the installer, put my server's name

Now when trying to start the service and getting the status I only get:

root@matrix:~# systemctl status matrix-synapse
● matrix-synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Thu 2021-03-25 01:26:42 UTC; 1min 44s ago
    Process: 17375 ExecStartPre=/usr/bin/python3 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserve>
    Process: 17389 ExecStart=/usr/bin/python3 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.y>
   Main PID: 17389 (code=exited, status=0/SUCCESS)

Mar 25 01:26:42 matrix synapse[17389]: synapse.storage.data_stores.main.event_push_actions: [None] Searching for strea>
Mar 25 01:26:42 matrix synapse[17389]: synapse.storage.data_stores.main.event_push_actions: [None] Found stream orderi>
Mar 25 01:26:42 matrix synapse[17389]: synapse.storage.data_stores: [None] Starting 'state' data store
Mar 25 01:26:42 matrix synapse[17389]: synapse.storage.data_stores: [None] Database 'master' prepared
Mar 25 01:26:42 matrix synapse[17389]: synapse.server: [None] Finished setting up.
Mar 25 01:26:42 matrix synapse[17389]: synapse.app.homeserver: [None] Scheduling stats reporting for 3 hour intervals
Mar 25 01:26:42 matrix synapse[17389]: synapse.app.homeserver: [] Running
Mar 25 01:26:42 matrix synapse[17389]: synapse.app.homeserver: [] Set file limit to: 524288
Mar 25 01:26:42 matrix synapse[17389]: synapse.config.tls: [] Loading TLS key from /etc/matrix-synapse/homeserver.tls.>
Mar 25 01:26:42 matrix systemd[1]: matrix-synapse.service: Succeeded.

Is the default installation broken on Ubuntu? If I try to curl 127.0.0.1:8008 I simply get Connection refused I did not change any configuration settings, this is a clean install. Trying to make a new user results in:

root@matrix:~# register_new_matrix_user -c /etc/matrix-synapse/homeserver.yaml http://localhost:8008
New user localpart [root]: admin
Password:
Confirm password:
Make admin [no]: yes
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f21361a0520>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8008): Max retries exceeded with url: /_matrix/client/r0/admin/register (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f21361a0520>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/register_new_matrix_user", line 22, in <module>
    main()
  File "/usr/lib/python3/dist-packages/synapse/_scripts/register_new_matrix_user.py", line 224, in main
    register_new_user(
  File "/usr/lib/python3/dist-packages/synapse/_scripts/register_new_matrix_user.py", line 137, in register_new_user
    request_registration(
  File "/usr/lib/python3/dist-packages/synapse/_scripts/register_new_matrix_user.py", line 47, in request_registration
    r = requests.get(url, verify=False)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8008): Max retries exceeded with url: /_matrix/client/r0/admin/register (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f21361a0520>: Failed to establish a new connection: [Errno 111] Connection refused'))
theAeon commented 3 years ago

Are you using the matrix hosted repository or the default one? I'm having issues myself with the matrix repository.

EDIT: to be entirely clear I'm getting

Mar 27 22:41:38 dendrite systemd[1]: matrix-synapse.service: Scheduled restart job, restart counter is at 4.
Mar 27 22:41:38 dendrite systemd[1]: Stopped Synapse Matrix homeserver.
Mar 27 22:41:38 dendrite systemd[1]: Starting Synapse Matrix homeserver...
Mar 27 22:41:40 dendrite matrix-synapse[9449]: /opt/venvs/matrix-synapse/lib/python3.8/site-packages/twisted/conch/ssh/>
Mar 27 22:41:40 dendrite matrix-synapse[9449]:   from cryptography.utils import int_from_bytes, int_to_bytes
Mar 27 22:41:41 dendrite matrix-synapse[9458]: /opt/venvs/matrix-synapse/lib/python3.8/site-packages/twisted/conch/ssh/>
Mar 27 22:41:41 dendrite matrix-synapse[9458]:   from cryptography.utils import int_from_bytes, int_to_bytes
Mar 27 22:41:41 dendrite matrix-synapse[9458]: Config is missing macaroon_secret_key

followed by a timeout reset. Nothing present in the homeserver.log.

MasterJubei commented 3 years ago

@theAeon I am using their repository for it to work, the one in Ubuntu does not work. It looks for a key like you are experiencing in the config file and there isn't one.

You might have to delete the homeserver.yaml after you uninstall before installing again.

theAeon commented 3 years ago

Oh to be entirely clear I never installed the default repo. I suppose what this means is that unless I'm being stupid this is a different bug. Weirdly enough the macaroon_secret_key isn't set and according to the config file

a secret which is used to sign access tokens. If none is specified,
# the registration_shared_secret is used, if one is given; otherwise,
# a secret key is derived from the signing key.

EDIT: Am being stupid, ignore.

richvdh commented 3 years ago 
Mar 27 22:41:41 dendrite matrix-synapse[9458]: Config is missing macaroon_secret_key

this is a warning, not an error. It is unrelated to your problem.

Can you try running synapse directly from the commandline using the same command that systemd would use, to see what happens?

theAeon commented 3 years ago

I got it. No issue.

richvdh commented 3 years ago

I got it. No issue.

so can I close this issue?

MasterJubei commented 3 years ago

I got it. No issue.

so can I close this issue?

The default configuration for Ubuntu 20.04 is still broken for the version in the repository, so I think it is best left open. Their issue seems to have been different. https://www.reddit.com/r/matrixdotorg/comments/mcmue7/is_the_default_installer_on_ubuntu_server_20042/gs5waix?utm_source=share&utm_medium=web2x&context=3

I just tried the ubuntu 20.04 apt installer. Did you read your systemd log what it tries to do at the end? It tries Loading TLS key from /etc/matrix-synapse/homeserver.tls.key. Have you seen that file in your setup? I don't, which is exactly why it crashes. If you disable tls with no_tls: true, it works... I am not sure why this is the default settings and why no certificates are created, nor do I know why they use synapse with tls, which is not encouraged to use by the devs afaik.

richvdh commented 3 years ago

@MasterJubei I'm afraid I'm very confused about what your issue is; please don't make me pick it out of a reddit thread.

If Synapse won't start, then the same question applies to you:

Can you try running synapse directly from the commandline using the same command that systemd would use, to see what happens?

MasterJubei commented 3 years ago

@richvdh 

2021-03-29 21:32:14,671 - root - 257 - WARNING - None- ***** STARTING SERVER *****
2021-03-29 21:32:14,671 - root - 258 - WARNING - None- Server /usr/lib/python3/dist-packages/synapse/app/homeserver.py version 1.11.0
2021-03-29 21:32:14,671 - root - 259 - INFO - None- Server hostname: matrix2.mydomain.net
2021-03-29 21:32:14,672 - synapse.app.homeserver - 340 - INFO - None- Setting up server
2021-03-29 21:32:14,672 - synapse.server - 240 - INFO - None- Setting up.
2021-03-29 21:32:14,674 - synapse.storage.data_stores - 48 - INFO - None- Preparing database 'master'...
2021-03-29 21:32:16,419 - synapse.storage.prepare_database - 309 - INFO - None- Upgrading schema to v55
2021-03-29 21:32:16,419 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 55/access_token_expiry.sql
2021-03-29 21:32:16,421 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 55/track_threepid_validations.sql
2021-03-29 21:32:16,421 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 55/users_alter_deactivated.sql
2021-03-29 21:32:16,423 - synapse.storage.prepare_database - 309 - INFO - None- Upgrading schema to v56
2021-03-29 21:32:16,423 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/add_spans_to_device_lists.sql
2021-03-29 21:32:16,425 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/current_state_events_membership.sql
2021-03-29 21:32:16,426 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/current_state_events_membership_mk2.sql
2021-03-29 21:32:16,426 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/delete_keys_from_deleted_backups.sql
2021-03-29 21:32:16,426 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/destinations_failure_ts.sql
2021-03-29 21:32:16,428 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/device_stream_id_insert.sql
2021-03-29 21:32:16,428 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/devices_last_seen.sql
2021-03-29 21:32:16,432 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/drop_unused_event_tables.sql
2021-03-29 21:32:16,432 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/event_expiry.sql
2021-03-29 21:32:16,433 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/event_labels.sql
2021-03-29 21:32:16,433 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/event_labels_background_update.sql
2021-03-29 21:32:16,433 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/fix_room_keys_index.sql
2021-03-29 21:32:16,434 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/hidden_devices.sql
2021-03-29 21:32:16,435 - synapse.storage.prepare_database - 393 - INFO - None- Applying engine-specific schema 56/hidden_devices_fix.sql.sqlite
2021-03-29 21:32:16,440 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/nuke_empty_communities_from_db.sql
2021-03-29 21:32:16,441 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/public_room_list_idx.sql
2021-03-29 21:32:16,441 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/redaction_censor.sql
2021-03-29 21:32:16,443 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/redaction_censor2.sql
2021-03-29 21:32:16,444 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/redaction_censor4.sql
2021-03-29 21:32:16,444 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/remove_tombstoned_rooms_from_directory.sql
2021-03-29 21:32:16,444 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/room_key_etag.sql
2021-03-29 21:32:16,446 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/room_membership_idx.sql
2021-03-29 21:32:16,446 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/room_retention.sql
2021-03-29 21:32:16,447 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/signing_keys.sql
2021-03-29 21:32:16,448 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/signing_keys_nonunique_signatures.sql
2021-03-29 21:32:16,448 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/state_group_room_idx.sql
2021-03-29 21:32:16,448 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/stats_separated.sql
2021-03-29 21:32:16,450 - synapse.storage.prepare_database - 378 - INFO - None- Running script 56/unique_user_filter_index.py
2021-03-29 21:32:16,488 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/user_external_ids.sql
2021-03-29 21:32:16,489 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 56/users_in_public_rooms_idx.sql
2021-03-29 21:32:16,489 - synapse.storage.prepare_database - 309 - INFO - None- Upgrading schema to v57
2021-03-29 21:32:16,489 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 57/delete_old_current_state_events.sql
2021-03-29 21:32:16,490 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 57/device_list_remote_cache_stale.sql
2021-03-29 21:32:16,490 - synapse.storage.prepare_database - 378 - INFO - None- Running script 57/local_current_membership.py
2021-03-29 21:32:16,490 - synapse.storage.prepare_database - 389 - INFO - None- Applying schema 57/rooms_version_column.sql
2021-03-29 21:32:16,492 - synapse.storage.prepare_database - 393 - INFO - None- Applying engine-specific schema 57/rooms_version_column_2.sql.sqlite
2021-03-29 21:32:16,502 - synapse.storage.data_stores - 58 - INFO - None- Starting 'main' data store
2021-03-29 21:32:16,504 - synapse.storage.data_stores.main.event_push_actions - 499 - INFO - None- Searching for stream ordering 1 month ago
2021-03-29 21:32:16,505 - synapse.storage.data_stores.main.event_push_actions - 503 - INFO - None- Found stream ordering 1 month ago: it's 0
2021-03-29 21:32:16,505 - synapse.storage.data_stores.main.event_push_actions - 506 - INFO - None- Searching for stream ordering 1 day ago
2021-03-29 21:32:16,505 - synapse.storage.data_stores.main.event_push_actions - 510 - INFO - None- Found stream ordering 1 day ago: it's 0
2021-03-29 21:32:16,507 - synapse.storage.data_stores - 68 - INFO - None- Starting 'state' data store
2021-03-29 21:32:16,507 - synapse.storage.data_stores - 81 - INFO - None- Database 'master' prepared
2021-03-29 21:32:16,508 - synapse.server - 243 - INFO - None- Finished setting up.
2021-03-29 21:32:16,515 - synapse.app.homeserver - 110 - INFO - - Running
2021-03-29 21:32:16,515 - synapse.app.homeserver - 30 - INFO - - Set file limit to: 524288
2021-03-29 21:32:16,516 - synapse.config.tls - 517 - INFO - - Loading TLS key from /etc/matrix-synapse/homeserver.tls.key
2021-03-29 21:32:28,026 - root - 257 - WARNING - None- ***** STARTING SERVER *****
2021-03-29 21:32:28,026 - root - 258 - WARNING - None- Server /usr/lib/python3/dist-packages/synapse/app/homeserver.py version 1.11.0
2021-03-29 21:32:28,026 - root - 259 - INFO - None- Server hostname: matrix2.3dslice.net
2021-03-29 21:32:28,026 - synapse.app.homeserver - 340 - INFO - None- Setting up server
2021-03-29 21:32:28,027 - synapse.server - 240 - INFO - None- Setting up.
2021-03-29 21:32:28,028 - synapse.storage.data_stores - 48 - INFO - None- Preparing database 'master'...
2021-03-29 21:32:28,030 - synapse.storage.prepare_database - 309 - INFO - None- Upgrading schema to v57
2021-03-29 21:32:28,032 - synapse.storage.data_stores - 58 - INFO - None- Starting 'main' data store
2021-03-29 21:32:28,034 - synapse.storage.data_stores.main.event_push_actions - 499 - INFO - None- Searching for stream ordering 1 month ago
2021-03-29 21:32:28,034 - synapse.storage.data_stores.main.event_push_actions - 503 - INFO - None- Found stream ordering 1 month ago: it's 0
2021-03-29 21:32:28,034 - synapse.storage.data_stores.main.event_push_actions - 506 - INFO - None- Searching for stream ordering 1 day ago
2021-03-29 21:32:28,035 - synapse.storage.data_stores.main.event_push_actions - 510 - INFO - None- Found stream ordering 1 day ago: it's 0
2021-03-29 21:32:28,036 - synapse.storage.data_stores - 68 - INFO - None- Starting 'state' data store
2021-03-29 21:32:28,037 - synapse.storage.data_stores - 81 - INFO - None- Database 'master' prepared
2021-03-29 21:32:28,037 - synapse.server - 243 - INFO - None- Finished setting up.
2021-03-29 21:32:28,044 - synapse.app.homeserver - 110 - INFO - - Running
2021-03-29 21:32:28,044 - synapse.app.homeserver - 30 - INFO - - Set file limit to: 524288
2021-03-29 21:32:28,045 - synapse.config.tls - 517 - INFO - - Loading TLS key from /etc/matrix-synapse/homeserver.tls.key

Here is the default config:

# vim:ft=yaml
# PEM encoded X509 certificate for TLS.
# You can replace the self-signed certificate that synapse
# autogenerates on launch with your own SSL certificate + key pair
# if you like.  Any required intermediary certificates can be
# appended after the primary certificate in hierarchical order.
tls_certificate_path: "/etc/matrix-synapse/homeserver.tls.crt"

# PEM encoded private key for TLS
tls_private_key_path: "/etc/matrix-synapse/homeserver.tls.key"

# PEM dh parameters for ephemeral keys
tls_dh_params_path: "/etc/matrix-synapse/homeserver.tls.dh"

# Don't bind to the https port
no_tls: False

# List of allowed TLS fingerprints for this server to publish along
# with the signing keys for this server. Other matrix servers that
# make HTTPS requests to this server will check that the TLS
# certificates returned by this server match one of the fingerprints.
#
# Synapse automatically adds the fingerprint of its own certificate
# to the list. So if federation traffic is handled directly by synapse
# then no modification to the list is required.
#
# If synapse is run behind a load balancer that handles the TLS then it
# will be necessary to add the fingerprints of the certificates used by
# the loadbalancers to this list if they are different to the one
# synapse is using.
#
# Homeservers are permitted to cache the list of TLS fingerprints
# returned in the key responses up to the "valid_until_ts" returned in
# key. It may be necessary to publish the fingerprints of a new
# certificate and wait until the "valid_until_ts" of the previous key
# responses have passed before deploying it.
#
# You can calculate a fingerprint from a given TLS listener via:
# openssl s_client -connect $host:$port < /dev/null 2> /dev/null |
#   openssl x509 -outform DER | openssl sha256 -binary | base64 | tr -d '='
# or by checking matrix.org/federationtester/api/report?server_name=$host
#
tls_fingerprints: []
# tls_fingerprints: [{"sha256": "<base64_encoded_sha256_fingerprint>"}]

## Server ##

# When running as a daemon, the file to store the pid in
pid_file: "/var/run/matrix-synapse.pid"

# CPU affinity mask. Setting this restricts the CPUs on which the
# process will be scheduled. It is represented as a bitmask, with the
# lowest order bit corresponding to the first logical CPU and the
# highest order bit corresponding to the last logical CPU. Not all CPUs
# may exist on a given system but a mask may specify more CPUs than are
# present.
#
# For example:
#    0x00000001  is processor #0,
#    0x00000003  is processors #0 and #1,
#    0xFFFFFFFF  is all processors (#0 through #31).
#
# Pinning a Python process to a single CPU is desirable, because Python
# is inherently single-threaded due to the GIL, and can suffer a
# 30-40% slowdown due to cache blow-out and thread context switching
# if the scheduler happens to schedule the underlying threads across
# different cores. See
# https://www.mirantis.com/blog/improve-performance-python-programs-restricting-single-cpu/.
#
# cpu_affinity: 0xFFFFFFFF

# The path to the web client which will be served at /_matrix/client/
# if 'webclient' is configured under the 'listeners' configuration.
#
# web_client_location: "/path/to/web/root"

# The public-facing base URL for the client API (not including _matrix/...)
# public_baseurl: https://example.com:8448/

# Set the soft limit on the number of file descriptors synapse can use
# Zero is used to indicate synapse should set the soft limit to the
# hard limit.
soft_file_limit: 0

# Set to false to disable presence tracking on this homeserver.
#
#use_presence: false

# Whether to require authentication to retrieve profile data (avatars,
# display names) of other users through the client API. Defaults to
# 'false'. Note that profile data is also available via the federation
# API, so this setting is of limited value if federation is enabled on
# the server.
#
#require_auth_for_profile_requests: true

# Uncomment to require a user to share a room with another user in order
# to retrieve their profile information. Only checked on Client-Server
# requests. Profile requests from other servers should be checked by the
# requesting server. Defaults to 'false'.
#
#limit_profile_requests_to_users_who_share_rooms: true

# If set to 'true', removes the need for authentication to access the server's
# public rooms directory through the client API, meaning that anyone can
# query the room directory. Defaults to 'false'.
#
#allow_public_rooms_without_auth: true

# If set to 'true', allows any other homeserver to fetch the server's public
# rooms directory via federation. Defaults to 'false'.
#
#allow_public_rooms_over_federation: true

# The default room version for newly created rooms.
#
# Known room versions are listed here:
# https://matrix.org/docs/spec/#complete-list-of-room-versions
#
# For example, for room version 1, default_room_version should be set
# to "1".
#
#default_room_version: "5"

# The GC threshold parameters to pass to `gc.set_threshold`, if defined
# gc_thresholds: [700, 10, 10]

# Set the limit on the returned events in the timeline in the get
# and sync operations. The default value is -1, means no upper limit.
# filter_timeline_limit: 5000

# Whether room invites to users on this server should be blocked
# (except those sent by local server admins). The default is False.
# block_non_admin_invites: True

# Restrict federation to the following whitelist of domains.
# N.B. we recommend also firewalling your federation listener to limit
# inbound federation traffic as early as possible, rather than relying
# purely on this application-layer restriction.  If not specified, the
# default is to whitelist everything.
#
# federation_domain_whitelist:
#  - lon.example.com
#  - nyc.example.com
#  - syd.example.com

# Prevent federation requests from being sent to the following
# blacklist IP address CIDR ranges. If this option is not specified, or
# specified with an empty list, no ip range blacklist will be enforced.
#
# As of Synapse v1.4.0 this option also affects any outbound requests to identity
# servers provided by user input.
#
# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
# listed here, since they correspond to unroutable addresses.)
#
federation_ip_range_blacklist:
  - '127.0.0.0/8'
  - '10.0.0.0/8'
  - '172.16.0.0/12'
  - '192.168.0.0/16'
  - '100.64.0.0/10'
  - '169.254.0.0/16'
  - '::1/128'
  - 'fe80::/64'
  - 'fc00::/7'

# List of ports that Synapse should listen on, their purpose and their
# configuration.
listeners:
  # Main HTTPS listener
  # For when matrix traffic is sent directly to synapse.
  -
    # The port to listen for HTTPS requests on.
    port: 8448

    # Local addresses to listen on.
    # On Linux and Mac OS, `::` will listen on all IPv4 and IPv6
    # addresses by default. For most other OSes, this will only listen
    # on IPv6.
    bind_addresses:
      - '::1'
      - '127.0.0.1'
      # - '::'
      # - '0.0.0.0'

    # This is a 'http' listener, allows us to specify 'resources'.
    type: http

    tls: true

    # Use the X-Forwarded-For (XFF) header as the client IP and not the
    # actual client IP.
    x_forwarded: false

    # List of HTTP resources to serve on this listener.
    resources:
      -
        # List of resources to host on this listener.
        names:
          - client     # The client-server APIs, both v1 and v2

        # Should synapse compress HTTP responses to clients that support it?
        # This should be disabled if running synapse behind a load balancer
        # that can do automatic compression.
        compress: true

      - names: [federation]  # Federation APIs
        compress: false

    # optional list of additional endpoints which can be loaded via
    # dynamic modules
    # additional_resources:
    #   "/_matrix/my/custom/endpoint":
    #     module: my_module.CustomRequestHandler
    #     config: {}

  # Unsecure HTTP listener,
  # For when matrix traffic passes through loadbalancer that unwraps TLS.
  - port: 8008
    tls: false
    bind_addresses:
      - '::1'
      - '127.0.0.1'
      # - '::'
      # - '0.0.0.0'
    type: http

    x_forwarded: false

    resources:
      - names: [client]
        compress: true
      - names: [federation]
        compress: false

  # Turn on the twisted ssh manhole service on localhost on the given
  # port.
  # - port: 9000
  #   bind_addresses:
  #     - '::1'
  #     - '127.0.0.1'
  #   type: manhole

# Database configuration
database:
  # The database engine name
  name: "sqlite3"
  # Arguments to pass to the engine
  args:
    # Path to the database
    database: "/var/lib/matrix-synapse/homeserver.db"

# Number of events to cache in memory.
event_cache_size: "10K"

# A yaml python logging config file
log_config: "/etc/matrix-synapse/log.yaml"

## Ratelimiting ##

# Number of messages a client can send per second
rc_messages_per_second: 0.2

# Number of message a client can send before being throttled
rc_message_burst_count: 10.0

# The federation window size in milliseconds
federation_rc_window_size: 1000

# The number of federation requests from a single server in a window
# before the server will delay processing the request.
federation_rc_sleep_limit: 10

# The duration in milliseconds to delay processing events from
# remote servers by if they go over the sleep limit.
federation_rc_sleep_delay: 500

# The maximum number of concurrent federation requests allowed
# from a single server
federation_rc_reject_limit: 50

# The number of federation requests to concurrently process from a
# single server
federation_rc_concurrent: 3

# Directory where uploaded images and attachments are stored.
media_store_path: "/var/lib/matrix-synapse/media"

# Media storage providers allow media to be stored in different
# locations.
# media_storage_providers:
# - module: file_system
#   # Whether to write new local files.
#   store_local: false
#   # Whether to write new remote media
#   store_remote: false
#   # Whether to block upload requests waiting for write to this
#   # provider to complete
#   store_synchronous: false
#   config:
#     directory: /mnt/some/other/directory

# Directory where in-progress uploads are stored.
uploads_path: "/var/lib/matrix-synapse/uploads"

# The largest allowed upload size in bytes
max_upload_size: "10M"

# Maximum number of pixels that will be thumbnailed
max_image_pixels: "32M"

# Whether to generate new thumbnails on the fly to precisely match
# the resolution requested by the client. If true then whenever
# a new resolution is requested by the client the server will
# generate a new thumbnail. If false the server will pick a thumbnail
# from a precalculated list.
dynamic_thumbnails: false

# List of thumbnail to precalculate when an image is uploaded.
thumbnail_sizes:
- width: 32
  height: 32
  method: crop
- width: 96
  height: 96
  method: crop
- width: 320
  height: 240
  method: scale
- width: 640
  height: 480
  method: scale
- width: 800
  height: 600
  method: scale

# Is the preview URL API enabled?  If enabled, you *must* specify
# an explicit url_preview_ip_range_blacklist of IPs that the spider is
# denied from accessing.
url_preview_enabled: False

# List of IP address CIDR ranges that the URL preview spider is denied
# from accessing.  There are no defaults: you must explicitly
# specify a list for URL previewing to work.  You should specify any
# internal services in your network that you do not want synapse to try
# to connect to, otherwise anyone in any Matrix room could cause your
# synapse to issue arbitrary GET requests to your internal services,
# causing serious security issues.
#
# url_preview_ip_range_blacklist:
# - '127.0.0.0/8'
# - '10.0.0.0/8'
# - '172.16.0.0/12'
# - '192.168.0.0/16'
# - '100.64.0.0/10'
# - '169.254.0.0/16'
#
# List of IP address CIDR ranges that the URL preview spider is allowed
# to access even if they are specified in url_preview_ip_range_blacklist.
# This is useful for specifying exceptions to wide-ranging blacklisted
# target IP ranges - e.g. for enabling URL previews for a specific private
# website only visible in your network.
#
# url_preview_ip_range_whitelist:
# - '192.168.1.1'

# Optional list of URL matches that the URL preview spider is
# denied from accessing.  You should use url_preview_ip_range_blacklist
# in preference to this, otherwise someone could define a public DNS
# entry that points to a private IP address and circumvent the blacklist.
# This is more useful if you know there is an entire shape of URL that
# you know that will never want synapse to try to spider.
#
# Each list entry is a dictionary of url component attributes as returned
# by urlparse.urlsplit as applied to the absolute form of the URL.  See
# https://docs.python.org/2/library/urlparse.html#urlparse.urlsplit
# The values of the dictionary are treated as an filename match pattern
# applied to that component of URLs, unless they start with a ^ in which
# case they are treated as a regular expression match.  If all the
# specified component matches for a given list item succeed, the URL is
# blacklisted.
#
# url_preview_url_blacklist:
# # blacklist any URL with a username in its URI
# - username: '*'
#
# # blacklist all *.google.com URLs
# - netloc: 'google.com'
# - netloc: '*.google.com'
#
# # blacklist all plain HTTP URLs
# - scheme: 'http'
#
# # blacklist http(s)://www.acme.com/foo
# - netloc: 'www.acme.com'
#   path: '/foo'
#
# # blacklist any URL with a literal IPv4 address
# - netloc: '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'

# The largest allowed URL preview spidering size in bytes
max_spider_size: "10M"

## Captcha ##
# See docs/CAPTCHA_SETUP for full details of configuring this.

# This Home Server's ReCAPTCHA public key.
recaptcha_public_key: "YOUR_PUBLIC_KEY"

# This Home Server's ReCAPTCHA private key.
recaptcha_private_key: "YOUR_PRIVATE_KEY"

# Enables ReCaptcha checks when registering, preventing signup
# unless a captcha is answered. Requires a valid ReCaptcha
# public/private key.
enable_registration_captcha: False

# A secret key used to bypass the captcha test entirely.
#captcha_bypass_secret: "YOUR_SECRET_HERE"

# The API endpoint to use for verifying m.login.recaptcha responses.
recaptcha_siteverify_api: "https://www.google.com/recaptcha/api/siteverify"

## Turn ##

# The public URIs of the TURN server to give to clients
turn_uris: []

# The shared secret used to compute passwords for the TURN server
turn_shared_secret: "YOUR_SHARED_SECRET"

# The Username and password if the TURN server needs them and
# does not use a token
#turn_username: "TURNSERVER_USERNAME"
#turn_password: "TURNSERVER_PASSWORD"

# How long generated TURN credentials last
turn_user_lifetime: "1h"

# Whether guests should be allowed to use the TURN server.
# This defaults to True, otherwise VoIP will be unreliable for guests.
# However, it does introduce a slight security risk as it allows users to
# connect to arbitrary endpoints without having first signed up for a
# valid account (e.g. by passing a CAPTCHA).
turn_allow_guests: False

## Registration ##

# Enable registration for new users.
enable_registration: False

# The user must provide all of the below types of 3PID when registering.
#
# registrations_require_3pid:
#     - email
#     - msisdn

# Mandate that users are only allowed to associate certain formats of
# 3PIDs with accounts on this server.
#
# allowed_local_3pids:
#     - medium: email
#       pattern: ".*@matrix\.org"
#     - medium: email
#       pattern: ".*@vector\.im"
#     - medium: msisdn
#       pattern: "\+44"

# If set, allows registration by anyone who also has the shared
# secret, even if registration is otherwise disabled.
# registration_shared_secret: <PRIVATE STRING>

# Set the number of bcrypt rounds used to generate password hash.
# Larger numbers increase the work factor needed to generate the hash.
# The default number is 12 (which equates to 2^12 rounds).
# N.B. that increasing this will exponentially increase the time required
# to register or login - e.g. 24 => 2^24 rounds which will take >20 mins.
bcrypt_rounds: 12

# Allows users to register as guests without a password/email/etc, and
# participate in rooms hosted on this server which have been made
# accessible to anonymous users.
allow_guest_access: False

# The list of identity servers trusted to verify third party
# identifiers by this server.
#
# Also defines the ID server which will be called when an account is
# deactivated (one will be picked arbitrarily).
#
# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
# background migration script, informing itself that the identity server all of its
# 3PIDs have been bound to is likely one of the below.
#
# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
# it is now solely used for the purposes of the background migration script, and can be
# removed once it has run.
trusted_third_party_id_servers:
    - matrix.org
    - vector.im
    - riot.im

# Users who register on this homeserver will automatically be joined
# to these rooms
#auto_join_rooms:
#    - "#example:example.com"

## Metrics ###

# Enable collection and rendering of performance metrics
enable_metrics: False

## API Configuration ##

# A list of event types that will be included in the room_invite_state
room_invite_state_types:
    - "m.room.join_rules"
    - "m.room.canonical_alias"
    - "m.room.avatar"
    - "m.room.name"

# A list of application service config file to use
app_service_config_files: []

# macaroon_secret_key: <PRIVATE STRING>

# Used to enable access token expiration.
expire_access_token: False

## Signing Keys ##

# Path to the signing key to sign messages with
signing_key_path: "/etc/matrix-synapse/homeserver.signing.key"

# The keys that the server used to sign messages with but won't use
# to sign new messages. E.g. it has lost its private key
old_signing_keys: {}
#  "ed25519:auto":
#    # Base64 encoded public key
#    key: "The public part of your old signing key."
#    # Millisecond POSIX timestamp when the key expired.
#    expired_ts: 123456789123

# How long key response published by this server is valid for.
# Used to set the valid_until_ts in /key/v2 APIs.
# Determines how quickly servers will query to check which keys
# are still valid.
key_refresh_interval: "1d" # 1 Day.

# The trusted servers to download signing keys from.
#
# When we need to fetch a signing key, each server is tried in parallel.
#
# Normally, the connection to the key server is validated via TLS certificates.
# Additional security can be provided by configuring a `verify key`, which
# will make synapse check that the response is signed by that key.
#
# This setting supercedes an older setting named `perspectives`. The old format
# is still supported for backwards-compatibility, but it is deprecated.
#
# 'trusted_key_servers' defaults to matrix.org, but using it will generate a
# warning on start-up. To suppress this warning, set
# 'suppress_key_server_warning' to true.
#
# Options for each entry in the list include:
#
#    server_name: the name of the server. required.
#
#    verify_keys: an optional map from key id to base64-encoded public key.
#       If specified, we will check that the response is signed by at least
#       one of the given keys.
#
#    accept_keys_insecurely: a boolean. Normally, if `verify_keys` is unset,
#       and federation_verify_certificates is not `true`, synapse will refuse
#       to start, because this would allow anyone who can spoof DNS responses
#       to masquerade as the trusted key server. If you know what you are doing
#       and are sure that your network environment provides a secure connection
#       to the key server, you can set this to `true` to override this
#       behaviour.
#
# An example configuration might look like:
#
#trusted_key_servers:
#  - server_name: "my_trusted_server.example.com"
#    verify_keys:
#      "ed25519:auto": "abcdefghijklmnopqrstuvwxyzabcdefghijklmopqr"
#  - server_name: "my_other_trusted_server.example.com"
#
trusted_key_servers:
  - server_name: "matrix.org"

# Uncomment the following to disable the warning that is emitted when the
# trusted_key_servers include 'matrix.org'. See above.
#
#suppress_key_server_warning: true

# The signing keys to use when acting as a trusted key server. If not specified
# defaults to the server signing key.
#
# Can contain multiple keys, one per line.
#
#key_server_signing_keys_path: "key_server_signing_keys.key"

# Enable SAML2 for registration and login. Uses pysaml2.
#
# At least one of `sp_config` or `config_path` must be set in this section to
# enable SAML login.
#
# (You will probably also want to set the following options to `false` to
# disable the regular login/registration flows:
#   * enable_registration
#   * password_config.enabled
#
# Once SAML support is enabled, a metadata file will be exposed at
# https://<server>:<port>/_matrix/saml2/metadata.xml, which you may be able to
# use to configure your SAML IdP with. Alternatively, you can manually configure
# the IdP to use an ACS location of
# https://<server>:<port>/_matrix/saml2/authn_response.
#
saml2_config:
  # `sp_config` is the configuration for the pysaml2 Service Provider.
  # See pysaml2 docs for format of config.
  #
  # Default values will be used for the 'entityid' and 'service' settings,
  # so it is not normally necessary to specify them unless you need to
  # override them.
  #
  #sp_config:
  #  # point this to the IdP's metadata. You can use either a local file or
  #  # (preferably) a URL.
  #  metadata:
  #    #local: ["saml2/idp.xml"]
  #    remote:
  #      - url: https://our_idp/metadata.xml
  #
  #    # By default, the user has to go to our login page first. If you'd like
  #    # to allow IdP-initiated login, set 'allow_unsolicited: true' in a
  #    # 'service.sp' section:
  #    #
  #    #service:
  #    #  sp:
  #    #    allow_unsolicited: true
  #
  #    # The examples below are just used to generate our metadata xml, and you
  #    # may well not need them, depending on your setup. Alternatively you
  #    # may need a whole lot more detail - see the pysaml2 docs!
  #
  #    description: ["My awesome SP", "en"]
  #    name: ["Test SP", "en"]
  #
  #    organization:
  #      name: Example com
  #      display_name:
  #        - ["Example co", "en"]
  #      url: "http://example.com"
  #
  #    contact_person:
  #      - given_name: Bob
  #        sur_name: "the Sysadmin"
  #        email_address": ["admin@example.com"]
  #        contact_type": technical

  # Instead of putting the config inline as above, you can specify a
  # separate pysaml2 configuration file:
  #
  #config_path: "CONFDIR/sp_conf.py"

  # the lifetime of a SAML session. This defines how long a user has to
  # complete the authentication process, if allow_unsolicited is unset.
  # The default is 5 minutes.
  #
  #saml_session_lifetime: 5m

  # The SAML attribute (after mapping via the attribute maps) to use to derive
  # the Matrix ID from. 'uid' by default.
  #
  #mxid_source_attribute: displayName

  # The mapping system to use for mapping the saml attribute onto a matrix ID.
  # Options include:
  #  * 'hexencode' (which maps unpermitted characters to '=xx')
  #  * 'dotreplace' (which replaces unpermitted characters with '.').
  # The default is 'hexencode'.
  #
  #mxid_mapping: dotreplace

  # In previous versions of synapse, the mapping from SAML attribute to MXID was
  # always calculated dynamically rather than stored in a table. For backwards-
  # compatibility, we will look for user_ids matching such a pattern before
  # creating a new account.
  #
  # This setting controls the SAML attribute which will be used for this
  # backwards-compatibility lookup. Typically it should be 'uid', but if the
  # attribute maps are changed, it may be necessary to change it.
  #
  # The default is 'uid'.
  #
  #grandfathered_mxid_source_attribute: upn

# Enable CAS for registration and login.
#cas_config:
#   enabled: true
#   server_url: "https://cas-server.com"
#   service_url: "https://homeserver.domain.com:8448"
#   #required_attributes:
#   #    name: value

# The JWT needs to contain a globally unique "sub" (subject) claim.
#
# jwt_config:
#    enabled: true
#    secret: "a secret"
#    algorithm: "HS256"

# Enable password for login.
password_config:
   enabled: true
   # Uncomment and change to a secret random string for extra security.
   # DO NOT CHANGE THIS AFTER INITIAL SETUP!
   #pepper: ""

# Enable sending emails for notification events
# Defining a custom URL for Riot is only needed if email notifications
# should contain links to a self-hosted installation of Riot; when set
# the "app_name" setting is ignored.
#
# If your SMTP server requires authentication, the optional smtp_user &
# smtp_pass variables should be used
#
#email:
#   enable_notifs: false
#   smtp_host: "localhost"
#   smtp_port: 25
#   smtp_user: "exampleusername"
#   smtp_pass: "examplepassword"
#   require_transport_security: False
#   notif_from: "Your Friendly %(app)s Home Server <noreply@example.com>"
#   app_name: Matrix
#   template_dir: res/templates
#   notif_template_html: notif_mail.html
#   notif_template_text: notif_mail.txt
#   notif_for_new_users: True
#   riot_base_url: "http://localhost/riot"

# password_providers:
#     - module: "ldap_auth_provider.LdapAuthProvider"
#       config:
#         enabled: true
#         uri: "ldap://ldap.example.com:389"
#         start_tls: true
#         base: "ou=users,dc=example,dc=com"
#         attributes:
#            uid: "cn"
#            mail: "email"
#            name: "givenName"
#         #bind_dn:
#         #bind_password:
#         #filter: "(objectClass=posixAccount)"

# Clients requesting push notifications can either have the body of
# the message sent in the notification poke along with other details
# like the sender, or just the event ID and room ID (`event_id_only`).
# If clients choose the former, this option controls whether the
# notification request includes the content of the event (other details
# like the sender are still included). For `event_id_only` push, it
# has no effect.

# For modern android devices the notification content will still appear
# because it is loaded by the app. iPhone, however will send a
# notification saying only that a message arrived and who it came from.
#
#push:
#   include_content: true

# spam_checker:
#     module: "my_custom_project.SuperSpamChecker"
#     config:
#         example_option: 'things'

# Whether to allow non server admins to create groups on this server
enable_group_creation: false

# If enabled, non server admins can only create groups with local parts
# starting with this prefix
# group_creation_prefix: "unofficial/"

# User Directory configuration
#
# 'search_all_users' defines whether to search all users visible to your HS
# when searching the user directory, rather than limiting to users visible
# in public rooms.  Defaults to false.  If you set it True, you'll have to run
# UPDATE user_directory_stream_pos SET stream_id = NULL;
# on your database to tell it to rebuild the user_directory search indexes.
#
#user_directory:
#   search_all_users: false

This is a clean install of matrix-synapse on Ubuntu's repository version. The default config does not allow it to start since it looking for a key that is not there. Anyway my issue is since the default configuration on the Ubuntu repository version is broken, why is it not fixed? I got matrix-synapse working fine with the manual repository, but the default one shouldn't be broken by default.

richvdh commented 3 years ago

This is a clean install of matrix-synapse on Ubuntu's repository version.

sigh. When you file an issue, please make it clear which version of the software you are talking about, to avoid wasting everyone's time.

We are not responsible for Ubuntu, nor its decision to ship broken packages. You may like to upvote https://bugs.launchpad.net/ubuntu/+source/matrix-synapse/+bug/1848709.

josteink commented 2 years ago

I had a similar issue after dist-upgrading from Ubuntu 18.04 to Ubuntu 20.04.

This involved a change in both Python-version and in PostgreSQL version in my setup, and Matrix-synapse failed to start after the upgrade.

After manually completing the PostegreSQL upgrade, the matrix-synapse service still failed to start and kept on auto-restarting.

However after re-enabling the matrix.org apt-repo, the following commands solved this issue for me:

sudo pip3 install psycopg2 --upgrade
sudo apt reinstall matrix-synapse-py3

Hopefully this can help others too :smile: