hachem2001 commented 1 year ago

Description

PROBLEM

Upon updating from synapse v1.89.0 to synapse 1.90.0, the home server stops at startup on 504 No response error during OIDC configuration setup.

ANALYSIS

Well I checked the code that changed for OIDC metadata loading since 1.89.0 : nothing absurd. Then I thought of verifying whether a certain library version update might have caused the issue, and indeed : only one single dependency change is responsible for the error, that of service-identity which changed from version = "21.1.0" to version = "23.1.0" in poetry.lock

SOLUTION

Revert service-identity to 21.1.0, and lock the dependency version to less than 23.x.x And if you're motivated enough, try to figure out why the dependency version change caused this issue.

Steps to reproduce

have docker (24.0.50), npm (9.2.0), node (18.13.0), and docker-compose (2.18.2) installed.
clone this repository (thanks @jcabannes !)
npm install
npm run start:debug
Take note of synapse's docker service logs.

To see the difference, modify synapse's version from 1.90.0 to 1.89.0 in __test-data__/docker-compose.yml, the error should disappear.

Homeserver

local test

Synapse Version

1.90.0

Installation Method

Docker (matrixdotorg/synapse)

Database

SQLite

Workers

Single process

Platform

Debian 12 stable, amd64

Configuration

in homeserver.yaml

server_name: "example.com"
public_baseurl: "https://matrix.example.com/"
pid_file: /data/homeserver.pid
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    resources:
      - names: [client, federation]
        compress: false
database:
  name: sqlite3
  args:
    database: /data/homeserver.db
log_config: "/data/matrix.example.com.log.config"
media_store_path: /data/media_store
registration_shared_secret: "u+Q^i6&*Y9azZ*~pID^.a=qrvd+mUIBX9SAreEPGJ=xzP&c+Sk"
report_stats: false
macaroon_secret_key: "=0ws-1~ztzXm&xh+As;7YL5.-U~r-T,F4zR3mW#E;6Y::Rb7&G"
form_secret: "&YFO.XSc*2^2ZsW#hmoR+t:wf03~u#fin#O.R&erFcl9_mEayv"
signing_key_path: "/data/matrix.example.com.signing.key"
trusted_key_servers:
  - server_name: "matrix.org"
    accept_keys_insecurely: true
accept_keys_insecurely: true
app_service_config_files:
  - /data/registration.yaml
oidc_config:
  idp_id: lemonldap
  idp_name: lemonldap
  enabled: true
  issuer: "https://auth.example.com/"
  client_id: "matrix1"
  client_secret: "matrix1*"
  scopes: ["openid", "profile"]
  discover: true
  user_profile_method: "userinfo_endpoint"
  user_mapping_provider:
    config:
      subject_claim: "sub"
      localpart_template: "{{ user.preferred_username }}"
      display_name_template: "{{ user.name }}"

Relevant log output

Error during startup:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/oidc.py", line 140, in load_metadata
    await p.load_metadata()
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/oidc.py", line 588, in load_metadata
    return await self._provider_metadata.get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/util/caches/cached_call.py", line 136, in get
    return await self._cachedcall.get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/util/caches/cached_call.py", line 106, in get
    self._result.raiseException()
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 504, in raiseException
    raise self.value.with_traceback(self.tb)
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/util/caches/cached_call.py", line 126, in _wrapper
    return await f()
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/oidc.py", line 600, in _load_metadata
    metadata_response = await self._http_client.get_json(url)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/http/client.py", line 559, in get_json
    body = await self.get_raw(uri, args, headers=actual_headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/http/client.py", line 644, in get_raw
    response = await self.request("GET", uri, headers=Headers(actual_headers))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/http/client.py", line 411, in request
    response = await make_deferred_yieldable(request_deferred)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/http/client.py", line 952, in _timeout_to_request_timed_out_error
    raise RequestTimedOutError("Timeout waiting for response from remote server")
synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/app/_base.py", line 254, in wrapper
    await cb(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 362, in start
    await oidc.load_metadata()
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/oidc.py", line 144, in load_metadata
    raise Exception(
Exception: Error while initialising OIDC provider 'oidc-lemonldap'

Anything else that would be useful to know?

No response