Federation errors on Dendrite Homserver

[matrixbot]

This issue was originally created by @sytallax at https://github.com/matrix-org/dendrite/issues/3352.

Background information

• Dendrite version or git SHA: Release v0.13.6
• SQLite3 or Postgres?: Postgres
• Running in Docker?: No
• go version: 1.22.2
• Client used (if applicable): Element and Cinny

Description

• What is the problem: Homeserver fails federation tester and its users cannot direct message matrix.org users
• Who is affected: Homeserver admin (me)
• How is this bug manifesting: Federation tester errors on "Version" record. Homeserver users cannot resolve to matrix.org users and vice versa
• When did this first appear: Homeserver has never federated. The homeserver went live one hour before posting

Steps to reproduce

• Followed the installation guide on the Matrix github.io site.
• I use mydomain.net in place of my actual domain going forward
• My homeserver is mydomain.net instead of the usual matrix.mydomain.net
• Dendrite listens on port 8008 (firewall allows ports 80, 443, 8448, 8008)
• Used nginx for my reverse proxy
• Used python3-certbot-nginx Debian package to create ssl certificate for mydomain.net
• Performed a manual installation of Dendrite
• Config files and relevent logs found below

Federation test raw JSON output

{
  "WellKnownResult": {
    "m.server": "mydomain.net:443",
    "CacheExpiresAt": 0
  },
  "DNSResult": {
    "SRVSkipped": true,
    "SRVCName": "",
    "SRVRecords": null,
    "SRVError": null,
    "Hosts": {},
    "Addrs": null
  },
  "ConnectionReports": {},
  "ConnectionErrors": {},
  "Version": {
    "error": "Get \"matrix://mydomain.net/_matrix/federation/v1/version\": dial tcp: lookup mydomain.net on 8.8.8.8:53: no such host"
  },
  "FederationOK": false
}

Relevent logs

Attempted to direct message my matrix.org account

Apr 05 22:15:37 REDACTED dendrite[53766]: time="2024-04-05T22:15:37.065414664Z" level=error msg="getProfile failed" error="contents=[123 34 101 114 114 99 111 100 101 34 58 34 77 95 85 78 65 85 84 72 79 82 73 90 69 68 34 44 34 101 114 114 111 114 34 58 34 70 97 105 108 101 100 32 116 111 32 102 105 110 100 32 97 110 121 32 107 101 121 32 116 111 32 115 97 116 105 115 102 121 58 32 95 70 101 116 99 104 75 101 121 82 101 113 117 101 115 116 40 115 101 114 118 101 114 95 110 97 109 101 61 39 105 98 114 97 104 105 109 109 117 102 116 101 101 46 110 101 116 39 44 32 109 105 110 105 109 117 109 95 118 97 108 105 100 95 117 110 116 105 108 95 116 115 61 49 55 49 50 51 53 53 51 51 57 54 57 51 44 32 107 101 121 95 105 100 115 61 91 39 101 100 50 53 53 49 57 58 112 121 75 67 117 106 39 93 41 34 125] msg=Failed to GET JSON (hostname \"matrix.org\" path \"/_matrix/federation/v1/query/profile\") code=401 wrapped=M_UNAUTHORIZED: Failed to find any key to satisfy: _FetchKeyRequest(server_name='mydomain.net', minimum_valid_until_ts=1712355339693, key_ids=['ed25519:pyKCuj'])" req.id=Q3DwDWHaf9vT req.method=GET req.path="/_matrix/client/v3/profile/@sytallax:matrix.org"

Apr 05 22:15:37 REDACTED dendrite[53766]: time="2024-04-05T22:15:37.317436431Z" level=error msg="Failed to manually update device lists for user" error="DeviceKeysForUser @sytallax:matrix.org returned no keys but wanted all keys, falling back to remote" server=matrix.org user_id="@sytallax:matrix.org"

Apr 05 22:15:51 REDACTED dendrite[53766]: time="2024-04-05T22:15:51.364564887Z" level=info msg="Sending invite" destination=matrix.org event_id="$gcd4br-I9D9EJN43ReadlSBI1ROVC-8EPS2l7GQlisc" room_id="!ooT0edQIcx1yyOQD:mydomain.net" room_version=10 user_id="@sytallax:matrix.org"

Apr 05 22:15:51 REDACTED dendrite[53766]: time="2024-04-05T22:15:51.773934479Z" level=error msg="fedClient.SendInvite failed" error="r.federation.SendInviteV2: failed to send invite: contents=[123 34 101 114 114 99 111 100 101 34 58 34 77 95 85 78 65 85 84 72 79 82 73 90 69 68 34 44 34 101 114 114 111 114 34 58 34 70 97 105 108 101 100 32 116 111 32 102 105 110 100 32 97 110 121 32 107 101 121 32 116 111 32 115 97 116 105 115 102 121 58 32 95 70 101 116 99 104 75 101 121 82 101 113 117 101 115 116 40 115 101 114 118 101 114 95 110 97 109 101 61 39 105 98 114 97 104 105 109 109 117 102 116 101 101 46 110 101 116 39 44 32 109 105 110 105 109 117 109 95 118 97 108 105 100 95 117 110 116 105 108 95 116 115 61 49 55 49 50 51 53 53 51 53 52 49 50 56 44 32 107 101 121 95 105 100 115 61 91 39 101 100 50 53 53 49 57 58 112 121 75 67 117 106 39 93 41 34 125] msg=Failed to PUT JSON (hostname \"matrix.org\" path \"/_matrix/federation/v2/invite/!ooT0edQIcx1yyOQD:mydomain.net/$gcd4br-I9D9EJN43ReadlSBI1ROVC-8EPS2l7GQlisc\") code=401 wrapped=M_UNAUTHORIZED: Failed to find any key to satisfy: _FetchKeyRequest(server_name='mydomain.net', minimum_valid_until_ts=1712355354128, key_ids=['ed25519:pyKCuj'])" event_id="$gcd4br-I9D9EJN43ReadlSBI1ROVC-8EPS2l7GQlisc" invitee="@sytallax:matrix.org" inviter="@REDACTED:mydomain.net" req.id=U7sV0JoWZZYC req.method=POST req.path=/_matrix/client/v3/createRoom room_id="!ooT0edQIcx1yyOQD:mydomain.net" user_id="@REDACTED:mydomain.net"

NGINX Config

server {

    server_name mydomain.net www.mydomain.net;

    root /var/www/mydomain.net;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mydomain.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain.net/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    listen 8448 ssl;
    listen [::]:8448 ssl;

    location ~* ^(\/_matrix|\/_synapse|\/_client) {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
                client_max_body_size 50M;
        }

        # These sections are required for client and federation discovery
        # (AKA: Client Well-Known URI)
        location /.well-known/matrix/client {
                return 200 '{"m.homeserver": {"base_url": "https://mydomain.net:443"}}';
                default_type application/json;
                add_header Access-Control-Allow-Origin *;
        }

        location /.well-known/matrix/server {
                return 200 '{"m.server": "ibrahimuftee.net:443"}';
                default_type application/json;
                add_header Access-Control-Allow-Origin *;
        }

}
server {
    if ($host = www.mydomain.net) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = mydomain.net) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;

    server_name mydomain.net www.mydomain.net;
    return 404; # managed by Certbot

}

NOTE: I modeled my nginx config after this sample config:

server {
        server_name example.org;

        listen 80;
        listen [::]:80;

        listen 443 ssl http2 default_server;
        listen [::]:443 ssl http2 default_server;

        listen 8448 ssl http2 default_server;
        listen [::]:8448 ssl http2 default_server;

        location ~* ^(\/_matrix|\/_synapse|\/_client) {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
                client_max_body_size 50M;
        }

        # These sections are required for client and federation discovery
        # (AKA: Client Well-Known URI)
        location /.well-known/matrix/client {
                return 200 '{"m.homeserver": {"base_url": "https://example.org"}}';
                default_type application/json;
                add_header Access-Control-Allow-Origin *;
        }

        location /.well-known/matrix/server {
                return 200 '{"m.server": "example.org:443"}';
                default_type application/json;
                add_header Access-Control-Allow-Origin *;
        }
}

Dendrite YAML file (stripped of comments for legibility)

version: 2

global:
  server_name: mydomain.net

  private_key: matrix_key.pem

  old_private_keys:

  key_validity_period: 168h0m0s

  database:
    connection_string: postgresql://dendrite:REDACTED_SECRET@localhost/dendrite?sslmode=disable
    max_open_conns: 90
    max_idle_conns: 5
    conn_max_lifetime: -1

  cache:
    max_size_estimated: 1gb

    max_age: 1h

  well_known_server_name: ""

  well_known_client_name: ""

  well_known_sliding_sync_proxy: ""

  trusted_third_party_id_servers:
    - matrix.org
    - vector.im

  disable_federation: false

  presence:
    enable_inbound: true
    enable_outbound: true

  report_stats:
    enabled: false
    endpoint: https://panopticon.matrix.org/push

  server_notices:
    enabled: false
    local_part: "_server"
    display_name: "Server Alerts"
    avatar_url: ""
    room_name: "Server Alerts"

  jetstream:
    addresses:

    disable_tls_validation: false

    storage_path: ./

    topic_prefix: Dendrite

  metrics:
    enabled: false
    basic_auth:
      username: metrics
      password: metrics

  dns_cache:
    enabled: false
    cache_size: 256
    cache_lifetime: "5m" # 5 minutes; https://pkg.go.dev/time@master#ParseDuration

app_service_api:
  disable_tls_validation: false

  config_files:

client_api:
  registration_disabled: true

  guests_disabled: true

  registration_shared_secret: "REDACTED_SECRET"

  enable_registration_captcha: false

  recaptcha_public_key: ""
  recaptcha_private_key: ""
  recaptcha_bypass_secret: ""

  turn:
    turn_user_lifetime: "5m"
    turn_uris:
    turn_shared_secret: ""

  rate_limiting:
    enabled: true
    threshold: 20
    cooloff_ms: 500
    exempt_user_ids:

federation_api:
  send_max_retries: 16

  disable_tls_validation: false

  disable_http_keepalives: false

  key_perspectives:
    - server_name: matrix.org
      keys:
        - key_id: ed25519:auto
          public_key: Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw
        - key_id: ed25519:a_RXGa
          public_key: l8Hft5qXKn1vfHrg3p4+W8gELQVo8N13JkluMfmn2sQ

  prefer_direct_fetch: false

media_api:
  base_path: ./media_store

  max_file_size_bytes: 10485760

  dynamic_thumbnails: false

  max_thumbnail_generators: 10

  thumbnail_sizes:
    - width: 32
      height: 32
      method: crop
    - width: 96
      height: 96
      method: crop
    - width: 640
      height: 480
      method: scale

mscs:
  mscs:

sync_api:

  search:
    enabled: true

    index_path: "./searchindex"

    language: "en"

user_api:
  bcrypt_cost: 10

  auto_join_rooms:

tracing:
  enabled: false
  jaeger:
    serviceName: ""
    disabled: false
    rpc_metrics: false
    tags: []
    sampler: null
    reporter: null
    headers: null
    baggage_restrictions: null
    throttler: null

logging:
  - type: std
    level: info
  - type: file
    level: info
    params:
      path: ./logs