element-hq / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://element-hq.github.io/synapse
GNU Affero General Public License v3.0
1.55k stars 190 forks source link

Server-side key backups are not available (Postgres is missing new data in the e2e_room_keys and e2e_room_keys_versions tables) #17895

Open ponfertato opened 1 week ago

ponfertato commented 1 week ago

Description

Good afternoon, I encountered a problem when it is impossible to reserve keys between client and server, and the ability to create a backup copy of encryption keys does not work, but the ability to export and transfer them between devices works. Probably something is broken in my configuration, but I can't figure out what it is.... Having tried different third-party clients (ShildiChat, Cinny, Element Web Dev) the error is the same, but nothing abnormal is found in the logs (p.s. exactly the address /_matrix/client/v3/room_keys/version works, but, for some reason, when trying to access it gives error 404). Probably the problem with the OIDC authorization system is also because of this, because after the problem with keys it stopped allowing to manually reset sessions. I have attached the Synapse server log, no anomalies were encountered in the database logs.

It also constantly swears at the GET method /_matrix/client/unstable/org.matrix.msc2965/auth_issuer, but probably these problems are somehow interrelated...

Steps to reproduce

Homeserver

https://potatoenergy.ru

Synapse Version

{"server_version":"1.118.0"}

Installation Method

Other (Docker Compose)

---
networks:
  prometheus:
    driver: bridge
    external: true
    name: prometheus
  synapse:
    attachable: true
    name: synapse
  traefik:
    driver: bridge
    external: true
    name: traefik
services:
  element:
    container_name: element
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      synapse:
        condition: service_healthy
    image: vectorim/element-web:latest
    labels:
      diun.enable: true
    networks:
      - traefik
    restart: unless-stopped
    volumes:
      - /home/ponfertato/Docker/element/config.json:/app/config.chat.potatoenergy.ru.json
  postgres:
    container_name: synapse-postgres
    env_file:
      - stack.env
    healthcheck:
      interval: 5s
      retries: 5
      test:
        - CMD-SHELL
        - pg_isready -U synapse
      timeout: 5s
    image: postgres:15-alpine
    labels:
      diun.enable: true
    networks:
      - synapse
    restart: unless-stopped
    volumes:
      - database:/var/lib/postgresql/data
  redis:
    command:
      - /bin/sh
      - '-c'
      - |
        redis-server --requirepass "$${REDIS_PASSWORD}"
    container_name: synapse-redis
    env_file:
      - stack.env
    healthcheck:
      interval: 10s
      retries: 5
      start_period: 10s
      test:
        - CMD
        - redis-cli
        - ping
      timeout: 10s
    image: redis:alpine
    labels:
      diun.enable: true
    networks:
      - synapse
    restart: unless-stopped
    volumes:
      - cache:/data
  synapse:
    container_name: synapse
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    env_file:
      - stack.env
    healthcheck:
      interval: 15s
      retries: 3
      start_period: 5s
      test:
        - CMD
        - curl
        - "-fSs"
        - http://localhost:8008/health
      timeout: 5s
    image: ghcr.io/element-hq/synapse:latest
    labels:
      diun.enable: true
    networks:
      - traefik
      - prometheus
      - synapse
    ports:
      - "8008:8008"
    restart: unless-stopped
    volumes:
      - /home/ponfertato/Docker/synapse:/data
      - uploads:/data/uploads
      - media:/data/media_store
volumes:
  cache:
    driver: local
  database:
    driver: local
  media:
    driver: local
  uploads:
    driver: local

Database

PostgreSQL (single, not ported, not restored)

Workers

Single process

Platform

distro: Armbian Bookworm (aarch64) hardware: Rockchip RK3566 OPi 3B (ARM Cortex-A55) container (docker): ghcr.io/element-hq/synapse:latest (sha256:713a1525d4d35892a9ebd82f0ffdbf9c82bfcd01add13b4cdc3d2fcac474d564)

Configuration

Synapse Config:

---
admin_contact: mailto:mail@potatoenergy.ru
admin_users:
  - '@ponfertato:potatoenergy.ru'
  - '@ponfertato:matrix.org'
alias_creation_rules:
  - action: allow
allow_device_name_lookup_over_federation: false
allow_profile_lookup_over_federation: false
allow_public_rooms_over_federation: true
allow_public_rooms_without_auth: true
allowed_avatar_mimetypes: ["image/png", "image/jpeg", "image/gif"]
auto_join_mxid_localpart: system
auto_join_rooms:
  - '#channel:potatoenergy.ru'
  - '#general:potatoenergy.ru'
  - '#local:potatoenergy.ru'
  - '#potatoenergy:matrix.org'
  - '#room:potatoenergy.ru'
autocreate_auto_join_room_preset: trusted_private_chat
autocreate_auto_join_rooms: true
autocreate_auto_join_rooms_federated: false
caches:
  cache_autotuning:
    max_cache_memory_usage: 1024M
    min_cache_ttl: 5m
    target_cache_memory_usage: 758M
  global_factor: 1
  per_cache_factors:
    get_users_who_share_room_with_user: 2
  sync_response_cache_duration: 2m
database:
  args:
    cp_max: 10
    cp_min: 5
    database: synapse
    host: postgres
    password: <EDIT>
    user: synapse
  name: psycopg2
  txn_limit: 10000
# default_identity_server: https://matrix.org
delete_stale_devices_after: 30d
dynamic_thumbnails: true
enable_3pid_lookup: false
enable_authenticated_media: true
enable_media_repo: true
enable_metrics: true
encryption_enabled_by_default_for_room_type: invite
federation:
  client_timeout: 180s
  destination_max_retry_interval: 12h
  destination_min_retry_interval: 1m
  destination_retry_multiplier: 5
  max_long_retries: 20
  max_long_retry_delay: 100s
  max_short_retries: 5
  max_short_retry_delay: 7s
federation_client_minimum_tls_version: '1.2'
forget_rooms_on_leave: true
forgotten_room_retention_period: 1d
form_secret: <EDIT>
gc_thresholds: [700, 10, 10]
ip_range_blacklist:
  - 127.0.0.0/8
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16
  - 100.64.0.0/10
  - 192.0.0.0/24
  - 169.254.0.0/16
  - 192.88.99.0/24
  - 198.18.0.0/15
  - 192.0.2.0/24
  - 198.51.100.0/24
  - 203.0.113.0/24
  - 224.0.0.0/4
  - '::1/128'
  - fe80::/10
  - fc00::/7
  - 2001:db8::/32
  - ff00::/8
  - fec0::/10
limit_remote_rooms:
  admins_can_join: true
  complexity: 0.5
  enabled: true
listeners:
  - bind_addresses: ['::']
    port: 8008
    resources:
      - compress: false
        names: [client, federation, media, static, keys, openid]
    tls: false
    type: http
    x_forwarded: true
  - port: 9000
    type: metrics
log_config: /data/potatoenergy.ru.log.config
macaroon_secret_key: <EDIT>
max_upload_size: 120M
media_retention:
  local_media_lifetime: 30d
  remote_media_lifetime: 7d
media_store_path: /data/media_store
oidc_providers:
  - allow_existing_users: true
    client_id: matrix
    client_secret: <EDIT>
    idp_icon: mxc://authelia.com/cKlrTPsGvlpKxAYeHWJsdVHI
    idp_id: authelia
    idp_name: Authelia
    skip_verification: true
    discover: true
    issuer: https://auth.potatoenergy.ru
    scopes: ["openid", "profile", "email"]
    user_mapping_provider:
      config:
        display_name_template: "{{ user.name }}"
        email_template: "{{ user.email }}"
        localpart_template: "{{ user.preferred_username }}"
        subject_template: "{{ user.sub }}"
password_config:
  enabled: true
  localdb_enabled: true
pid_file: /data/homeserver.pid
presence:
  enabled: untracked
  include_offline_users_on_sync: false
public_baseurl: https://potatoenergy.ru
push:
  jitter_delay: 10s
redis:
  enabled: true
  host: redis
  password: <EDIT>
remote_media_download_burst_count: 200M
remote_media_download_per_second: 40K
report_stats: false
retention:
  allowed_lifetime_max: 1y
  allowed_lifetime_min: 1d
  default_policy:
    max_lifetime: 1y
    min_lifetime: 1d
  enabled: true
  purge_jobs:
    - interval: 12h
      longest_max_lifetime: 3d
    - interval: 1d
      shortest_max_lifetime: 3d
room_list_publication_rules:
  - action: allow
serve_server_wellknown: true
server_name: potatoenergy.ru
signing_key_path: /data/potatoenergy.ru.signing.key
sso:
  client_whitelist:
    - https://auth.potatoenergy.ru/
  update_profile_information: false
suppress_key_server_warning: true
thumbnail_sizes:
  - height: 32
    method: crop
    width: 32
  - height: 96
    method: crop
    width: 96
  - height: 240
    method: scale
    width: 320
trusted_key_servers:
  - server_name: potatoenergy.ru
  - server_name: matrix.org
  - server_name: t2bot.io
turn_allow_guests: false
turn_password: <EDIT>
turn_uris:
  - stun:stun.cloudflare.com:3478
  - turn:turn.cloudflare.com:3478?transport=udp
  - turn:turn.cloudflare.com:3478?transport=tcp
  - turns:turn.cloudflare.com:5349?transport=tcp
turn_user_lifetime: 1h
turn_username: <EDIT>
url_preview_accept_language:
  - ru-RU,ru;q=0.8
  - en-US;q=0.5
  - en;q=0.3
url_preview_enabled: true
url_preview_ip_range_blacklist:
  - 127.0.0.0/8
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16
  - 100.64.0.0/10
  - 192.0.0.0/24
  - 169.254.0.0/16
  - 192.88.99.0/24
  - 198.18.0.0/15
  - 192.0.2.0/24
  - 198.51.100.0/24
  - 203.0.113.0/24
  - 224.0.0.0/4
  - '::1/128'
  - fe80::/10
  - fc00::/7
  - 2001:db8::/32
  - ff00::/8
  - fec0::/10
url_preview_url_blacklist:
  - scheme: http
  - netloc: ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$
user_directory:
  prefer_local_users: true
web_client_location: https://chat.potatoenergy.ru/

Self-hosted Element Web Config:

{
  "default_server_name": "potatoenergy.ru",
  "default_server_config": {
    "m.homeserver": {
      "base_url": "https://potatoenergy.ru"
    }
  },
  "brand": "Potato Energy Team (Element)",
  "branding": {
    "welcome_background_url": "<EDIT>",
    "auth_header_logo_url": "https://potatoenergy.ru/favicon.ico",
    "permalink_prefix": "https://chat.potatoenergy.ru",
    "auth_footer_links": [
      {
        "text": "Home",
        "url": "https://potatoenergy.ru/"
      },
      {
        "text": "Mastodon",
        "url": "https://social.potatoenergy.ru/"
      }
    ]
  },
  "default_theme": "dark",
  "default_federate": false,
  "disable_3pid_login": false,
  "show_labs_settings": true,
  "room_directory": {
    "servers": [
      "potatoenergy.ru",
      "matrix.org"
    ]
  },
  "disable_custom_urls": true,
  "force_verification": true,
  "disable_guests": true,
  "enable_presence_by_hs_url": {
    "https://potatoenergy.ru": true,
    "https://matrix.org": false,
    "https://matrix-client.matrix.org": false
  },
  "terms_and_conditions_links": [
    {
      "url": "https://element.io/privacy",
      "text": "Privacy Policy"
    },
    {
      "url": "https://element.io/cookie-policy",
      "text": "Cookie Policy"
    }
  ],
  "oidc_static_clients": {
    "https://auth.potatoenergy.ru/": {
        "client_id": "matrix"
    }
  },
  "features": {
    "feature_oidc_native_flow": true,
    "feature_element_call_video_rooms": true,
    "feature_group_calls": true,
    "feature_latex_maths": true,
    "feature_pinning": true,
    "feature_html_topic": true,
    "feature_video_rooms": true,
    "threadsActivityCentre": true
  },
  "setting_defaults": {
    "RustCrypto.staged_rollout_percent": 100,
    "UIFeature.deactivate": false,
    "UIFeature.identityServer": false,
    "UIFeature.passwordReset": false,
    "UIFeature.registration": true,
    "UIFeature.thirdPartyId": false,
    "UIFeature.urlPreviews": true,
    "Registration.mobileRegistrationHelper": true
  },
  "login_for_welcome": false,
  "sso_redirect_options": {
    "immediate": false,
    "on_welcome_page": false,
    "on_login_page": false
  },
  "disable_login_language_selector": false,
  "default_widget_container_height": 280,
  "default_country_code": "RU",
  "jitsi": {
      "preferred_domain": "meet.element.io"
  },
  "element_call": {
      "url": "https://call.element.io",
      "participant_limit": 8,
      "brand": "Element Call"
  },
  "integrations_ui_url": "https://scalar.vector.im/",
  "integrations_rest_url": "https://scalar.vector.im/api",
  "integrations_widgets_urls": [
      "https://scalar.vector.im/_matrix/integrations/v1",
      "https://scalar.vector.im/api",
      "https://scalar-staging.vector.im/_matrix/integrations/v1",
      "https://scalar-staging.vector.im/api",
      "https://scalar-staging.riot.im/scalar/api"
  ],
  "uisi_autorageshake_app": "element-auto-uisi"
}

Traefik Config:

---
http:
  middlewares:
...
    securityHeaders:
      headers:
        browserXssFilter: true
        customResponseHeaders:
          Referrer-Policy: same-origin
          Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
          X-Content-Type-Options: nosniff
          X-Frame-Options: SAMEORIGIN
          X-XSS-Protection: 1; mode=block
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
        hostsProxyHeaders:
          - Host
          - X-Real-IP
          - X-Forwarded-Host
          - X-Forwarded-For
          - X-Forwarded-Proto
        sslProxyHeaders: 
          X-Forwarded-Proto: https
        sslRedirect: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 63072000
    corsHeaders:
      headers:
        accessControlAllowCredentials: true
        accessControlAllowOriginListRegex:
          - ^(.*\.)?potatoenergy\.ru$
        accessControlAllowHeaders:
          - Accept
          - Authorization
          - Client-Security-Token
          - Content-Type
          - Origin
          - X-Requested-With
        accessControlAllowMethods:
          - DELETE
          - GET
          - OPTIONS
          - POST
          - PUT
        accessControlMaxAge: 100
        addVaryHeader: true
    error-pages:
      errors:
        query: /{status}.html
        service: error-pages
        status:
          - 400-599
...
---
http:
  routers:
...
    element:
      entryPoints:
        - websecure
      middlewares:
        - corsHeaders
        - error-pages
        - securityHeaders
      rule: Host(`chat.potatoenergy.ru`)
      service: element
      tls:
        certResolver: letsencrypt
        options: default
    synapse:
      entryPoints:
        - websecure
      middlewares:
        - corsHeaders
        - error-pages
        - securityHeaders
      rule: Host(`potatoenergy.ru`) && (PathPrefix(`/_matrix/`) || PathPrefix(`/_synapse/client/`) || PathPrefix(`/_synapse/admin/`) || PathPrefix(`/.well-known/matrix/`))
      service: synapse
      tls:
        certResolver: letsencrypt
        options: default

Relevant log output

Starting synapse with args -m synapse.app.homeserver --config-path /data/homeserver.yaml
2024-11-02 09:14:24,466 - root - 352 - WARNING - main - ***** STARTING SERVER *****
2024-11-02 09:14:24,467 - root - 353 - WARNING - main - Server /usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py version 1.118.0
2024-11-02 09:14:24,467 - root - 358 - WARNING - main - Copyright (c) 2023 New Vector, Inc
2024-11-02 09:14:24,467 - root - 359 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
2024-11-02 09:14:24,468 - root - 362 - INFO - main - Server hostname: potatoenergy.ru
2024-11-02 09:14:24,469 - root - 363 - INFO - main - Instance name: master
2024-11-02 09:14:24,470 - root - 364 - INFO - main - Twisted reactor: EPollReactor
2024-11-02 09:14:24,470 - synapse.app.homeserver - 370 - INFO - main - Setting up server
2024-11-02 09:14:24,471 - synapse.server - 355 - INFO - main - Setting up.
2024-11-02 09:14:24,529 - synapse.storage.databases - 73 - INFO - main - [database config 'master']: Checking database server
2024-11-02 09:14:24,540 - synapse.storage.databases - 76 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state']
2024-11-02 09:14:24,540 - synapse.storage.prepare_database - 135 - INFO - main - ['main', 'state']: Checking existing schema version
2024-11-02 09:14:24,557 - synapse.storage.prepare_database - 139 - INFO - main - ['main', 'state']: Existing schema is 88 (+2 deltas)
2024-11-02 09:14:24,558 - synapse.storage.databases.main - 395 - INFO - main - Checking database for consistency with configuration...
2024-11-02 09:14:24,563 - synapse.storage.prepare_database - 433 - INFO - main - Applying schema deltas for v88
2024-11-02 09:14:24,569 - synapse.storage.prepare_database - 568 - INFO - main - Schema now up to date
2024-11-02 09:14:24,575 - synapse.storage.databases - 91 - INFO - main - [database config 'master']: Starting 'main' database
2024-11-02 09:14:24,847 - synapse.storage.util.id_generators - 98 - INFO - main - Initialising stream generator for push_rules(id): 5
2024-11-02 09:14:24,852 - synapse.storage.util.id_generators - 98 - INFO - main - Initialising stream generator for push_rules_enable(id): 5
2024-11-02 09:14:24,907 - synapse.storage.databases.main.event_push_actions - 1242 - INFO - main - Searching for stream ordering 1 month ago
2024-11-02 09:14:24,954 - synapse.storage.databases.main.event_push_actions - 1246 - INFO - main - Found stream ordering 1 month ago: it's 546
2024-11-02 09:14:24,955 - synapse.storage.databases.main.event_push_actions - 1249 - INFO - main - Searching for stream ordering 1 day ago
2024-11-02 09:14:24,991 - synapse.storage.databases.main.event_push_actions - 1253 - INFO - main - Found stream ordering 1 day ago: it's 4329
2024-11-02 09:14:25,045 - synapse.storage.util.id_generators - 98 - INFO - main - Initialising stream generator for access_tokens(id): 36
2024-11-02 09:14:25,054 - synapse.storage.util.id_generators - 98 - INFO - main - Initialising stream generator for refresh_tokens(id): 1
2024-11-02 09:14:25,078 - synapse.storage.util.id_generators - 98 - INFO - main - Initialising stream generator for event_reports(id): 2
2024-11-02 09:14:25,083 - synapse.storage.util.id_generators - 98 - INFO - main - Initialising stream generator for room_reports(id): 1
2024-11-02 09:14:25,253 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database
2024-11-02 09:14:25,261 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared
2024-11-02 09:14:25,262 - synapse.server - 358 - INFO - main - Finished setting up.
2024-11-02 09:14:25,371 - synapse.push.pusher - 45 - INFO - main - email enable notifs: False
2024-11-02 09:14:25,399 - synapse.server - 895 - INFO - main - Connecting to redis (host='redis' port=6379) for external cache
2024-11-02 09:14:25,401 - synapse.replication.tcp.redis - 292 - INFO - main - Connecting to redis server redis:6379
2024-11-02 09:14:25,413 - synapse.federation.federation_server - 1397 - INFO - main - Registering federation EDU handler for 'm.device_list_update'
2024-11-02 09:14:25,414 - synapse.federation.federation_server - 1417 - INFO - main - Registering federation query handler for 'profile'
2024-11-02 09:14:25,422 - synapse.federation.federation_server - 1397 - INFO - main - Registering federation EDU handler for 'm.presence'
2024-11-02 09:14:25,424 - synapse.federation.federation_server - 1397 - INFO - main - Registering federation EDU handler for 'm.typing'
2024-11-02 09:14:25,429 - synapse.federation.federation_server - 1417 - INFO - main - Registering federation query handler for 'directory'
2024-11-02 09:14:25,433 - synapse.handlers.pagination - 116 - INFO - main - Setting up purge job with config: RetentionPurgeJob(interval=43200000, shortest_max_lifetime=None, longest_max_lifetime=259200000)
2024-11-02 09:14:25,435 - synapse.handlers.pagination - 116 - INFO - main - Setting up purge job with config: RetentionPurgeJob(interval=86400000, shortest_max_lifetime=259200000, longest_max_lifetime=None)
2024-11-02 09:14:25,437 - twisted - 279 - INFO - main - Redirected stdout/stderr to logs
2024-11-02 09:14:25,438 - synapse.app.homeserver - 180 - INFO - sentinel - Running
2024-11-02 09:14:25,449 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576
2024-11-02 09:14:25,479 - synapse.handlers.deactivate_account - 245 - INFO - user_parter_loop-0 - Starting user parter
2024-11-02 09:14:26,240 - synapse.handlers.deactivate_account - 255 - INFO - user_parter_loop-0 - User parter finished: stopping
2024-11-02 09:14:26,421 - synapse.http.client - 428 - INFO - sentinel - Received response to GET https://auth.potatoenergy.ru/.well-known/openid-configuration: 200
2024-11-02 09:14:26,452 - synapse.http.client - 428 - INFO - sentinel - Received response to GET https://auth.potatoenergy.ru/jwks.json: 200
2024-11-02 09:14:26,533 - synapse.util.caches.lrucache - 231 - INFO - sentinel - Expiring LRU caches after 1800 seconds
2024-11-02 09:14:26,558 - synapse.replication.tcp.redis - 292 - INFO - sentinel - Connecting to redis server redis:6379
2024-11-02 09:14:26,645 - synapse.federation.federation_server - 1397 - INFO - sentinel - Registering federation EDU handler for 'm.receipt'
2024-11-02 09:14:26,646 - synapse.federation.federation_server - 1397 - INFO - sentinel - Registering federation EDU handler for 'm.signing_key_update'
2024-11-02 09:14:26,647 - synapse.federation.federation_server - 1397 - INFO - sentinel - Registering federation EDU handler for 'org.matrix.signing_key_update'
2024-11-02 09:14:26,648 - synapse.federation.federation_server - 1417 - INFO - sentinel - Registering federation query handler for 'client_keys'
2024-11-02 09:14:26,652 - synapse.federation.federation_server - 1397 - INFO - sentinel - Registering federation EDU handler for 'm.direct_to_device'
2024-11-02 09:14:26,736 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.health.HealthResource object at 0xffffa73c71d0> to path b'/health'
2024-11-02 09:14:26,736 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.ClientRestResource object at 0xffffa73c7210> to path b'/_matrix/client'
2024-11-02 09:14:26,737 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <twisted.web.resource.Resource object at 0xffffa73ff790> to path b'/.well-known'
2024-11-02 09:14:26,737 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.admin.AdminRestResource object at 0xffffa80b9510> to path b'/_synapse/admin'
2024-11-02 09:14:26,738 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.synapse.client.pick_idp.PickIdpResource object at 0xffffa6510090> to path b'/_synapse/client/pick_idp'
2024-11-02 09:14:26,739 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <twisted.web.resource.Resource object at 0xffffa73d4890> to path b'/_synapse/client/pick_username'
2024-11-02 09:14:26,739 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.synapse.client.new_user_consent.NewUserConsentResource object at 0xffffa650ce10> to path b'/_synapse/client/new_user_consent'
2024-11-02 09:14:26,740 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.synapse.client.sso_register.SsoRegisterResource object at 0xffffa6510790> to path b'/_synapse/client/sso_register'
2024-11-02 09:14:26,740 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.synapse.client.unsubscribe.UnsubscribeResource object at 0xffffa73fcd90> to path b'/_synapse/client/unsubscribe'
2024-11-02 09:14:26,740 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.synapse.client.oidc.OIDCResource object at 0xffffa73f6e90> to path b'/_synapse/client/oidc'
2024-11-02 09:14:26,741 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching FilePath('/usr/local/lib/python3.11/site-packages/synapse/static') to path b'/_matrix/static'
2024-11-02 09:14:26,742 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xffffa73d4bd0> to path b'/_matrix/media/r0'
2024-11-02 09:14:26,743 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xffffa73d4bd0> to path b'/_matrix/media/v3'
2024-11-02 09:14:26,744 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xffffa73d4bd0> to path b'/_matrix/media/v1'
2024-11-02 09:14:26,744 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.federation.transport.server.TransportLayerServer object at 0xffffa6511590> to path b'/_matrix/federation'
2024-11-02 09:14:26,747 - synapse.util.httpresourcetree - 56 - INFO - sentinel - Attaching <synapse.rest.key.v2.KeyResource object at 0xffffa80b3d90> to path b'/_matrix/key'
2024-11-02 09:14:26,750 - twisted - 279 - INFO - sentinel - SynapseSite starting on 8008
2024-11-02 09:14:26,753 - synapse.app._base - 439 - INFO - sentinel - Synapse now listening on TCP port 8008
2024-11-02 09:14:26,754 - synapse.app._base - 295 - INFO - sentinel - Starting metrics listener on 0.0.0.0:9000
2024-11-02 09:14:27,255 - synapse.storage.background_updates - 410 - INFO - background_updates-0 - Starting background schema updates for database master
...
2024-11-02 09:14:27,314 - synapse.replication.tcp.redis - 126 - INFO - sentinel - Connected to redis
2024-11-02 09:14:27,318 - synapse.replication.tcp.redis - 138 - INFO - subscribe-replication-0 - Sending redis SUBSCRIBE for ['potatoenergy.ru/USER_IP', 'potatoenergy.ru']
2024-11-02 09:14:27,320 - synapse.storage.background_updates - 428 - INFO - background_updates-0 - No more background updates to do. Unscheduling background update task.
2024-11-02 09:14:27,370 - synapse.replication.tcp.redis - 141 - INFO - subscribe-replication-0 - Successfully subscribed to redis stream, sending REPLICATE command
2024-11-02 09:14:27,373 - synapse.push.pusherpool - 372 - INFO - start_pushers-0 - Started pushers
2024-11-02 09:14:27,376 - synapse.replication.tcp.redis - 146 - INFO - subscribe-replication-0 - REPLICATE successfully sent
2024-11-02 09:14:34,531 - synapse.access.http.8008 - 473 - INFO - GET-1 - <EDIT> - 8008 - {None} Processed request: 0.001sec/0.002sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 1062B 200 "GET /_matrix/client/versions HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
...
2024-11-02 09:14:44,000 - synapse.access.http.8008 - 473 - INFO - GET-7 - <EDIT> - 8008 - {None} Processed request: 0.002sec/0.005sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 56B 200 "GET /.well-known/matrix/client HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
2024-11-02 09:14:44,160 - synapse.access.http.8008 - 473 - INFO - GET-8 - <EDIT> - 8008 - {None} Processed request: 0.005sec/0.016sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 1062B 200 "GET /_matrix/client/versions HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
2024-11-02 09:14:44,264 - synapse.http.server - 130 - INFO - GET-9 - <XForwardedForRequest at 0xffffa658ff50 method='GET' uri='/_matrix/client/unstable/org.matrix.msc2965/auth_issuer' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Unrecognized request
2024-11-02 09:14:44,281 - synapse.access.http.8008 - 473 - INFO - GET-9 - <EDIT> - 8008 - {None} Processed request: 0.007sec/0.011sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 59B 404 "GET /_matrix/client/unstable/org.matrix.msc2965/auth_issuer HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
2024-11-02 09:14:44,925 - synapse.access.http.8008 - 473 - INFO - GET-10 - <EDIT> - 8008 - {@ponfertato:potatoenergy.ru} Processed request: 0.006sec/0.002sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "GET /_matrix/client/v3/thirdparty/protocols HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
2024-11-02 09:14:45,056 - synapse.access.http.8008 - 473 - INFO - GET-11 - <EDIT> - 8008 - {None} Processed request: 0.090sec/0.003sec (0.020sec, 0.003sec) (0.011sec/0.047sec/4) 79B 200 "GET /_matrix/client/v3/directory/room/<EDIT>potatoenergy.ru HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
2024-11-02 09:14:46,152 - synapse.http.server - 130 - INFO - GET-12 - <XForwardedForRequest at 0xffffa652b550 method='GET' uri='/_matrix/client/v3/room_keys/version' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - No backup found
2024-11-02 09:14:46,159 - synapse.access.http.8008 - 473 - INFO - GET-12 - <EDIT> - 8008 - {@ponfertato:potatoenergy.ru} Processed request: 0.025sec/0.005sec (0.005sec, 0.002sec) (0.001sec/0.010sec/1) 51B 404 "GET /_matrix/client/v3/room_keys/version HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
2024-11-02 09:14:46,438 - synapse.access.http.8008 - 473 - INFO - GET-14 - <EDIT> - 8008 - {@ponfertato:potatoenergy.ru} Processed request: 0.012sec/0.009sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 316B 200 "GET /_matrix/client/v3/voip/turnServer HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
...
2024-11-02 09:14:46,906 - synapse.http.server - 130 - INFO - GET-19 - <XForwardedForRequest at 0xffffa65a3950 method='GET' uri='/_matrix/client/v3/room_keys/version' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - No backup found
2024-11-02 09:14:46,911 - synapse.access.http.8008 - 473 - INFO - GET-19 - <EDIT> - 8008 - {@ponfertato:potatoenergy.ru} Processed request: 0.017sec/0.002sec (0.005sec, 0.002sec) (0.002sec/0.006sec/1) 51B 404 "GET /_matrix/client/v3/room_keys/version HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
...

Anything else that would be useful to know?

Screenshots

изображение

изображение

изображение

изображение

{DAD1E417-A052-4D63-AABF-2FD869629E60}

ponfertato commented 22 hours ago

More info

Postgres:

synapse=# \d e2e_room_keys
                  Table "public.e2e_room_keys"
       Column        |  Type   | Collation | Nullable | Default 
---------------------+---------+-----------+----------+---------
 user_id             | text    |           | not null | 
 room_id             | text    |           | not null | 
 session_id          | text    |           | not null | 
 version             | bigint  |           | not null | 
 first_message_index | integer |           |          | 
 forwarded_count     | integer |           |          | 
 is_verified         | boolean |           |          | 
 session_data        | text    |           | not null | 
Indexes:
    "e2e_room_keys_room_id" btree (room_id)
    "e2e_room_keys_with_version_idx" UNIQUE, btree (user_id, version, room_id, session_id)

synapse=# \d e2e_room_keys_versions
         Table "public.e2e_room_keys_versions"
  Column   |   Type   | Collation | Nullable | Default 
-----------+----------+-----------+----------+---------
 user_id   | text     |           | not null | 
 version   | bigint   |           | not null | 
 algorithm | text     |           | not null | 
 auth_data | text     |           | not null | 
 deleted   | smallint |           | not null | 0
 etag      | bigint   |           |          | 
Indexes:
    "e2e_room_keys_versions_idx" UNIQUE, btree (user_id, version)
ERROR:  relation "room_keys" does not exist at character 15
STATEMENT:  select * from room_keys;
ERROR:  relation "key_backups" does not exist at character 15
STATEMENT:  select * from key_backups;

Example e2e_room_keys_versions

synapse=# select * from e2e_room_keys_versions;
            user_id             | version |               algorithm                |                                                                                                                                                                                                                                                                                                auth_data                                                                                                                                                                                                                                                                                                | deleted | etag 
--------------------------------+---------+----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+------
 @<USER_ONE>:potatoenergy.ru |       2 | m.megolm_backup.v1.curve25519-aes-sha2 | {"public_key":"<EDIT>","signatures":{"@<USER_ONE>:potatoenergy.ru":{"ed25519:KDETVOBTKA":"<EDIT>","ed25519:FDUCWWGAKB":"<EDIT>","ed25519:VcScmR5edfhYW8bgVK6RNoBVsmWkeyVRaN87O0fmhxI":"<EDIT>","ed25519:MRASKJNEOU":"<EDIT>"}}} |       0 |   59
 @ponfertato:potatoenergy.ru    |       1 | m.megolm_backup.v1.curve25519-aes-sha2 | {"public_key":"<EDIT>","signatures":{"@ponfertato:potatoenergy.ru":{"ed25519:KWMORYTWEU":"<EDIT>"}}}                                                                                                                                                                                                                                                                                                                                                                               |       1 |    6
 @<USER_TWO>:potatoenergy.ru    |       1 | m.megolm_backup.v1.curve25519-aes-sha2 | {"public_key":"<EDIT>","signatures":{"@<USER_TWO>:potatoenergy.ru":{"ed25519:JGJLOTMEKY":"<EDIT>","ed25519:Yz8bti9B/unDrHbjUbwn31NIHyUhJGO/qPziWRqhMlw":"<EDIT>"}}}                                                                                                                                                                                                                                |       0 |    3
 @<USER_ONE>:potatoenergy.ru |       1 | m.megolm_backup.v1.curve25519-aes-sha2 | {"public_key":"<EDIT>","signatures":{"@<USER_ONE>:potatoenergy.ru":{"ed25519:KDETVOBTKA":"<EDIT>"}}}                                                                                                                                                                                                                                                                                                                                                                            |       1 |    6
(4 rows)

Synapse (with debug database):

2024-11-13 09:26:43,595 - synapse.http.server - 130 - INFO - GET-213 - <XForwardedForRequest at 0xffff913f7050 method='GET' uri='/_matrix/client/v3/room_keys/version' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - No backup found
2024-11-13 09:26:43,599 - synapse.access.http.8008 - 473 - INFO - GET-213 - <EDIT> - 8008 - {@ponfertato:potatoenergy.ru} Processed request: 0.013sec/0.002sec (0.001sec, 0.002sec) (0.001sec/0.004sec/1) 67B 404 "GET /_matrix/client/v3/room_keys/version HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0" [0 dbevts]
2024-11-13 09:26:43,764 - synapse.storage.SQL - 468 - DEBUG - GET-214 - [SQL] {get_e2e_room_keys_version_info-2b5} SELECT MAX(version) FROM e2e_room_keys_versions WHERE user_id=? AND deleted=0
2024-11-13 09:26:43,769 - synapse.storage.SQL - 468 - DEBUG - prune_old_user_ips-68 - [SQL] {_prune_old_user_ips-2b6} DELETE FROM user_ips WHERE last_seen IN ( SELECT last_seen FROM user_ips WHERE last_seen <= ? ORDER BY last_seen ASC LIMIT 5000 )
2024-11-13 09:26:43,775 - synapse.storage.SQL - 473 - DEBUG - GET-214 - [SQL values] {get_e2e_room_keys_version_info-2b5} ('@ponfertato:potatoenergy.ru',)
2024-11-13 09:26:43,776 - synapse.storage.SQL - 473 - DEBUG - prune_old_user_ips-68 - [SQL values] {_prune_old_user_ips-2b6} (1729070803759,)
2024-11-13 09:26:43,787 - synapse.storage.SQL - 494 - DEBUG - prune_old_user_ips-68 - [SQL time] {_prune_old_user_ips-2b6} 0.007188 sec
2024-11-13 09:26:43,799 - synapse.storage.SQL - 494 - DEBUG - GET-214 - [SQL time] {get_e2e_room_keys_version_info-2b5} 0.022402 sec

After looking at the tables, I did not find that it writes new users there at one point (apparently when configuring the configuration on a live server) I broke something, which is why indexing stopped working.

It seems like what I changed (and it was a test to disable the built-in login system, all sorts of encryption settings in the rooms, a test to connect workers to differentiate tasks) should not have affected this in any way, however, I'm not sure what can be done about it except to recreate the database. But I would not like to lose the rooms that have already been created.

ponfertato commented 12 hours ago

In general, I went to desperate measures and decided to clean these tables with the following commands:

TRUNCATE TABLE e2e_room_keys_versions;
TRUNCATE TABLE e2e_room_keys;

Then on startup Synapse asked to correct some stream_ positions, I cleaned them up with the following commands:

DELETE FROM stream_positions WHERE stream_name = 'receipts';
DELETE FROM stream_positions WHERE stream_name = 'events';
DELETE FROM stream_positions WHERE stream_name = 'to_device';
DELETE FROM stream_positions WHERE stream_name = 'account_data';
DELETE FROM stream_positions WHERE stream_name = 'presence_stream';

After which the server started up and... As soon as I added the resource again:

...
listeners:
  - bind_addresses: ['0.0.0.0'] # I've changed the configuration a little bit here
    port: 8008
    resources: # and here
      - compress: false
        names: [openid, keys]
      - compress: true
        names: [client, federation]
    tls: false
    type: http
    x_forwarded: true
...

Deleting accounts via OIDC finally worked (using a token, but only from the web version for some reason), but the keys are still not written (I logged out of all my sessions, logged in again from Element Desktop, but when I tried to reset the keys, the situation repeated itself) I'm not sure what else the problem could be related to.....