user api stops working

[matrixbot]

This issue was originally created by @Joshix-1 at https://github.com/matrix-org/dendrite/issues/2105.

Background information

• Dendrite version or git SHA: 0.5.1
• Monolith or Polylith?: Monolith
• SQLite3 or Postgres?: Postgres
• Running in Docker?: no
• go version: 1.16.12
• Client used (if applicable): - - -

Description

• What is the problem: The user api stops working
• Who is affected: The user api, the federation api still seems to be working
• How is this bug manifesting: The user api doesn't respond anymore/ times out
• When did this first appear: I don't know; A while after starting the instance

Steps to reproduce

It just happens all the time, every few minutes. Sometimes after half an hour.

[matrixbot]

This comment was originally posted by @kegsay at https://github.com/matrix-org/dendrite/issues/2105#issuecomment-1020302268.

How do you know it is the userapi if you're running in Monolith mode?

[matrixbot]

This comment was originally posted by @Joshix-1 at https://github.com/matrix-org/dendrite/issues/2105#issuecomment-1023545793.

We have a script running that checks periodically if the get public rooms endpoint still responds. And after a while it stops responding or times out. The script we use:

#!/usr/bin/env python3

import xmlrpc.client
import time

import urllib3

urllib3.disable_warnings()

HOMESERVER = "localhost:8448"
SUPERVISOR = "localhost:9001"
XMLRPC_USER = "ReoNa"
XMLRPC_PASS = "forget-me-not"

if __name__ == "__main__":
    counter = 0
    running = False
    server = xmlrpc.client.ServerProxy(f"http://{XMLRPC_USER}:{XMLRPC_PASS}@{SUPERVISOR}/RPC2")
    http = urllib3.PoolManager(timeout=5.0, cert_reqs = "CERT_NONE")
    while True:
        try:
            check_start_time = time.monotonic()
            http.request("GET", f"https://{HOMESERVER}/_matrix/client/r0/publicRooms?limit=10")
            counter = 0
            running = True
            time.sleep(10)
        except urllib3.exceptions.MaxRetryError:
            if not running:
                if time.monotonic() - check_start_time >= 20:
                    counter += 1
                else:
                    time.sleep(5)
                    counter = 0
                if counter < 6:
                    continue
            counter = 0
            running = False
            process_info = server.supervisor.getProcessInfo("dendrite")
            if process_info["statename"] != "RUNNING": continue
            uptime = process_info["now"] - process_info["start"]
            server.supervisor.stopProcess("dendrite")
            server.supervisor.startProcess("dendrite")