Server is tremendously slow

[matrixbot]

This issue was originally created by @viasux at https://github.com/matrix-org/dendrite/issues/1818.

Background information

• Dendrite version or git SHA:
• 0.3.11-1
• Monolith or Polylith?:
• Monolith
• SQLite3 or Postgres?:
• Postgres
• Running in Docker?:
• go version:
• 2:1.16.3-1

Description

Steps to reproduce

• Start the server
• Send some messages
• Observe crashing and tremendous latency, while not many resources are being used.

free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       519Mi       2.4Gi       147Mi       1.0Gi       3.0Gi
Swap:          4.4Gi          0B       4.4Gi

As you can see, not many resources are being used. (cpu only peaks at 40%), but there's a tremendous delay to messages sent.

top - 22:50:34 up  7:54,  1 user,  load average: 0.29, 0.38, 0.47
Tasks: 127 total,   1 running, 126 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.3 sy,  0.0 ni, 98.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3929.5 total,   2482.8 free,    370.2 used,   1076.5 buff/cache
MiB Swap:   4512.0 total,   4512.0 free,      0.0 used.   3191.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    279 dendrite  20   0 3199080 266896  21196 S   1.0   6.6   9:46.42 dendrite
    199 root      20   0  386920 262216 260652 S   0.3   6.5   0:19.82 systemd-journal
  21952 postgres  20   0  208708  17992  15496 S   0.3   0.4   0:00.05 postgres
      1 root      20   0   23824  10260   7928 S   0.0   0.3   0:01.86 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd

https://user-images.githubusercontent.com/47131867/113816775-cd310e00-9764-11eb-9b14-8ac619b87c49.mp4

[matrixbot]

This comment was originally posted by @kegsay at https://github.com/matrix-org/dendrite/issues/1818#issuecomment-814781420.

What is logged when it crashes?

[matrixbot]

This comment was originally posted by @viasux at https://github.com/matrix-org/dendrite/issues/1818#issuecomment-814874290.

What is logged when it crashes?

I sent logs to one of the people listed

[matrixbot]

This comment was originally posted by @viasux at https://github.com/matrix-org/dendrite/issues/1818#issuecomment-815123526.

What is logged when it crashes?

Left the server up for abt a day and didn't touch it. It went from being completely unusable to slightly responsive, and now it seems mostly functional. I have no clue what caused this, but it seems to be functioning better now. Should I close this thread? I believe some sort of bug may have caused this, although I'm unsure.

[matrixbot]

This comment was originally posted by @viasux at https://github.com/matrix-org/dendrite/issues/1818#issuecomment-815293924.

I tried changing my profile picture and it caused the entire server to break again. nginx is spamming no live upstreams while connecting to upstream

[matrixbot]

This comment was originally posted by @viasux at https://github.com/matrix-org/dendrite/issues/1818#issuecomment-815300586.

I tried changing my profile picture and it caused the entire server to break again. nginx is spamming no live upstreams while connecting to upstream

seems to have been a result of changing my profille picture, restarting fixed.

[matrixbot]

This comment was originally posted by @viasux at https://github.com/matrix-org/dendrite/issues/1818#issuecomment-815315175.

Randomly, my server has deleted all messages since Sunday from my history in my group chat.

My server: Alt matrix acc:

[matrixbot]

This comment was originally posted by @kegsay at https://github.com/matrix-org/dendrite/issues/1818#issuecomment-815648385.

There doesn't seem to be anything actionable here. It may be a dupe of https://github.com/matrix-org/dendrite/issues/1676 but it's hard to tell.

Randomly, my server has deleted all messages since Sunday

The server won't delete anything - there's physically no code to do this. Redacted messages will redact the event content (but the event itself has to stay for DAG purposes).