search

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://matrix-org.github.io/dendrite/
Apache License 2.0
5.67k stars 664 forks source link

Server is tremendously slow #1818

Closed viasux closed 3 years ago

viasux commented 3 years ago

Background information

Description

Steps to reproduce

free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       519Mi       2.4Gi       147Mi       1.0Gi       3.0Gi
Swap:          4.4Gi          0B       4.4Gi

As you can see, not many resources are being used. (cpu only peaks at 40%), but there's a tremendous delay to messages sent.

top - 22:50:34 up  7:54,  1 user,  load average: 0.29, 0.38, 0.47
Tasks: 127 total,   1 running, 126 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.3 sy,  0.0 ni, 98.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3929.5 total,   2482.8 free,    370.2 used,   1076.5 buff/cache
MiB Swap:   4512.0 total,   4512.0 free,      0.0 used.   3191.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    279 dendrite  20   0 3199080 266896  21196 S   1.0   6.6   9:46.42 dendrite
    199 root      20   0  386920 262216 260652 S   0.3   6.5   0:19.82 systemd-journal
  21952 postgres  20   0  208708  17992  15496 S   0.3   0.4   0:00.05 postgres
      1 root      20   0   23824  10260   7928 S   0.0   0.3   0:01.86 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd

https://user-images.githubusercontent.com/47131867/113816775-cd310e00-9764-11eb-9b14-8ac619b87c49.mp4

kegsay commented 3 years ago

What is logged when it crashes?

viasux commented 3 years ago

What is logged when it crashes?

I sent logs to one of the people listed

viasux commented 3 years ago

What is logged when it crashes?

Left the server up for abt a day and didn't touch it. It went from being completely unusable to slightly responsive, and now it seems mostly functional. I have no clue what caused this, but it seems to be functioning better now. Should I close this thread? I believe some sort of bug may have caused this, although I'm unsure.

viasux commented 3 years ago

I tried changing my profile picture and it caused the entire server to break again. nginx is spamming no live upstreams while connecting to upstream

viasux commented 3 years ago

I tried changing my profile picture and it caused the entire server to break again. nginx is spamming no live upstreams while connecting to upstream

seems to have been a result of changing my profille picture, restarting fixed.

viasux commented 3 years ago

Randomly, my server has deleted all messages since Sunday from my history in my group chat.

My server: image Alt matrix acc: image

kegsay commented 3 years ago

There doesn't seem to be anything actionable here. It may be a dupe of https://github.com/matrix-org/dendrite/issues/1676 but it's hard to tell.

Randomly, my server has deleted all messages since Sunday

The server won't delete anything - there's physically no code to do this. Redacted messages will redact the event content (but the event itself has to stay for DAG purposes).