High CPU constantly (OpenBSD vmm machine)

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!

https://matrix-org.github.io/dendrite/

Apache License 2.0

5.51k stars 655 forks source link

High CPU constantly (OpenBSD vmm machine) #3359

Open gonzalo- opened 3 months ago

gonzalo- commented 3 months ago

Background information

Dendrite version or git SHA: 0.13.5
SQLite3 or Postgres?: postgresql-server-16.2
Running in Docker?:
go version: go-1.22.1
Client used (if applicable): Element

Description

What is the problem: 90% of the CPU usage constantly.
Who is affected: Since 0.13.5 I can see this problem
How is this bug manifesting: Just right after start the service.
When did this first appear: Never, well if I stop it :)

Steps to reproduce

I just upgrade the package, the previous version has no issue.

I got some logs from PPROF here

Disabled presence{enable_inbound,enable_outbound} doesn't fix the issue.

S7evinK commented 3 months ago

The provided pprof log is a bit difficult to read. Can you get a CPU profile using wget -O 'profile.pprof' 'http://localhost:65432/debug/pprof/profile?seconds=10' (change port accordingly)?

Also, if you can, try updating to 0.13.7.

gonzalo- commented 3 months ago

@S7evinK thanks for replied:

Profile file here

I will try to update to the latest.

S7evinK commented 3 months ago

From what I can tell so far, the majority is spent in runtime/sys_openbsd3.go, which seems to be called from bleve (the fulltextsearch engine we use) and the underlying database bbolt

gonzalo- commented 3 months ago

@S7evinK any recommendation or possible fix?

gonzalo- commented 3 months ago

@S7evinK I can confirm that with 0.13.7 still the same behaviour.

S7evinK commented 2 months ago

Assuming you have fulltext search enabled, can you try disabling it? i.e. by setting sync_api.search.enabled to false in your Dendrite config.

gonzalo- commented 2 months ago

@S7evinK yes, that drops the CPU usage to 2%