High CPU consumption due to frequently Garbage Collect (GC)

[Syize]

Description

Hi,

Recently I've been troubled by the issue of high CPU consumption of Gitea. The CPU consumption often become very high after running about 10 hours, and restart gitea can temporally solve this problem.

The most suspicious behavior of gitea during high CPU consumption is frequent GC. I have taken a screen record when cpu consumption becomes high.

https://github.com/user-attachments/assets/da2e4e8a-bf8c-468d-8a60-8f7b8a6dfaa4

After restarting Gitea, GC seems "calm down".

Here is the log of gitea (Though I can't find any related information.)

2024/09/02 05:06:14 ...s/setting/session.go:77:loadSessionFrom() [I] Session Service Enabled
2024/09/02 05:06:14 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/attachments
2024/09/02 05:06:14 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/avatars
2024/09/02 05:06:14 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/repo-avatars
2024/09/02 05:06:14 ...s/storage/storage.go:186:initLFS() [I] Initialising LFS storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/lfs
2024/09/02 05:06:14 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/repo-archive
2024/09/02 05:06:14 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/packages
2024/09/02 05:06:14 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/actions_log
2024/09/02 05:06:14 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/actions_artifacts
2024/09/02 05:06:14 routers/init.go:138:InitWebInstalled() [I] SQLite3 support is enabled
2024/09/02 05:06:14 routers/common/db.go:23:InitDBEngine() [I] Beginning ORM engine initialization.
2024/09/02 05:06:14 routers/common/db.go:30:InitDBEngine() [I] ORM engine initialization attempt #1/10...
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [I] PING DATABASE mysql
2024/09/02 05:06:14 ...dels/db/collation.go:184:preprocessDatabaseCollation() [W] Current database is using a case-insensitive collation "utf8mb4_unicode_ci", although Gitea could work with it, there might be some rare cases which don't work as expected.
2024/09/02 05:06:14 ...dels/db/collation.go:188:preprocessDatabaseCollation() [E] There are 278 table columns using inconsistent collation, they should use "utf8mb4_unicode_ci". Please go to admin panel Self Check page
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table gpg_key_import Column content db nullable is true, struct nullable is false
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project column board_type db type is INT(10) UNSIGNED, struct type is INT UNSIGNED
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project column card_type db type is INT(11), struct type is INT UNSIGNED
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project Column card_type db default is 0, struct default is 
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project Column card_type db nullable is false, struct nullable is true
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project column type db type is INT(10) UNSIGNED, struct type is INT UNSIGNED
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table review_state Column commit_sha db nullable is true, struct nullable is false
2024/09/02 05:06:15 cmd/web.go:194:serveInstalled() [W] Table label Column archived_unix db default is , struct default is NULL
2024/09/02 05:06:15 routers/init.go:144:InitWebInstalled() [I] ORM engine initialization successful!
2024/09/02 05:06:15 ...er/issues/indexer.go:76:func1() [I] PID 1132: Initializing Issue Indexer: bleve
2024/09/02 05:06:15 ...xer/stats/indexer.go:41:populateRepoIndexer() [I] Populating the repo stats indexer with existing repositories
2024/09/02 05:06:15 ...xer/stats/indexer.go:87:populateRepoIndexer() [I] Done (re)populating the repo stats indexer with existing repositories
2024/09/02 05:06:15 ...er/issues/indexer.go:153:func2() [I] Issue Indexer Initialization took 1.682015ms
2024/09/02 05:06:15 ...s/actions/cleanup.go:36:cleanExpiredArtifacts() [I] Found 0 expired artifacts
2024/09/02 05:06:15 ...s/actions/cleanup.go:60:cleanNeedDeleteArtifacts() [I] Found 0 artifacts pending deletion
2024/09/02 05:06:15 cmd/web.go:304:listen() [I] Listen: http://0.0.0.0:3000
2024/09/02 05:06:15 cmd/web.go:308:listen() [I] AppURL(ROOT_URL): http://gitea.syize.cn/
2024/09/02 05:06:15 cmd/web.go:311:listen() [I] LFS server enabled
2024/09/02 05:06:15 ...s/graceful/server.go:50:NewServer() [I] Starting new Web server: tcp:0.0.0.0:3000 on PID: 1132

Gitea Version

1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.46.0

Operating System

Arch Linux

How are you running Gitea?

I install the latest gitea from Arch Linux repo

pacman -Qi gitea

Name            : gitea
Version         : 1.22.1-1
Description     : Painless self-hosted Git service, community managed.
Architecture    : x86_64
URL             : https://gitea.io
Licenses        : MIT
Groups          : None
Provides        : None
Depends On      : git
Optional Deps   : mariadb: MariaDB support [installed]
                  memcached: MemCached support
                  openssh: GIT over SSH support [installed]
                  pam: Authentication via PAM support [installed]
                  postgresql: PostgreSQL support
                  redis: Redis support
                  sqlite: SQLite support [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 101.67 MiB
Packager        : Robin Candau <antiz@archlinux.org>
Build Date      : 2024年08月04日 星期日 16时13分12秒
Install Date    : 2024年08月05日 星期一 11时16分13秒
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

and make it a system service by using systemd

sudo systemctl status gitea.service

● gitea.service - Gitea (Git with a cup of tea)
     Loaded: loaded (/usr/lib/systemd/system/gitea.service; enabled; preset: disabled)
     Active: active (running) since Mon 2024-09-02 16:14:55 CST; 51min ago
 Invocation: 0b2f0c623432438cbb02ae9595534496
   Main PID: 108749 (gitea)
      Tasks: 25 (limit: 38295)
     Memory: 173.8M (peak: 175.5M)
        CPU: 3.414s
     CGroup: /system.slice/gitea.service
             └─108749 /usr/bin/gitea web -c /etc/gitea/app.ini

Database

MySQL/MariaDB

[silverwind]

Could be be the bleve indexer, see https://github.com/go-gitea/gitea/issues/31565 and try disabling it via REPO_INDEXER_ENABLED = false.

[Syize]

set REPO_INDEXER_ENABLED = false doesn't help.

[Syize]

set REPO_INDEXER_ENABLED = false doesn't help.

I set REPO_INDEXER_ENABLED = false and observed Gitea for a few days. This issue won't happen even Gitea has run 1-2 days, but it happens after system reboot, and the change of system time seems to be one of the causes triggering this problem.

Sometimes I switch to Windows to play games and then switch back to Arch Linux. Because of the different ways Windows and Linux handle the hardware clock (its a normal issue), the system time in Linux is 8 hours ahead(because I'm in UTC+8 timezone), and Linux will update system time as soon as it connect to Internet. I found this issue occurs immediately after system time update. Restart gitea manually could calm Gitea down.

As for Gitea's log, I have changed log level to trace, but Gitea doesn't output any additional information in log/gitea.log.