Performance issues with conversation tab in PR with comments/threads and images/rst files changes

[Persimmonboy]

Description

Hello Team,

We are experiencing significant slow loading times (about 3 minutes) in Gitea when dealing with pull requests (/docs/application-performance-management/pulls/6) that include about 580 review sections with 700 comments and file changes, with about 75 files involved. This problem occurs in Gitea version 1.19.3 and persists after upgrading to version 1.21.6, deployed on Kubernetes with Redis and an external PostgreSQL database. We've confirmed there are no issues with disk, CPU, or RAM usage, as well as with the database itself.

The pull request page, especially the "Conversation" tab, loads very slowly. However, if we recreate the pull request for the same branches and add a similar number of comments, using a simple word like "test" in each, the loading time is normal. This suggests that the slowdown may be related specifically to the opening of issue threads linked to certain parts of the code. The other tabs in the pull request, such as "Commits" and "Files Changes", load within a reasonable time.

It's important to note that there are several pull requests experiencing this issue, which suggests a consistent pattern or underlying problem.

Logs and Stacktrace outputs are in attachments:

gitea-logs.log gitea-diagnosis-20240618-103535.zip

Could this issue be indicative of a bug, or is there a potential optimization strategy that we might consider? Any advice or assistance in resolving these slowdowns would be greatly appreciated.

Gitea Version

1.21.6

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

2.40.1

Operating System

Linux gitea-85d5dc6896-6brvw 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 Linux

How are you running Gitea?

We are running Gitea on Kubernetes, deployed using the official Helm chart available at https://gitea.com/gitea/helm-chart. This setup includes Redis for caching and an external PostgreSQL database for data storage.

Database

PostgreSQL

[Persimmonboy]

I think the problem is related to the frontend part on the client side that handles the review sections with comments, as the backend seems to load in ~15 seconds, after which about 3 minutes or more comments are loaded:

[wxiaoguang]

Since 1.21 is outdated and won't get regular fixes, could you upgrade to 1.22-nighty (it will be 1.22.1 soon and contains a lot of bug fixes)?

Then could you provide the browser's profile?

Update: if it's impossible to upgrade, continue debugging with 1.21 might also be OK .....


[wxiaoguang]

Well, indeed ..... slow $.fn.tab

[Persimmonboy]

@wxiaoguang I updated to version 1.22-nightly, but nothing changed. PR conversation tab loading remained slow.

Current version:

gitea-69c59cbf5f-d6nd2:/var/lib/gitea$ gitea --version
Gitea version 1.22.0+56-ge8e43a7ee4 built with GNU Make 4.4.1, go1.22.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify
gitea-69c59cbf5f-d6nd2:/var/lib/gitea$

Link to bucket with performance profle json

[wxiaoguang]

Thank you very much. I will try to reproduce it and try to fix

[wxiaoguang]

I can reproduce it now. WIP: Fix tab performance #31437 .

Need to figure out how to fix ....

[Persimmonboy]

@wxiaoguang

Thanks for the quick reply!

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

I'm trying to reproduce the problem, and perfomance profile looks similar, but it loads faster, because in my problematic PR resolved reviews with comments and the comments themselves are more, which I mentioned in the first post, than in the test one.

Here is my test repository:

https://demo.gitea.com/ashakhba/application-perfomance-managment/pulls/1

[wxiaoguang]

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

That's another point, so maybe also related.

[wxiaoguang]

Link to bucket with performance profle json

According to this profile report, your instance's slowdown is 100% related to the $.fn.tab

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

However, it could still be a problem (another problem ......)

---

Are you able to run main-nightly (aka unreleased 1.23-nightly) in your production? If yes, I could provide more changes in the main branch, because some refactoring changes might not be suitable to backport to the stable 1.22 release.

If you could only use 1.22.x at the moment .... I could try to backport some changes if they are proven to work in 1.23, but I can't 100% guarantee at the moment.

[Persimmonboy]

@wxiaoguang

This is the dev environment I reproduced using helm chart gitea from the prod gitea data and database, so I can update.

The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right?

[wxiaoguang]

The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right?

Yes, nightly-rootless is for unreleased 1.23-nightly, but I haven't made any attempt yet 🤣 , I will notify here when I would have made some progress.

[wxiaoguang]

A new gitea/gitea:nightly-rootless is ready. Could you try it? I guess:

• The frontend slow-down problem should be almost resolved (it might be still a little slow not that slow)
• If it is still very slow, you could try to append _ui_performance_trace=1 as query parameter to that page, then the browser's console could print some trace logs when the page is fully loaded (eg: access https://localhost/?_ui_performance_trace=1 , https://localhost/?key=value&_ui_performance_trace=1: ref #31459
• And the browser's performance profile is still helpful if you could collect that data when a page is slow

[Persimmonboy]

@wxiaoguang

Thanky you for quick update.

I installed the version, and yes, it is indeed a bit slow, but not as slow as it was before, the difference is very strong, the page loads much faster (180 seconds -> ~30 seconds):

Perfomance trace with option /docs/application-performance-management/pulls/6?_ui_performance_trace=1:

Link to bucket with performance profle json

UPD: Changed to private.

[wxiaoguang]

According to the latest profile:

1. network takes about 22s (server backend & transport)
2. frontend JS takes about 7s
   • dropdown takes about 3s
   • tab takes about 3s

I guess I have no new idea to do quick optimization at the moment, since that PR is really large.

[wxiaoguang]

There are some config options in the [git] section of app.ini (eg: MAX_GIT_DIFF_LINES, MAX_GIT_DIFF_FILES), I am not sure whether reducing these numbers could make backend respond more quickly.

[Persimmonboy]

And can you please explain, in the end no changes were made?

I just see that only changed within the issue was the addition of "Add simple JS init performance trace #31459" and the behavior just changed within the latest changes from the main branch (nightly container in hub)?

[wxiaoguang]

And can you please explain, in the end no changes were made?

There are something made: for example: "Refactor image diff #31444" and optimize the tab related logic, it's a quite complex PR.

So I can say that "The frontend slow-down problem should be almost resolved (it might be still a little slow not that slow)", these changes did improve the performance.

---

So now the remaining problems are:

1. The backend response and transport are slow
   • Maybe it could be slightly improved by fine-tuning some options in [git], but since your commits are quite large, I guess it really needs some time to respond so many changed files.
   • It is a long-standing task in Gitea to optimize the performance for large repo & PRs, but at the moment I didn't see there would be some easy/quick methods to implement.
2. The frontend JS is not fast
   • See above: frontend JS takes about 7s: dropdown takes about 3s, tab takes about 3s
   • It needs more frontend work to refactor & rewrite (well, still not an easy task)

These tasks all depend on whether there are some contributors who have interests and time for it .....