search

WeblateOrg / weblate

Web based localization tool with tight version control integration.
https://weblate.org/
GNU General Public License v3.0
4.5k stars 993 forks source link

Translating is slow #9791

Closed tbodt closed 1 year ago

tbodt commented 1 year ago

Describe the issue

Every page load takes about 2 seconds. This doesn't sound like a lot but waiting 2 seconds just to see the next message after every 2 seconds spent editing or reading a message means everything takes twice as long and it drives me crazy. The only way to reasonably get anything done is zen mode. This is a problem in every weblate instance I've used, it's not specific to a single instance's setup.

I already tried

Steps to reproduce the behavior

Click almost any button or link on the tool that loads a new page

Expected behavior

Page should load in less than a second

Screenshots

No response

Exception traceback

No response

How do you run Weblate?

Other

Weblate versions

No response

Weblate deploy checks

No response

Additional context

No response

tbodt commented 1 year ago

I attempted to reproduce this locally and my local environment ended up being actually pretty fast. Wat

tomkolp commented 1 year ago

The larger project and translation memory, the slower it runs.

@nijel Often does not need bookmarks: Nearby strings, Other occurrences, Comments, Automatic suggestions, Other languages, History and I am interested in the text of the string in the first place. Is the initial information for these tabs being loaded? If so, it may allow you to load this information only after clicking on the tab. Such loading may be disabled by default for everyone and available for enabling in the profile.

Unless the reason for slow loading is something else, then it needs improvement rather than a workaround.

Big project slow loading, next string: https://weblate.eso-spolszczenie.eu/translate/eso-spolszczenie/101286772/pl/?q= Small project fast loading , next string: https://weblate.eso-spolszczenie.eu/translate/tylko-interfejs/client/pl/?q=

Same machine and weblate installation for both examples.

nijel commented 1 year ago

Automatic suggestions and Other languages are loaded on demand. Others are loaded directly because they are typically cheap to fetch. You can enable performance monitoring via Sentry and inspect what is causing the slowness.

tomkolp commented 1 year ago

Automatic suggestions and Other languages are loaded on demand. Others are loaded directly because they are typically cheap to fetch. You can enable performance monitoring via Sentry and inspect what is causing the slowness.

I have performance checking enabled in sentry. Unfortunately, the free plan only allows me to sign up for 2-3 days. I have records from a few days ago, is this an inquiry we are interested in? image

nijel commented 1 year ago

Yes, just scroll down to see what takes most of these 5 seconds.

tomkolp commented 1 year ago

image

image

image

image

image

image

tbodt commented 1 year ago

Have you restored the entire project and translation memory locally?

I haven't tried this - i'm not sure how to export the entire state like this of a weblate instance I don't own.

Looks like most of the time is spent on sql queries - good to know! Maybe the query plan analyzer is the next thing to check.

nijel commented 1 year ago

@tbodt You can use https://docs.weblate.org/en/latest/admin/backup.html#project-level-backups to transfer project from another instance.

@tomkolp

tomkolp commented 1 year ago

@tomkolp

  • There are nearly 2 seconds spent on the "Other translations" tab, this is not something we see on our servers. I guess that's on one of the huge projects you have.
  • Is there anything outstanding in the permissions span? This one is double compared to what I typically see.

in the "Other translations" tab there are from 2 to 200 items, but most are 2-5 items.

What other information can I provide?

Personally, I don't do translations and I didn't know it was so slow now.

Edit: This year we got massive 8,000-30,000 files. These are the cumulative files of the updates made by the game developers.

They help us by using "other instances" to assign existing segments to a specific update.

By means of automatic translation and propagation, these files show us what is missing.

Our base files are 900,000 segments, about 400 files. Every update to these files Only new segments are added.

To translate the quest, you would have to jump through all the files and look for segments. Therefore, we create new components that "group" scattered segments in one file. Using propagation, we complete the base 400 files. This is a convoluted solution, but it works great.

nijel commented 1 year ago

The trans_label query performance should be improved with 36c779b976. The original query shows a lot of variance (1ms - 2s), while the new one seems pretty stable around 1ms in my test environment.

in the "Other translations" tab there are from 2 to 200 items, but most are 2-5 items.

You can see the actual URL in the Sentry sidebar, can you share it?

tomkolp commented 1 year ago

The trans_label query performance should be improved with 36c779b. The original query shows a lot of variance (1ms - 2s), while the new one seems pretty stable around 1ms in my test environment.

in the "Other translations" tab there are from 2 to 200 items, but most are 2-5 items.

You can see the actual URL in the Sentry sidebar, can you share it?

https://eso-spolszczenie.sentry.io/share/issue/e42e66cf65454c509c9f5e04241be4e2/

tomkolp commented 1 year ago

Alternatively, I can send sentry login details to email. I only use Sentry to debug weblate. I think that the errors collected there will be more useful to you, I can't do anything with this knowledge. I don't report all bugs, only the critical ones.

tomkolp commented 1 year ago

The trans_label query performance should be improved with 36c779b. The original query shows a lot of variance (1ms - 2s), while the new one seems pretty stable around 1ms in my test environment.

I uploaded weblate updates, unfortunately I don't see any improvement in loading speed. EDIT: I can't confirm the invite in sentry because the free plan only allows 1 user. I have sent my login details.

nijel commented 1 year ago

Thanks, I've looked at some performance events and did fixes in e692544506e08604ac94b961bee385bef6cb0458 and 47178b51f3d7fb35f05f3bf9307654f5522f9982.

tomkolp commented 1 year ago

Thanks, I've looked at some performance events and did fixes in e692544 and 47178b5.

I have uploaded the patches. The loading for the logged in user is unchanged, while the anonymous user improved the loading speed to about 3s. Previously it was 5s for logged in and anonymous.

What does this difference mean? Can permission checks affect loading like this? Is there a way to check "permission audit" for a specific user? Edit: I did tests on 3 users with access to the whole project, here is the average result of loading 10 strings one by one: logged in= 80s anonymous=35s

I don't yet have results from users who have limited access to the project.

nijel commented 1 year ago

@tomkolp Looking at current metrics in Sentry (thanks for providing that!), nearly all /translate/ requests are now below 10 seconds. I've done some performance improvements today as well, so I think we should get somewhere under 5 seconds. This is still far from being perfect, but it's progress.

Two outstanding issues would be get_other_units and glossary lookups performance.

@tbodt Can you please share where did you observe the slowness so that we can investigate that as well?

tomkolp commented 1 year ago

@tomkolp Looking at current metrics in Sentry (thanks for providing that!), nearly all /translate/ requests are now below 10 seconds. I've done some performance improvements today as well, so I think we should get somewhere under 5 seconds. This is still far from being perfect, but it's progress.

Two outstanding issues would be get_other_units and glossary lookups performance.

@tbodt Can you please share where did you observe the slowness so that we can investigate that as well?

@nijel I applied the fixes. Previously, the average string loading was 8s, now it is 4.55s.

tbodt commented 1 year ago

An example of the kind of page that I'm having problems with is https://hosted.weblate.org/translate/weblate/application/tok/ (just a translation page). It feels like it loads faster today, but there's still room for improvement, I'd hope for a page like that to load in under a second.

nijel commented 1 year ago

This one should get better once 75486c6634b5f418dd3ba0f2b24ae4ee70965afb is deployed (currently scheduled for next weekend together with https://github.com/weblate-status-bot/status/issues/570).

nijel commented 1 year ago

Just to provide some insight, this is how /translate/ performance was improved over the past 30 days:

image

I'll close this as resolved for 5.0.1. The only remaining issue is the performance of “Other units”, let's address this in https://github.com/WeblateOrg/weblate/issues/9903.

tomkolp commented 1 year ago

@tomkolp Looking at current metrics in Sentry (thanks for providing that!), nearly all /translate/ requests are now below 10 seconds. I've done some performance improvements today as well, so I think we should get somewhere under 5 seconds. This is still far from being perfect, but it's progress. Two outstanding issues would be get_other_units and glossary lookups performance. @tbodt Can you please share where did you observe the slowness so that we can investigate that as well?

@nijel I applied the fixes. Previously, the average string loading was 8s, now it is 4.55s.

After moving weblate from SAS drives to nvme, I managed to get down to 3.25s per string.

github-actions[bot] commented 1 year ago

Thank you for your report; the issue you have reported has just been fixed.

tbodt commented 1 year ago

Thanks for working on this!