Extremely slow translation system

[ioweb-gr]

Prerequisites

• [X] I understand and accept the project's code of conduct.
• [X] I have already searched in existing issues and found no previous report of this bug.

Describe the bug and add attachments

When trying to access the backend translations to actually translate strings I have to wait for minutes until the files are parsed and loaded. This wasn't the case in 1.7.5.2 but it is now with 1.7.8.9

https://blackfire.io/profiles/c53b4ada-4115-4778-a31f-5c8c87bf4b9e/graph

I'm attaching a profile result to give you a better idea.

Expected behavior

Translation page loads fast

Steps to reproduce

Go to translations page on 1.7.8.9 and watch how slow it is.

PrestaShop version(s) where the bug happened

1.7.8.9

PHP version(s) where the bug happened

7.4+

If your bug is related to a module, specify its name and its version

No response

Your company or customer's name goes here (if applicable).

IOWEB TECHNOLOGIES

[AureRita]

Hi @ioweb-gr

Thank you for your report, currently we can't go on other website to reproduce an issue, I tried to reproduce your delai but currently on my shop it's take not enought time as you can see :

Untitled_ Nov 17, 2023 4_54 PM.webm

Did you do anything else or did you use another theme ?

If you need support, you can find resources to help you on this page: Get help with PrestaShop.

Waiting for your feedback

[ioweb-gr]

I think your translation keys are too small a sample. I have 87367 translateable strings for example accross 670 translation files.

Yes we do use a custom theme but as seen in the blackfire profile, there's no custom function in the reading process. It's all core related so it's not something that's overriding the core functionality.

[ioweb-gr]

For example blackfire points out that all time is lost in PrestaShop\TranslationToolsBundle\Translation\Extractor\PhpExtractor::extract function in getting modules translations. We also have 152 modules installed.

[ShaiMagal]

@ioweb-gr

87k strings are NOK.

How can you have so much strings in BO translation? I checked about 10 real PrestaShops and every of them have approximatelly 5k.

Don't understand, how you you can have so much translations in BO.

[ioweb-gr]

I'm not sure how to respond to that. I guess the counting is accross languages?

I counted them roughly by parsing the translation files from the filesystem and counting the translation lines

If you have a way to count the exact number of translation strings in my installation I can provide it however does it change the fact of the slowness as depicted in the profiler?

[ShaiMagal]

@ioweb-gr if you open BO translation, what number you see on top right side? Post screenshot.

[ioweb-gr]

For a single language back office is 5056

[ShaiMagal]

And it's slow in BO translation as well?

[ioweb-gr]

No BO translation is fast. Front office translations for themes are slow.

Also it seems that in my case it's the GET request to api/translations/tree/el/themes/alysumchild

Which is about translating 450 strings only while taking 1.5mins to load

Perhaps to replicate you need a theme and a child theme?

[ioweb-gr]

I also noticed that on the default theme translations it's not invoking the same function callstack. The slow function is not invoked when trying to translate Core (no theme selected)

[ShaiMagal]

I tested core translation of front office and I see 800 strings. How you can have 80000? Can you send us image? I already asked before.

[ioweb-gr]

I think you misunderstood. It's 80k strings in all translation files not separated by area. However the system seems to parse all module files to determine which one to present. The end result may be 500 or 1000 per language per theme but in total it has to search a lot more to filter them out

[ShaiMagal]

Why you still didn't give us more informations and giving us only small parts of informations...?

A) how many languages do you have?

B) how you count 80000 language strings? How I can count number of language strings total in my PS? I would like to count this for our PS as well for comparing.

C) I tested translation on few PrestaShop (big - 100000products, ) with 10 languages and no performance problem here with translation.

If you don't provide us details, we can't help you.

@AureRita already tested and me as well. We aren't able to reproduce problem.

We need to reproduce it. If we aren't able to reproduce, we can't continue with investigating.

[ioweb-gr]

Why you still didn't give us more informations and giving us only small parts of informations...?

You didn't ask for anything specific my friend other than the exact way to reproduce it.

As far as information is needed I already gave you a full profile of the application's bottleneck where you can get the call stack

PrestaShop\TranslationToolsBundle\Translation\Extractor\PhpExtractor::extract
PrestaShop\PrestaShop\Core\Translation\Storage\Extractor\LegacyModuleExtractor::extract
PrestaShop\PrestaShop\Core\Translation\Storage\Provider\ModuleCatalogueLayersProvider::buildFreshDefaultCatalogueFromTemplates
PrestaShop\PrestaShop\Core\Translation\Storage\Provider\ModuleCatalogueLayersProvider::getDefaultCatalogueExtractedFromTemplates
PrestaShop\PrestaShop\Core\Translation\Storage\Provider\ModuleCatalogueLayersProvider::getDefaultCatalogue
PrestaShop\PrestaShop\Core\Translation\Storage\Provider\ModuleCatalogueLayersProvider::buildTranslationCatalogueFromLegacyFiles
PrestaShop\PrestaShop\Core\Translation\Storage\Provider\ModuleCatalogueLayersProvider::getFileTranslatedCatalogue
PrestaShop\PrestaShop\Core\Translation\Storage\Provider\ThemeCatalogueLayersProvider::getModulesTranslations
PrestaShop\PrestaShop\Core\Translation\Storage\Provider\ThemeCatalogueLayersProvider::getFileTranslatedCatalogue
PrestaShop\PrestaShop\Core\Translation\Builder\TranslationCatalogueBuilder::getRawCatalogue
PrestaShop\PrestaShop\Core\Translation\Builder\TranslationsTreeBuilder::getTree
PrestaShopBundle\Service\TranslationService::getTranslationsTree
PrestaShopBundle\Controller\Api\TranslationController::getTree
PrestaShopBundle\Controller\Api\TranslationController::listTreeAction
Symfony\Component\HttpKernel\HttpKernel::handleRaw
Symfony\Component\HttpKernel\HttpKernel::handle
AppKernel::handle

How more specific could I have been?

As far as your questions are concerned:

A) we're using only two languages English / Greek B) Try executing this simple oneliner which will aggregate all translation files and check translations and count the totals

find . -path "*translations/*.php" -exec cat {} \; | grep "= '" | wc -l
62341

C)

• Are you using any theme and / or child theme?
• Are you checking the translations there?
• Do you have enough modules installed? We have 152 including core + third party
• Is your version the same as ours 1.7.8.9?

Honestly I don't know what else to give you to reproduce this other than the solution itself.

[AureRita]

Hi @ioweb-gr

Thank for your report and your details, with the last details, I know I can say : this issue is too technical for me, ping @PrestaShop/prestashop-core-developers, can someone please reproduce this issue ?

Thanks

[Slews]

There's a discussion on the subject here: https://github.com/PrestaShop/PrestaShop/discussions/35557

I've posted a new comment there describing my background and analysis. To summarize here, the problem increases with the number of modules you have (about 150 in my case).

We're working on updating some stores from PS 1.7.6.5 to PS 8.1.7. Just by doing the PS upgrade, the loading time increase from less than 4 seconds to more than 30 seconds.

For now, we're using two tweaks to alleviate the problem a bit (we're now at 8-12 seconds), but it's really just that, tweaks that can't stay as is (I described what we did in my comment on the discussion).