Slow product edit page on large product site

magento / magento2

Prior to making any Submission(s), you must sign an Adobe Contributor License Agreement, available here at: https://opensource.adobe.com/cla.html. All Submissions you make to Adobe Inc. and its affiliates, assigns and subsidiaries (collectively “Adobe”) are subject to the terms of the Adobe Contributor License Agreement.

http://www.magento.com

Open Software License 3.0

11.44k stars 9.29k forks source link

Slow product edit page on large product site #24683

Closed visualwebs closed 4 years ago

visualwebs commented 4 years ago

Preconditions (*)

Tested in 2.2.5 & current 2.3.2 Open source.

Steps to reproduce (*)

We have a system with > 50.000 products and 6000 attributes, all caches enabled, using Redis for both caches and session, indexes set to scheduled mode and ready.
When editing the product in the backend it takes long time to be ready to operate. We tried to disable all the commercial modules, leaving only "Magento_" modules with the same results. We use a module to debug (Mgt_DeveloperToolbar) and informs the php part takes around 6 seconds and the rest is the loading components in the page. We see a kind of bottleneck in this call

eav_load

Expected result (*)

Page should load in an reasonable time.

Actual result (*)

It takes around 20 seconds for the page be ready to operate

m2-assistant[bot] commented 4 years ago

Hi @visualwebs. Thank you for your report. To help us process this issue please make sure that you provided the following information:

[ ] Summary of the issue
[ ] Information on your environment
[ ] Steps to reproduce
[ ] Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.3-develop instance - upcoming 2.3.x release

For more details, please, review the Magento Contributor Assistant documentation.

@visualwebs do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

[ ] yes
[ ] no

Join Magento Community Engineering Slack and ask your questions in #github channel.

orlangur commented 4 years ago

@visualwebs so why are there almost 7000 calls to getAttribute? Can you determine some attributes which could not be loaded for a particular product editing?

visualwebs commented 4 years ago

We use icecat module to autofill product info.

https://marketplace.magento.com/iceshop-icecatlive.html

that uses hundred of custom attributes starting with "i_"

See image.

icecat_custom_attributes

If I modify \Magento\Catalog\Model\Attribute\ScopeOverriddenValue, method ->getAttributes only to test how it works with all those attributes removed, modifying the searchCriteriaBuilder filters and removing all attributes with code like '%i_%' I save 10 seconds on load page. Any clues to exclude those non required attributes on page load. Looks system loads all attributes instead only those that are part of the current attribute set.

orlangur commented 4 years ago

@visualwebs,

do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

Looks like not. You should contact problematic extension developers.

visualwebs commented 4 years ago

But is possible to know why Magento load the entire list of 6000 attributes when editing the product? A lot of those attributes are not required to have loaded.

orlangur commented 4 years ago

@visualwebs it has nothing to do with Magento. It's how they implemented this extension.

orlangur commented 4 years ago

@visualwebs,

only to test how it works with all those attributes removed, modifying the searchCriteriaBuilder filters and removing all attributes with code like '%i_%' I save 10 seconds on load page

Ah, ok, from

We tried to disable all the commercial modules, leaving only "Magento_" modules with the same results

Looks like you disabled Iceshop_Icecatlive as well and it still loads too many attributes. My understanding is that Magento should only loaf attributes from current attribute set.

m2-assistant[bot] commented 4 years ago

Hi @engcom-Charlie. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

[ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
Details
If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.
[ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.
[ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
[ ] 4. Verify that the issue is reproducible on 2.3-develop branch
Details
- Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
- If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
- If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
[ ] 5. Add label Issue: Confirmed once verification is complete.
[ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

engcom-Charlie commented 4 years ago

Hello @visualwebs We are not able to reproduce this issue on a fresh Magento 2.3-develop install. Result: When "Edit product" page loading getAttributes()method finds only attributes for current attribute set. So i have to close this issue. Thanks for your report!

ruospo commented 4 years ago

But is possible to know why Magento load the entire list of 6000 attributes when editing the product? A lot of those attributes are not required to have loaded.

have solved your problem?

visualwebs commented 4 years ago

No. I have a store with 7000 attributes created by a service called icecat that autofills product data. In Magento 2.3.5-p1 a product edit page in the backend, takes around 15-20 seconds to be available. This happens with early 2.2.x, 2.3.x versions, too. If I delete those secondary attributes and leave only around 500 main attributes, the page takes around 5-8 seconds to be ready. So the amount of attributes affects seriously to product edit speed.

JelleGe commented 3 years ago

No. I have a store with 7000 attributes created by a service called icecat that autofills product data. In Magento 2.3.5-p1 a product edit page in the backend, takes around 15-20 seconds to be available. This happens with early 2.2.x, 2.3.x versions, too. If I delete those secondary attributes and leave only around 500 main attributes, the page takes around 5-8 seconds to be ready. So the amount of attributes affects seriously to product edit speed.

Well we use the same service from icecat and we face the same issue. Do you found a solution to speed up the product edit page?

visualwebs commented 3 years ago

No, but almost caching not system attributes speeds up the backend edit.

https://devdocs.magento.com/guides/v2.4/extension-dev-guide/attributes.html

bin/magento config:set dev/caching/cache_user_defined_attributes 1

JelleGe commented 3 years ago

Thanks! Sadly we already had enabled this, so seems that we can not benefit from this.

visualwebs commented 3 years ago

I discovered with this change in \Magento\Catalog\Model\Attribute\ScopeOverriddenValue

public function getDefaultValues($entityType, $entity)
{
            if ($entityType == "Magento\Catalog\Api\Data\ProductInterface")
                 return [];
    ...
}

The product edit starts to render in 4 seconds. So you can create an admin plugin to modify the behavior as proper way. Note this is just a temporary solution for icecat attribute slow loading and needs to be tested in env environment first.

JelleGe commented 3 years ago

Thanks! Seems indeed that this loads the product edit page much faster. Need to take a better look for implementing. We also still face a issue with the product grid with search enabled. The product grid without search is quite fast, but when you search the results are loaded within 7 to 12 seconds. Do you face the same issue?

visualwebs commented 3 years ago

Thanks! Seems indeed that this loads the product edit page much faster. Need to take a better look for implementing. We also still face a issue with the product grid with search enabled. The product grid without search is quite fast, but when you search the results are loaded within 7 to 12 seconds. Do you face the same issue?

Its very fast for price filtering and 5-10 seconds for sku search(not cached), so depends the complexity of filter used and the amount of products. In our case we have 50.000.

JelleGe commented 3 years ago

Seems that we always used the default search bar on the product grid, that is much much slower than searching on sku. When searching on sku the results are loaded within 2 seconds. Do you already created a module for your fix, that we also can use?

paragpadsumbiya commented 1 year ago

@visualwebs We are facing the same issue after upgrade to Magento 2.4.5 as it was working fine with 2.4.2 .. Anyone have idea how to resolve this ?

JelleGe commented 1 year ago

@paragpadsumbiya Already tried this? This worked perfect for us. https://github.com/magento/magento2/issues/24683#issuecomment-707156254

paragpadsumbiya commented 1 year ago

@JelleGe

Thanks for reply but we have noticed something very strange that issue is only with some of products only as its working fast for other products.

So its really strange as we have only 184 attributes.