search

magento / commerce-data-export

Open Software License 3.0
25 stars 22 forks source link

Performance issues with `catalog_data_exporter_products` indexer #25

Open jochhop opened 1 year ago

jochhop commented 1 year ago

There's a performance issue with the module magento/module-catalog-data-exporter, it takes more than 1 hour on processing catalog_data_exporter_products indexer, when the reindex happen through cron, the indexer takes so long that it puts in backlog other indexers scheduled after this one, see below image:

image

The issue seems to be related to out of memory issues when a few crons are scheduled at similar time to catalog_data_exporter_products indexer

Preconditions (*)

  1. Magento Cloud 2.4.3-p2 (it happens also in version 2.4.6, but we have widely tested in 2.4.3-p2)
  2. 40000 skus, 20 stores
  3. Install Adobe Product Recommendations (https://experienceleague.adobe.com/docs/commerce-merchant-services/live-search/guide-overview.html) and/or Adobe Live Search (https://experienceleague.adobe.com/docs/commerce-merchant-services/live-search/guide-overview.html)

Steps to reproduce (*)

  1. Have all indexers in mode: Update by schedule
  2. Having cron running indexers, from time to time there'll be memory issues and catalog_data_exporter_products will keep status = Working and multiple items in backlog (the issue specially could happen after massive products changes, eg. adding a catalog price rule which affects multiple products for multiple stores)

Expected result (*)

  1. Indexers should be processed correctly by cron and in a not escessive amount of time (at least catalog_data_exporter_products should be executed in much less than 1 hour in order not to backlog next scheduled indexers)

Actual result (*)

  1. catalog_data_exporter_products will keep status = Working and multiple items in backlog, see below screenshot

image

igorwulff commented 1 year ago

We've had similar issues that were caused by an badly written query. Could be the same? https://github.com/magento/commerce-data-export/issues/24 I'm not sure how versions are exactly handled with commerce releases, but could be worth a check? It is also fixed in the master branch for quite a while.

realadityayadav commented 9 months ago

@jochhop The product team has recently done a performance improvement on data exporter indexation and syncing. The improvements were released in the latest commerce data exporter version [ 103.1.5 ]. It seems that you are not in the latest commerce data exporter version. This helped us fix the problem of no results besing send to live search. Please update live-search with --with-dependencies flag to get this fixed.