magento / magento2

Prior to making any Submission(s), you must sign an Adobe Contributor License Agreement, available here at: https://opensource.adobe.com/cla.html. All Submissions you make to Adobe Inc. and its affiliates, assigns and subsidiaries (collectively “Adobe”) are subject to the terms of the Adobe Contributor License Agreement.
http://www.magento.com
Open Software License 3.0
11.54k stars 9.32k forks source link

Dependencies of catalogsearch_fulltext index in scheduled mode #22885

Open flauschi opened 5 years ago

flauschi commented 5 years ago

I have found a serious problem with the scheduled reindex of the catalogsearch_fulltext index and the use of Elasticsearch.

Dependencies between the catalogsearch_fulltext index and other indexes (probably catalog_category_product, cataloginventory_stock, catalog_product_price as in Magento/Elasticsearch/etc/indexer.xml) are not taken into account if the mode is set to "UPDATE BY SCHEDULE".

The described steps to reproduce are a very theoretically procedure. But the problem becomes really annoying with imports of many products (and large sets of products waiting for reindex).

The problem could lead to missing products on the frontend. But I think there could be more things.

Could be related to: https://github.com/magento/magento2/issues/13927

Preconditions (*)

  1. Magento Community Edition (CE) 2.3.1 & 2.3-develop

Steps to reproduce (*)

The problem is not easy to reproduce, because it is a timing issue:

  1. Set mode of all indexers to "UPDATE BY SCHEDULE".
  2. Activate Elasticsearch (Stores/Configuration/Catalog/Catalog/Catalog Search) and make sure it works.
  3. Have an inactive product which is assigned to a category.
  4. Run magento indexer:reindex on console to ensure all indexes are up to date.
  5. Make sure cron is configured.
  6. Now you have to activate the product during the run of the cron reindex after catalog_category_product has been processed. To achieve this scenario, I suggest to add a "sleep(5);" after $view->update(); in Magento\Framework\Mview\Processor::update. For debugging, I also suggest to execute this cron separatly and to echo the Ids.
  7. catalogsearch_fulltext is processed for the activated product.
  8. Wait for second run of cron reindex. This updates the catalog_category_product index.

Expected result (*)

  1. Activated product should be visible on frontend.

Actual result (*)

  1. Activated product is not visible on frontend.
m2-assistant[bot] commented 5 years ago

Hi @flauschi. Thank you for your report. To help us process this issue please make sure that you provided the following information:

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me 2.3-develop instance - upcoming 2.3.x release

For more details, please, review the Magento Contributor Assistant documentation.

@flauschi do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?


m2-assistant[bot] commented 5 years ago

Hi @engcom-Charlie. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

magento-engcom-team commented 5 years ago

:white_check_mark: Confirmed by @engcom-Charlie Thank you for verifying the issue. Based on the provided information internal tickets MC-19840 were created

Issue Available: @engcom-Charlie, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

flauschi commented 4 years ago

Is there an update on the issue?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

flauschi commented 3 years ago

This shouldn't be marked as stale.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

flauschi commented 3 years ago

Again: This shouldn't be marked as stale.

danilkrasnorudskyi commented 3 years ago

same issue on magento 2.3.7-p1

northdakota commented 2 years ago

Same thing on magento 2.4.2-p2

gerasart commented 1 year ago

any news ?

m2-assistant[bot] commented 5 months ago

Hi @engcom-Dash. Thank you for working on this issue. Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:


lily-PHP commented 5 months ago

same isssue with magento 2.4.6, anyone had fix this? please help...

engcom-Dash commented 5 months ago

Hi @flauschi

Thanks for reporting and collaboration.

I verified the issue on Magento 2.4.7 with Elasticsearch 7 but the issue is not reproducible.

Able to see the product on UI after activating the product

Steps to reproduce:

  1. Set the mode of all indexers to "UPDATE BY SCHEDULE".
  2. Activate Elasticsearch (Stores/Configuration/Catalog/Catalog/Catalog Search) and make sure it works.
  3. Have an inactive product which is assigned to a category.
  4. Run Magento indexer: reindex on the console to ensure all indexes are up to date.
  5. Make sure cron is configured.
  6. Now you have to activate the product during the run of the cron reindex after catalog_category_product has been processed. To achieve this scenario, I suggest adding a "sleep(5);" after $view->update(); in Magento\Framework\Mview\Processor::update. For debugging, I also recommend executing this cron separately and to echo the Ids.
  7. catalogsearch_fulltext is processed for the activated product.
  8. Wait for second run of cron reindex. This updates the catalog_category_product index.

Please take a look at the attached screen recording.

https://github.com/magento/magento2/assets/60198592/ee0edb41-6b6f-4f11-8211-8f25c49c50fb

flauschi commented 5 months ago

Hello, the order to trigger this bug is really important: During cron run enable the product after index catalog_category_product has finished updating but before catalogsearch_fulltext index has started processing. In this scenario catalog_category_product is updated in the next cron run. But catalogsearch_fulltext will not run again for this product. This leads to the missing product on the frontend.

This is very theoretically for a small set of products. But for a large set of products and longer running reindex processes this is really a huge problem.

Currently we have worked around this bug by completly avoiding product updates during cron run (by not using the Magento backend for product updates and a asynchronous API to the ERP/PIM systems).