search

magento / magento2

Prior to making any Submission(s), you must sign an Adobe Contributor License Agreement, available here at: https://opensource.adobe.com/cla.html. All Submissions you make to Adobe Inc. and its affiliates, assigns and subsidiaries (collectively “Adobe”) are subject to the terms of the Adobe Contributor License Agreement.
http://www.magento.com
Open Software License 3.0
11.52k stars 9.31k forks source link

Elasticsearch 7.0+ - 2.3.5- missing filters and products on Frontend #28966

Closed pics124 closed 4 years ago

pics124 commented 4 years ago

Preconditions (*)

1.Magento Version:2.3.5-p1

  1. Elastic search 7.0

Steps to reproduce (*)

  1. Create attribute filterable attributes for the products to be seen in search as well
  2. Setup and configure Elastic search 7.0+. In our case both Magento and Elastic search are configured on the same server. 3.Test successful configuration done in the Magento Admin for Elastic Search 4.Run reindexing of all tables, clear cache. 5.Test search results passing to elastic search and no firewalls is obstructing it.

Expected result (*)

  1. Whether MYSql search or Elastic Search, the category and attribute filters should work similarly. 2.All products on the category navigation should be seen as it is.

Actual result (*)

  1. When we switch from Mysql search to Elastic search, products, category and attribute filters go missing on the frontend. Although all the products and categories are in the backend. 2.When we directly hit the product link on the frontend, its seen. But not seen in default navigation.
  2. Again when we shift to Mysql search, all products are seen in category navigation. The products and category filters go missing only when Elastic search is on.

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

m2-assistant[bot] commented 4 years ago

Hi @pics124. Thank you for your report. To help us process this issue please make sure that you provided the following information:

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

m2-assistant[bot] commented 4 years ago

Hi @swathishs. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

swathishs commented 4 years ago

@magento give me 2.3.5-p1 instance

magento-engcom-team commented 4 years ago

Hi @swathishs. Thank you for your request. I'm working on Magento 2.3.5-p1 instance for you

magento-engcom-team commented 4 years ago

Hi @swathishs, here is your Magento instance. Admin access: https://i-28966-2-3-5-p1.instances.magento-community.engineering/admin_edf1 Login: 74fbe3e0 Password: 9b84f4fa71ab Instance will be terminated in up to 3 hours.

pics124 commented 4 years ago

Hi @swathishs

Please reply. Is the issue resolved?

swathishs commented 4 years ago

@magento give me 2.3.5-p1 instance

magento-engcom-team commented 4 years ago

Hi @swathishs. Thank you for your request. I'm working on Magento 2.3.5-p1 instance for you

yeegor commented 4 years ago

@pics124 @swathishs are there any updates on this issue? I am experiencing the same.

barryvdh commented 4 years ago

I can verify this issue on 2.3.5-p2 with ElasticSearch 7. Many products show correctly, but it's also missing quite a few products. In my case some configurable products. The products are visible when I visit the url directly, but they do not appear in the category pages or search results. All indexes were up-to-date and cache was cleared.

Switching back to MySQL solved the issue directly.

JoostWan commented 4 years ago

Same problem over here with configurable products that are missing with Elastic Search 7 like @barryvdh

mattias commented 4 years ago

We have the same issue here. They will come back as soon as you run bin/magento indexer:reindex manually, but after a few minutes there are products missing again. We have the exact same versions as @barryvdh

erfanimani commented 4 years ago

@mattias can you reproduce this on your local development environment?

mattias commented 4 years ago

@erfanimani We're trying hard to reproduce this. Locally nothing breaks, I'm using the exact same version of elasticsearch as I do on the production server. They're using position as sorting, and I've tried changing up positions locally to see if something would break but without success. Saving products randomly doesn't help either. Really not sure at all what's causing this.

I've also tried switching to production mode locally to see if it is specific to that but still nothing breaks.

One example of the issue is that the pagination will say 1-8 of 25, even though it's set to show 12 on each page. Basically skipping these missing products at the end. Moving to page two it'll say 13-24 of 25. And so on. Missing products can be on any page, doesn't have to be on page 1. The pagination will know this and tell exactly which range it is showing as shown in the example here.

The direct product link always works when this happens, but search and catalog will not show this product.

steved1982 commented 4 years ago

Same issue here, my 2 biggest grouped products with a 1200 configurations will not show in Elasticearch. All other grouped products with less configurations show fine. Same solution, going back to MySQL where all the products can be seen again but I would prefer to stay with Elasticsearch. @barryvdh

arek-neptuneweb commented 4 years ago

I had the same problem, found a solution here: https://magento.stackexchange.com/questions/322516/magento-2-x-elasticsearch-category-products-not-showing

alucardatem commented 4 years ago

This is still happening in my case. single store with 33k products however tho only way to show all products is to do bulk update and assign all products to website or re-save product. on issuing a reset on reindex, all products dissapear then when reindex finished not all of them are back again - newly created products do not show

m2-assistant[bot] commented 4 years ago

Hi @engcom-Oscar. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

engcom-Oscar commented 4 years ago

Hello @pics124

Thank you for your report. Unfortunately, I was not able to reproduce this issue on the latest 2.4-develop code, as MySQL is deprecated from it, and products with filterable attributes work well with Elasticsearch.
Probably issue was already fixed, or we missed some steps or preconditions. If you have some more details how to reproduce this issue on latest 2.4 Magento, feel free to update the description and reopen it.

alucardatem commented 4 years ago

@arek-neptuneweb unfortunately that solution does not work at all - at least not in 2.3.5-p1 and not only that, i have tried out ES 5.X 6.X and 7.X and all of them gave the exact same result. as @barryvdh has mentioned in his comments the exact same thing is happening to us. a massive reindex after doing a bulk update to assign products to website removes them again.

alucardatem commented 4 years ago

Hello I am afraid that this issue is still ongoing. why has it been closed?

alucardatem commented 4 years ago

@barryvdh - quick question --- do you happen to use Amasty Elastic search module by any chance ?

barryvdh commented 4 years ago

Ehm yes probably that is also enabled.

alucardatem commented 4 years ago

Did you try disabling amasty search extension, reset the indexes, change them to Magento elastic search, delete the indices from elastic search then reindex? Everything might be working after @barryvdh but we lose the search from Amasty.

tkashro commented 4 years ago

@alucardatem @barryvdh Just a FYI incase, I was experiencing basically the same issue. Our catalog_category_product and catalogrule_rule indexers were consequently always in the REINDEX REQUIRED state. In the end they were actually 2 separate causes, the first was due to Amasty Automatic Categories and second was due to Amasty Layered Navigation.

We had upgraded to 2.3.5-p2 and migrated servers consecutively and so it was a painful process trying to get to the bottom of it, turns out it was just due to third party modules.

alucardatem commented 4 years ago

I have the amasty layered nav also and the amasty advanced search with elastic search installed. However server spec wise 8gb ram with 4 cores and total no of products is around 30k mark. However the strange part is the fact that a bulk update to products setting them all to website or even creating a new product from admin they show instantly, but the moment the indexers run, it is all gone. What are your server specs @tkashro ?

tkashro commented 4 years ago

@alucardatem 26GB RAM and 26 VCPU, we have only around 10k mark for products. Are your indexers running successfully? our seemed to be in a constant loop - and the indices being created in elasticsearch were rapidly going up, we had something like magento_production_product1v281

alucardatem commented 4 years ago

@tkashro yes, same here

hostep commented 4 years ago

@tkashro:

I was experiencing basically the same issue. Our catalog_category_product and catalogrule_rule indexers were consequently always in the REINDEX REQUIRED state. In the end they were actually 2 separate causes, the first was due to Amasty Automatic Categories and second was due to Amasty Layered Navigation.

This is actually a core issue in Magento from my findings, see https://github.com/magento/magento2/pull/29196, Magento is currently refactoring the PR, but we had success with applying this patch: https://github.com/magento/magento2/pull/29196/commits/f0c645307e031f9c6b8beb31f18c801dc62c02e3 to get this resolved

alucardatem commented 4 years ago

@hostep do you think that this is in relation to the fact that products are falling off when performing a reindex however this is the cats just for newly added products? I mean... On 2.3.5 on switching to mysql search engine and reindexing, all products are showing, even the new ones, after changing to elastic search 6.8/7.5/7.6/7.9 or to Amasty elastic search and reindexing only a part of the products show on category pages, however the products are accessible by URL and they are all in stock, enabled, catalog, search and there are no bad attributes in there. The moment i do a bulk update and adding products to the only existing site/store/storeview we have after that is running products line up and show. However, the moment that the reindex starts running the products that started showing dissapear in a instant

tkashro commented 4 years ago

@tkashro:

I was experiencing basically the same issue. Our catalog_category_product and catalogrule_rule indexers were consequently always in the REINDEX REQUIRED state. In the end they were actually 2 separate causes, the first was due to Amasty Automatic Categories and second was due to Amasty Layered Navigation.

This is actually a core issue in Magento from my findings, see #29196, Magento is currently refactoring the PR, but we had success with applying this patch: f0c6453 to get this resolved

Wow, that seems spot on. Weirdly I could no longer and still can't reproduce the issue on our install after making the changes that I made, nevertheless that seems to be the problem.

hostep commented 4 years ago

@alucardatem, it doesn't sound related no. However if you have problems with indexers which are constantly being reindexed for no good reason (which can then cause your issue every few minutes I'm guessing), then that patch could already make it so that it doesn't happen every few minutes but only when actual reindexing is needed.

Your problem is probably something completely different then what I was talking about, this was just commenting on @tkashro's remark.

alucardatem commented 4 years ago

@hostep do you happen to have a composer version of the patch ?

hostep commented 4 years ago

Sure: PR-29196-Indexer.txt

thlassche commented 4 years ago

@hostep @pics124

I saw the same for some our clients, but in our case it were large bundles that would go missing.

Turned out in our case this was the culprit:

[2020-10-14T13:04:22,816][DEBUG][o.e.a.b.TransportShardBulkAction] [di-u1v-] [magento2-prod_product_3_v4418][0] failed to execute bulk item (index) index {[magento2-prod_product_3_v4418][document][13421], source[n/a, actual length: [106.5kb], max length: $
java.lang.IllegalArgumentException: DocValuesField "name.sort_name" is too large, must be <= 32766

That explains why only large products go missing: child product names are imploded in that field, so if there are too many child products the 32766 field size limit will be hit.

Place in the code that causes this: https://github.com/magento/magento2/blob/fa6d291646b1ca0ebc0ebd3798ec8aaae634d486/app/code/Magento/Elasticsearch/Model/Adapter/BatchDataMapper/ProductDataMapper.php#L298

I drafted a quick patch that caps the sortable field on 32766 characters, you can apply it to magento/module-elasticsearch in your composer.json. Have not yet found any side effects.

Patch: fix-for-too-long-sortable-attributes.txt

@engcom-Oscar I replicated this on 2.4.0 btw, so it's a real issue. To replicate, make sure you have a product with a lot of (in my case a few thousand) child products (e.g. a bundle or a configurable)

barryvdh commented 4 years ago

Interesting, I had the issue with a lot of configurables (eg few hundred simples) so might be the same issue.

@thlassche maybe open a new issue with the tests (not sure if they bother looking at closed requests?)

Does the productId value matter? Otherwise a hash might also be a good option, or the configurable sku + a hash of the options?

thlassche commented 4 years ago

@barryvdh I think the value is used for sorting, so a hash will not do the trick (did not check it in depth though), figured that if that was the case it was a fair tradeoff to just cut the string and accept that product name sorting for the very large products was not exactly right.

See also the commit that introduced this: https://github.com/magento/magento2/commit/edbd6b5668307a1a4343f8772ed5869985c4fe1e#diff-90b7adeaeb5217406c3f59114584fbf4d5159bc81cab99a794baaada0f1e8259R167

I'll open a fresh issue about it indeed.

dinkeronline commented 3 years ago

This could be possibly related to this thread Stack Overflow

I had a similar issue and I found it was an attribute creating problems.

iphigenie commented 3 years ago

I have an opposite issue where grouped products without any children aren't shown in catalog lists, but they are counted in the pagination

jurvi commented 3 years ago

Why the heck is this closed, it's definately a real issue a lot of people are clearly struggling with, and some basic developer just closes it?

georgebabarus commented 3 years ago

@magento give me 2.3.5-p1 instance

magento-deployment-service[bot] commented 3 years ago

Hi @georgebabarus. Thank you for your request. I'm working on Magento instance for you.

magento-deployment-service[bot] commented 3 years ago

Hi @georgebabarus, unfortunately there is no ability to deploy Magento instance at the moment. Please try again later.

georgebabarus commented 3 years ago

@magento give me 2.3.5-p1 instance

magento-deployment-service[bot] commented 3 years ago

Hi @georgebabarus. Thank you for your request. I'm working on Magento instance for you.

magento-deployment-service[bot] commented 3 years ago

Hi @georgebabarus, unfortunately there is no ability to deploy Magento instance at the moment. Please try again later.

georgebabarus commented 3 years ago

@magento give me 2.3.5-p1 instance

magento-deployment-service[bot] commented 3 years ago

Hi @georgebabarus. Thank you for your request. I'm working on Magento instance for you.

magento-deployment-service[bot] commented 3 years ago

Hi @georgebabarus, unfortunately there is no ability to deploy Magento instance at the moment. Please try again later.