search

Smile-SA / elasticsuite

Smile ElasticSuite - Magento 2 merchandising and search engine built on ElasticSearch
https://elasticsuite.io
Open Software License 3.0
761 stars 341 forks source link

Huge slowness and shards failing with big catalog #2496

Closed Vormillion closed 2 years ago

Vormillion commented 2 years ago

Preconditions

We have catalog_product index with 34748201 documents and 1.3 GB size. Elasticsearch heap size is set to 4GB. 0 replicas, 1 shard.

Magento Version : 2.4.3 EE

ElasticSuite Version : ~2.10.0

Environment :Production

Third party modules :

Steps to reproduce

  1. Navigate to any page and observer huge slowness in NewRelic for call: GuzzleHttp\Ring\Client\CurlHandler::_invokeAsArray (called by elastic module). It's resulting with 502/504 errors and Elasticsearch is throwing errors:

{"type": "server", "timestamp": "2022-03-21T17:13:09,312Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "docker-cluster", "node.name": "elasticsearch-76bf8cb768-r5jzz", "message": "path: /magento2_default_catalog_product/_search, params: {index=magento2_default_catalog_product}", "cluster.uuid": "r3q-R_fgRFSTx9IEqJ8UoQ", "node.id": "05MZI6SHQYGtpTSbK2BH8g" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.0.jar:7.9.0]",

Expected result

  1. Site is browsing smoothly.

Everything happened after migration from Magento 2.3.x to 2.4.3. On testing environments with lower load but the same database size we didn't observer similar issues. What are recommended settings in terms of number of shards for such amount of products?

romainruaud commented 2 years ago

Environment: Production and "docker-cluster" ?

Vormillion commented 2 years ago

Magento (production mode) and elasticsearch is running in Kubernetes .

romainruaud commented 2 years ago

can you enable debug mode of Elasticsuite and check if you have something relevant appearing on the system.log ?

Also, do you have any output from the Elasticsearch nodes ?

Vormillion commented 2 years ago

I enabled already debug mode but I can't find any debug message which correlates with Elasticsearch error that all shards failed. Magento is generation such queries and I assume some of them are failing:

curl -XPOST 'http://elasticsearch/magento2_default_catalog_product/_search?pretty=true' -d '{"size":"9","sort":[{"_score":{"order":"desc"}},{"entity_id":{"order":"desc","missing":"_first","unmapped_type":"keyword"}}],"from":0,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"must":[],"must_not":[],"should":[{"bool":{"must":[],"must_not":[],"should":[{"match":{"sku.reference":{"query":"SFS-007-WPS-PCD-1500","minimum_should_match":"100%","boost":1}}}],"minimum_should_match":1,"boost":1}},{"bool":{"must":[],"must_not":[],"should":[{"match":{"sku.reference":{"query":"SFS-007-WPS-CC1-1631","minimum_should_match":"100%","boost":1}}}],"minimum_should_match":1,"boost":1}}],"minimum_should_match":1,"boost":1}},{"term":{"stock.is_in_stock":{"value":true,"boost":1}}},{"terms":{"visibility":[2,4],"boost":1}},{"term":{"customer_group_id":{"value":13549,"boost":1}}},{"terms":{"visibility":[2,4],"boost":1}},{"terms":{"stock.is_in_stock":[true],"boost":1}}],"must_not":[],"should":[],"boost":1}},"boost":1}},"aggregations":{"attribute_set_id":{"terms":{"field":"attribute_set_id","size":100000,"order":{"_count":"desc"}}},"categories":{"filters":{"filters":{"3":{"nested":{"path":"category","score_mode":"none","query":{"bool":{"must":[{"bool":{"must_not":[{"term":{"category.is_virtual":{"value":true,"boost":1}}}],"boost":1}},{"terms":{"category.category_id":["3"],"boost":1}}],"must_not":[],"should":[],"boost":1}},"boost":1}},"31":{"nested":{"path":"category","score_mode":"none","query":{"bool":{"must":[{"bool":{"must_not":[{"term":{"category.is_virtual":{"value":true,"boost":1}}}],"boost":1}},{"terms":{"category.category_id":["31"],"boost":1}}],"must_not":[],"should":[],"boost":1}},"boost":1}},"53":{"nested":{"path":"category","score_mode":"none","query":{"bool":{"must":[{"bool":{"must_not":[{"term":{"category.is_virtual":{"value":true,"boost":1}}}],"boost":1}},{"terms":{"category.category_id":["53"],"boost":1}}],"must_not":[],"should":[],"boost":1}},"boost":1}},"58":{"nested":{"path":"category","score_mode":"none","query":{"bool":{"must":[{"bool":{"must_not":[{"term":{"category.is_virtual":{"value":true,"boost":1}}}],"boost":1}},{"terms":{"category.category_id":["58"],"boost":1}}],"must_not":[],"should":[],"boost":1}},"boost":1}},"61":{"nested":{"path":"category","score_mode":"none","query":{"bool":{"must":[{"bool":{"must_not":[{"term":{"category.is_virtual":{"value":true,"boost":1}}}],"boost":1}},{"terms":{"category.category_id":["61"],"boost":1}}],"must_not":[],"should":[],"boost":1}},"boost":1}},"62":{"nested":{"path":"category","score_mode":"none","query":{"bool":{"must":[{"bool":{"must_not":[{"term":{"category.is_virtual":{"value":true,"boost":1}}}],"boost":1}},{"terms":{"category.category_id":["62"],"boost":1}}],"must_not":[],"should":[],"boost":1}},"boost":1}}}}},"option_text_pt_kind":{"terms":{"field":"option_text_pt_kind.untouched","size":100,"order":{"_count":"desc"}}},"option_text_pt_type":{"terms":{"field":"option_text_pt_type.untouched","size":100,"order":{"_count":"desc"}}}},"track_total_hits":true,"_source":{"excludes":["price","customer_group_id"]}}'

Vormillion commented 2 years ago

I just spotted one strange thing - why in debug log, this curl is with http://elasticsearch/ (so port 80), while in core_config_data I've configured: smile_elasticsuite_core_base_settings/es_client/servers --> elasticsearch:9200

romainruaud commented 2 years ago

I just spotted one strange thing - why in debug log, this curl is with http://elasticsearch/ (so port 80), while in core_config_data I've configured: smile_elasticsuite_core_base_settings/es_client/servers --> elasticsearch:9200

This is just a misfit of the Elasticsearch logger, not an issue.

What you copy-pasted is the query sent to the engine, do you have lines below ? If there is an error during the response, it would be displayed.

What is the output of the Elasticsearch containers ?

no-response[bot] commented 2 years ago

This issue was waiting update from the author for too long. Without any update, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away! Thanks for your contribution.