query INSERT INTO “search_tmp…” never ends after mass attribute update

[rsulym]

Preconditions

Magento 2.2.1, php-fpm 7.1, MariaDB 10.3.5

Steps to reproduce

Perform mass attribute values update (~30000 items)

Expected result

1) Everything works fine

Actual result

1) re-index process finished ok, but: 2) A lot of queries INSERT INTO “search_tmp…” started, until server reaches pm.max_children parameter value and PHP-fpm die, and they constantly resurrecting even after mysql restart and process kill.

[luckyduck]

Same issue here. Thanks to @erikhansen and @sebastian-virtua , it could be fixed. Should be fixed in the core asap.

[euriskotech]

I assume now that M2.4 requires ElasticSearch this issue is resolved?

[gilbertsohal]

I m getting same in 2.3.4 below query killing my whole server

INSERT INTO search_tmp_5f733915d1be65_45326344 SELECT main_select.entity_id, SUM(score) AS `re

Can someone help me please..

[mnatsakanyan]

I m getting same in 2.3.4 below query killing my whole server

INSERT INTO search_tmp_5f733915d1be65_45326344 SELECT main_select.entity_id, SUM(score) AS `re

Can someone help me please..

the same problem we have - any solution?

[magento-engcom-team]

The report related to technology or functionality that was removed(or not supported) from the Magento code-base. Or cannot be fixed and delivered in the scope of this public repository.

Examples:

• Removed payment methods. Magento Commerce 2.4.0 Release Notes, Deprecation of Magento core payment integrations. All questions, issue reports, and fix for them should be addressed to the corresponding extension owners(support) on the Magento Market place page
• Mysql search. Replaced with Elasticsearch
• Not Supported browsers
• Not supported Operating systems
• Web Wizard install
• Not a bug
• etc

[bhaveshbalar]

[2020-12-14 17:29:31] main.ERROR: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'merkmale_filter.source_id' in 'on clause', query was: INSERT INTO qz8v_search_tmp_5fd7a0fb910a15_37972217 SELECT main_select.entity_id, SUM(score) AS relevance FROM (SELECT DISTINCT search_index.entity_id, (((0) + (0) + (0) + (0)) 1) AS score FROM qz8v_catalog_product_index_eav AS search_index INNER JOIN qz8v_cataloginventory_stock_status AS stock_index ON stock_index.product_id = search_index.entity_id AND stock_index.website_id = 0 AND stock_index.stock_status = 1 AND stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=187 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS merkmale_filter ON search_index.entity_id = merkmale_filter.entity_id AND merkmale_filter.attribute_id = 187 AND merkmale_filter.store_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=169 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS stoff_filter ON search_index.entity_id = stoff_filter.entity_id AND stoff_filter.attribute_id = 169 AND stoff_filter.store_id = 1 INNER JOIN qz8v_cataloginventory_stock_status AS merkmale_filter_stock_index ON merkmale_filter_stock_index.product_id = merkmale_filter.source_id AND merkmale_filter_stock_index.website_id = 0 AND merkmale_filter_stock_index.stock_status = 1 AND merkmale_filter_stock_index.stock_id = 1 INNER JOIN qz8v_cataloginventory_stock_status AS stoff_filter_stock_index ON stoff_filter_stock_index.product_id = stoff_filter.source_id AND stoff_filter_stock_index.website_id = 0 AND stoff_filter_stock_index.stock_status = 1 AND stoff_filter_stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=99 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS visibility_filter ON search_index.entity_id = visibility_filter.entity_id AND visibility_filter.attribute_id = 99 AND visibility_filter.value in ('2', '4') AND visibility_filter.store_id = '1' INNER JOIN qz8v_catalog_category_product_index_store1 AS category_ids_index ON search_index.entity_id = category_ids_index.product_id AND category_ids_index.store_id = '1' WHERE (search_index.store_id = '1') AND (category_ids_index.category_id in ('8')) AND (merkmale_filter.value IN ('5526')) AND (stoff_filter.value IN ('5448'))) AS main_select GROUP BY entity_id ORDER BY relevance DESC, entity_id DESC LIMIT 100000 [] [] [2020-12-14 17:33:27] main.ERROR: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'merkmale_filter.source_id' in 'on clause', query was: INSERT INTO qz8v_search_tmp_5fd7a1e746bd73_00356023 SELECT main_select.entity_id, SUM(score) AS relevance FROM (SELECT DISTINCT search_index.entity_id, (((0) + (0) + (0)) 1) AS score FROM qz8v_catalog_product_index_eav AS search_index INNER JOIN qz8v_cataloginventory_stock_status AS stock_index ON stock_index.product_id = search_index.entity_id AND stock_index.website_id = 0 AND stock_index.stock_status = 1 AND stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=187 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS merkmale_filter ON search_index.entity_id = merkmale_filter.entity_id AND merkmale_filter.attribute_id = 187 AND merkmale_filter.store_id = 1 INNER JOIN qz8v_cataloginventory_stock_status AS merkmale_filter_stock_index ON merkmale_filter_stock_index.product_id = merkmale_filter.source_id AND merkmale_filter_stock_index.website_id = 0 AND merkmale_filter_stock_index.stock_status = 1 AND merkmale_filter_stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=99 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS visibility_filter ON search_index.entity_id = visibility_filter.entity_id AND visibility_filter.attribute_id = 99 AND visibility_filter.value in ('2', '4') AND visibility_filter.store_id = '1' INNER JOIN qz8v_catalog_category_product_index_store1 AS category_ids_index ON search_index.entity_id = category_ids_index.product_id AND category_ids_index.store_id = '1' WHERE (search_index.store_id = '1') AND (search_index.attribute_id = 187) AND (search_index.value in ('5526')) AND (category_ids_index.category_id in ('8')) AND (merkmale_filter.value IN ('5526'))) AS main_select GROUP BY entity_id ORDER BY relevance DESC, entity_id DESC LIMIT 100000 [] [] [2020-12-14 17:33:48] main.ERROR: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'gewicht_filter.source_id' in 'on clause', query was: INSERT INTO qz8v_search_tmp_5fd7a1fcabb313_46452964 SELECT main_select.entity_id, SUM(score) AS relevance FROM (SELECT DISTINCT search_index.entity_id, (((0) + (0) + (0) + (0)) 1) AS score FROM qz8v_catalog_product_index_eav AS search_index INNER JOIN qz8v_cataloginventory_stock_status AS stock_index ON stock_index.product_id = search_index.entity_id AND stock_index.website_id = 0 AND stock_index.stock_status = 1 AND stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=189 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS gewicht_filter ON search_index.entity_id = gewicht_filter.entity_id AND gewicht_filter.attribute_id = 189 AND gewicht_filter.store_id = 1 INNER JOIN qz8v_cataloginventory_stock_status AS gewicht_filter_stock_index ON gewicht_filter_stock_index.product_id = gewicht_filter.source_id AND gewicht_filter_stock_index.website_id = 0 AND gewicht_filter_stock_index.stock_status = 1 AND gewicht_filter_stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=99 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS visibility_filter ON search_index.entity_id = visibility_filter.entity_id AND visibility_filter.attribute_id = 99 AND visibility_filter.value in ('2', '4') AND visibility_filter.store_id = '1' INNER JOIN qz8v_catalog_category_product_index_store1 AS category_ids_index ON search_index.entity_id = category_ids_index.product_id AND category_ids_index.store_id = '1' INNER JOIN qz8v_catalog_product_index_price AS price_index ON search_index.entity_id = price_index.entity_id AND price_index.website_id = '1' WHERE (search_index.store_id = '1') AND (search_index.attribute_id = 189) AND (search_index.value in ('5549')) AND (category_ids_index.category_id in ('8')) AND (price_index.min_price >= '1000' AND price_index.customer_group_id = 0) AND (gewicht_filter.value IN ('5549'))) AS main_select GROUP BY entity_id ORDER BY relevance DESC, entity_id DESC LIMIT 100000 [] [] [2020-12-14 17:34:26] main.ERROR: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'grosse_filter.source_id' in 'on clause', query was: INSERT INTO qz8v_search_tmp_5fd7a222274fb4_02852627 SELECT main_select.entity_id, SUM(score) AS relevance FROM (SELECT DISTINCT search_index.entity_id, (((0) + (0) + (0) + (0)) 1) AS score FROM qz8v_catalog_product_index_eav AS search_index INNER JOIN qz8v_cataloginventory_stock_status AS stock_index ON stock_index.product_id = search_index.entity_id AND stock_index.website_id = 0 AND stock_index.stock_status = 1 AND stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=188 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS grosse_filter ON search_index.entity_id = grosse_filter.entity_id AND grosse_filter.attribute_id = 188 AND grosse_filter.store_id = 1 INNER JOIN qz8v_cataloginventory_stock_status AS grosse_filter_stock_index ON grosse_filter_stock_index.product_id = grosse_filter.source_id AND grosse_filter_stock_index.website_id = 0 AND grosse_filter_stock_index.stock_status = 1 AND grosse_filter_stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=99 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS visibility_filter ON search_index.entity_id = visibility_filter.entity_id AND visibility_filter.attribute_id = 99 AND visibility_filter.value in ('2', '4') AND visibility_filter.store_id = '1' INNER JOIN qz8v_catalog_category_product_index_store1 AS category_ids_index ON search_index.entity_id = category_ids_index.product_id AND category_ids_index.store_id = '1' INNER JOIN qz8v_catalog_product_index_price AS price_index ON search_index.entity_id = price_index.entity_id AND price_index.website_id = '1' WHERE (search_index.store_id = '1') AND (search_index.attribute_id = 188) AND (search_index.value in ('5538')) AND (category_ids_index.category_id in ('8')) AND (price_index.min_price <= '999.999' AND price_index.customer_group_id = 0) AND (grosse_filter.value IN ('5538'))) AS main_select GROUP BY entity_id ORDER BY relevance DESC, entity_id DESC LIMIT 100000 [] [] [2020-12-14 17:37:23] main.ERROR: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'stoff_filter.source_id' in 'on clause', query was: INSERT INTO qz8v_search_tmp_5fd7a2d3e0d369_76651047 SELECT main_select.entity_id, SUM(score) AS relevance FROM (SELECT DISTINCT search_index.entity_id, (((0) + (0) + (0)) * 1) AS score FROM qz8v_catalog_product_index_eav AS search_index INNER JOIN qz8v_cataloginventory_stock_status AS stock_index ON stock_index.product_id = search_index.entity_id AND stock_index.website_id = 0 AND stock_index.stock_status = 1 AND stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=169 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS stoff_filter ON search_index.entity_id = stoff_filter.entity_id AND stoff_filter.attribute_id = 169 AND stoff_filter.store_id = 1 INNER JOIN qz8v_cataloginventory_stock_status AS stoff_filter_stock_index ON stoff_filter_stock_index.product_id = stoff_filter.source_id AND stoff_filter_stock_index.website_id = 0 AND stoff_filter_stock_index.stock_status = 1 AND stoff_filter_stock_index.stock_id = 1 INNER JOIN (SELECT entity_id, store_id, attribute_id, value FROM qz8v_catalog_product_index_eav WHERE attribute_id=99 AND store_id=1 GROUP BY CONCAT(entity_id," ",attribute_id," ",store_id," ",value) ) AS visibility_filter ON search_index.entity_id = visibility_filter.entity_id AND visibility_filter.attribute_id = 99 AND visibility_filter.value in ('2', '4') AND visibility_filter.store_id = '1' INNER JOIN qz8v_catalog_category_product_index_store1 AS category_ids_index ON search_index.entity_id = category_ids_index.product_id AND category_ids_index.store_id = '1' WHERE (search_index.store_id = '1') AND (search_index.attribute_id = 169) AND (search_index.value in ('5452')) AND (category_ids_index.category_id in ('8')) AND (stoff_filter.value IN ('5452'))) AS main_select GROUP BY entity_id ORDER BY relevance DESC, entity_id DESC LIMIT 100000 [] []

I am facing same problem when I applying multiple filter in catalog page and I have used amasty shopby extension.

[jamie-selesti]

I am facing same problem when I applying multiple filter in catalog page and I have used amasty shopby extension.

We have the exact same issue now too and also have Amasty Shopby extension installed

[bampisSyk]

The issue persists in 2.3.6p1 , and the patch doesnt seems to fully resolves it

[FrancescoTerreni]

Is this error also present in magento version 2.4.x? Thank you.

[LIXMLN]

Is this error also present in magento version 2.4.x? Thank you.

Magento 2.4.x only support elasticsearch

[bampisSyk]

The best way to resolve the issue is to switch to ElasticSearch

[DigitalDomainIE]

I was getting this issue happening on Magento v2.3.5-p2 and thought it was either Cron or MySQL related which it appears to be but it is caused by Layered Navigation on M2.

You need to go to "Admin/Stores/Attributes/Product" and change any Product Attribute that is NOT being used on the website from "Filterable (with Results)" to "No" and reindex the site, clear cache and you'll see your server going WAY faster and no more "search_tmp_53236464" etc. queries in the SQL Process list on your server, your CPU usage will go WAY down too.

This issue is really noticeable on a clothing store with 1,400 Configurable Products made up of 54,000 Simple Products because of the amount of different product attributes associated with clothing products; Size, Colour, Fitting, Length, Waist Size etc... So you'll probably only notice it on a site with lots of different product attributes.

We thought it was the Cron, then MySQL, then eventually came across this post and realised the issue was Layered Navigation and it's not a bug, as such, it's just settings that need changing for unused attributes. No more "Gateway Timeouts". Problem solved.

[marouan2020]

Any progress on this?