query INSERT INTO “search_tmp…” never ends after mass attribute update

rsulym commented 6 years ago

Preconditions

Magento 2.2.1, php-fpm 7.1, MariaDB 10.3.5

Steps to reproduce

Perform mass attribute values update (~30000 items)

Expected result

1) Everything works fine

Actual result

1) re-index process finished ok, but: 2) A lot of queries INSERT INTO “search_tmp…” started, until server reaches pm.max_children parameter value and PHP-fpm die, and they constantly resurrecting even after mysql restart and process kill.

jaromirmuller commented 5 years ago

I received a patch from Magento team for this issue , we are going to test it on our staging.

@mertgokceimam I'm interested in patch. Can you share it?

mertgokceimam commented 5 years ago

Did you configure it properly? It must use their template, then all data is via api. Also can you share that patch from magento here?

The issue is mainly related on slow queries within layered navigation. The only way Klevu can help you is if you allow Klevu to take over your categories and layered navigation i assume.

We need to test the patch before sharing.

R-K-Shrivastaw commented 5 years ago

I am also facing the same problem with the Magento 2.3.0 INSERT INTOsearch_tmp_5d5af23c3c8911_81047940SELECTmain_select.entity_id, SUM(score) ASrelevanceFROM (SELECT DISTINCTsearch_index.entity_id, (((0)) * 1) ASscoreFROMcatalog_product_index_eavASsearch_index INNER JOINcatalog_product_entityASproductON product.entity_id = search_index.entity_id INNER JOINinventory_stock_1ASstock_indexON stock_index.sku = product.sku INNER JOINcatalog_category_product_index_store1AScategory_ids_indexON search_index.entity_id = category_ids_index.product_id AND category_ids_index.store_id = '1' WHERE (search_index.store_id = '1') AND (category_ids_index.category_id IN (75&&SlEEp(3)))) ASmain_selectGROUP BYentity_idORDER BYrelevanceDESC,entity_idDESC LIMIT 10000

I am wander that on a serious note, no satisfactory answer found till date. Magento Team please release a patch for the CE 2.3.0

gondas33 commented 5 years ago

We're having the same issue with never ending INSERT QUERIES and CPU USAGE to 200% by mysqld. In order to fix it temporally, we have to either kill the query or just restart the process. Site's TTFB increases by alot for any page and then crashes(every 18 hours or so). Running on 2.3.2.

euriskotech commented 5 years ago

I'm having the same issue on a newly migrated Magento 2.3.2 install. I had everything ready for go live this week and then the database started maxing out the CPU, finally traced it to the INSERT INTO “search_tmp…” issue and reading all your comments have decided to hold off on go live and leave the Magento 1.x site running.

I guess there isn't any update on this issue being fixed by Magento?

s00071609 commented 5 years ago

Any solution found so far? Version 2.3.1 & 2.3.2 seem to have this issue. The serach_temp_xxxx query never completes.

pradeepnew commented 5 years ago

I am using magento 2.3.2 and I have removed query in mysql using kill process but after some time its showing again query search_tmp_xxxx. Please update any solution for this

Adel-Magebinary commented 5 years ago

Looks like this has something to do with price index running.

1587 192.168.130.76:44296 Query 259 Sending data INSERT INTO catalog_product_index_price (entity_id, customer_group_id, website_id

It some locks things up.

After killing the process from the mysql processlist, it releases the lock.

This issue happens at 2.3.2.

s00071609 commented 5 years ago

Hi, I think its being caused due to search_tmp table being stuck. Rest of the pending or missed issue is due to this as this process never completes. Some in the thread has suggested that its due to Search Bots, like google bot crawling the layered navigation with urls ............com?color=red which never end and runs in loop. I have blocked bots from crawling these pages, via robot.txt file. I did this yesterday and so far i haven't had any issue. I cant say 100% but give it a try. Google doe snot need to crawl these paginated canonical pages, so you can DISALLOW in robot.txt file. Please do confirm if it solves the issue or makes it better.

Its a bug in magento 2.3 and there is no doubt about it. As it can take long time for magento to fix anything, if above helps then at least you can run your store.

gondas33 commented 5 years ago

@s00071609 which pages exactly have you disallowed? Thanks!

s00071609 commented 5 years ago

I have put these in robot.txt,

It will disallow all the urls created by selecting items on layered navigation, eg colour, size, brand etc.

Adel-Magebinary commented 5 years ago

Changing MySQL transaction isolation to READ-COMMITTED does fix the issue.

transaction-isolation = READ-COMMITTED

euriskotech commented 4 years ago

Does anyone know of any progress regarding this issue, I'm holding back a M1 migration to M2 because of this issue during testing, but under pressure from the client to migrate. Thanks

mertgokceimam commented 4 years ago

@euriskotech Magento team supplied us a patch. If you a Magento EE client , ask them the following patch : MDVA-20708_EE_2.2.9_v1.composer.patch

euriskotech commented 4 years ago

@euriskotech Magento team supplied us a patch. If you a Magento EE client , ask them the following patch : MDVA-20708_EE_2.2.9_v1.composer.patch

@mertgokceimam I'm running community, it's about time they released this into community edition

gondas33 commented 4 years ago

500% cpu usage by mysql is really something I don't like seeing when I check the resources of my server. Any public fix yet?

euriskotech commented 4 years ago

@an4rei Just noticed that 2.3.3 has been released, scanning through release notes at the moment to see if issue addressed.

gondas33 commented 4 years ago

@euriskotech doesn't look like they did anything regarding this issue

euriskotech commented 4 years ago

How can they provide a patch for EE and still be ignoring the issue in the latest 2.3.3 CE version? This makes no sense.

vadim-shulyak commented 4 years ago

Probably it would be useful for someone who is still fighting with the issue. I've applied it as a hotfix until the patch will be available. Applied for Magento 2.2.3 CE

I've made the database copy of the production, and then manually performed execution of the problematic query without INSERT INTO "search_tmp..." (just took the SELECT part and try to execute it). In my case, it takes me a few minutes. So, I removed filter by filter and found the weak spot, in my case, it was INNER JOIN for visibility_filter (since the catalog_product_index_eav table has 11 millions records). So, I've tried to optimize the query, thanks to @shiftone242, I used your solution but for a different place.

vendor/magento/module-catalog-search/Model/Search/FilterMapper/VisibilityFilter.php:

add after: $mainTableAlias = $this->extractTableAliasFromSelect($select);

    $optimizedVisibilityQuery = new \Zend_Db_Expr('(SELECT `entity_id`, `store_id`, `attribute_id`, `value` FROM '
        . $this->resourceConnection->getTableName('catalog_product_index_eav')
        . sprintf(' WHERE `attribute_id`=%s AND `store_id`=%s GROUP BY CONCAT(`entity_id`," ",`attribute_id`," ",`store_id`," ",`value`) )', $this->getVisibilityAttributeId(), $this->storeManager->getStore()->getId())
    );

and use: ['visibility_filter' => $optimizedVisibilityQuery],

instead of: ['visibility_filter' => $this->resourceConnection->getTableName('catalog_product_index_eav')],

For me, the issue is gone. I hope this could help with your cases as well.

anatolij-p commented 4 years ago

@vadim-shulyak Looks like that might actually fixed the issue

LIXMLN commented 4 years ago

@vadim-shulyak Your fix doesn't work with Magento 2.3.3 CE. Search doesn't show any results

jorgb90 commented 4 years ago

@LIXMLN Its not only add after but also replace as far as I can see and in 2.3.3 there are two occurrences for $mainTableAlias = $this->extractTableAliasFromSelect($select);

I don't know if that helps.

LIXMLN commented 4 years ago

@jorgb90 I removed the character from the last line (`). I had mistaken it for a dust on the screen :' ) For now it seems to work. Thanks

LIXMLN commented 4 years ago

@vadim-shulyak the problem persists...

mertgokceimam commented 4 years ago

@vadim-shulyak the problem persists...

| 300135 | mg2_mage476 | localhost | mg2_mage476 | Query | 5901 | Creating sort index | INSERT INTO mgdb_search_tmp_5da86051c8f898_79461193 SELECT main_select.entity_id, SUM(score) A | 0 | 14155 | | 300353 | mg2_mage476 | localhost | mg2_mage476 | Query | 5500 | Creating sort index | INSERT INTO mgdb_search_tmp_5da861e288d5d9_60328026 SELECT main_select.entity_id, SUM(score) A | 0 | 14155 | | 300549 | mg2_mage476 | localhost | mg2_mage476 | Query | 5131 | Creating sort index | INSERT INTO mgdb_search_tmp_5da86353dbcbf8_15765315 SELECT main_select.entity_id, SUM(score) A | 0 | 14155 |

Make sure to upgrade Amasty Layered Navigation ;)

sebastian-virtua commented 4 years ago

Hey guys, since the fix for CE is not there yet we've created a module that fixes this issue. It uses the idea that @vadim-shulyak posted but we overwrite one more class - in our case the custom attributes were the bigger problem. We used it on m2.2.10 and m2.3.1 and it solved our problems. If you want you can find the module here -> https://packagist.org/packages/we-are-virtua/magento2-search-fix

Beowulf891 commented 4 years ago

@vadim-shulyak Lifesaver. Worked like a charm.

erikhansen commented 4 years ago

Just applying the patch from @vadim-shulyak didn't work for me. However the extension that @sebastian-virtua posted about did fix my issue, as it edits one additional method.

I prefer to apply changes like this as a patch, as if the underlying code ever changes, the patch will fail to apply and you can remedy the issue. So I converted the we-are-virtua/magento2-search-fix extension to a patch, and you can find that patch here: https://gist.github.com/erikhansen/a813f8cf9fe8e9836eea467bdd205659

unxsist commented 4 years ago

@erikhansen Thanks! Your patch fixed the issue at once!

alexandrosk commented 4 years ago

I need the patch for enterprise! if anyone has it can he please share it with me is really important! Thanks!

erikhansen commented 4 years ago

@alexandrosk Did you try applying the patch I linked to in my previous comment? If so, did it fail to apply, or did it apply, but not work?

alexandrosk commented 4 years ago

@Erik-ID-89 Actually it does apply, I'm just thinking that enterprise uses all the row_ids instead of entity_ids but I'm trying to find a way to test it right now and get back to you soon.

erikhansen commented 4 years ago

In case it's helpful to anyone else, here is how I reproduced the issue (which was necessary to confirm my patch solved the issue):

I ran SHOW FULL PROCESSLIST; to see all of the hung INSERT INTO queries.
I inspected the queries and found the search terms that were hung (most of them appeared to be generated by bots, Bing and others)
I then searched my Nginx access logs for those queries.
I found that not all queries easily reproduced the hung queries, so I grabbed all related queries from the access logs and put them in a file, and then ran all of them against the server using siege (the Authorization header was because I was testing a stage site behind HTTP auth):

siege -c 10 --no-parser --header="Authorization:Basic eGNoYWlyOnhjaGFpcjEyMw==" -f PATH_TO_FILE_CONTAINING_URLS.txt
Of all of the ~471 queries in my file, only a few dozen of them reproduced the hung queries.

toan-tam commented 4 years ago

@erikhansen I applied the patch that you suggested on a Magento 2.3.2 instance, but I couldn't filter because of this issue

I was able to fix that issue by adding the source_id field here

I'm adding my patch here in case someone needs it. https://gist.github.com/toanthree/26273f83c1ccb786c311ae21dabf566c

erikhansen commented 4 years ago

@toanthree Thanks for sharing. I noticed the Amasty_Shopby extension in the backtrace of your first screenshot. That may have had something to do with why you had to edit your patch? Regardless, I'm glad you got it working.

rakeshsigma commented 4 years ago

Can anyone know how to apply this fix in Magento 2.1.8 EE version? Because I have checked the above patch but those files are not available in 2.1.8 EE version.

Thanks in advance.

kunzi commented 4 years ago

I too have this kind of problem. Neverending similar mysql processes are making the server super slow many times a day. I have to manually kill the processes. Tried the extension provided here, but no luck. We do have plenty of attributes in use. Just upgraded to 2.3.3. Did not have this problem with 2.2.3. Already had to disable Amasty's Improved Layered Navigation as the result was immediate rise of server resource consumption , but this still is a problem many times a day.

xeontcs commented 4 years ago

None of the solutions worked for me as well. We have one attribute called Sale (Yes/No) which cause the issue. Removing this attribute temporary solved the issue. Magento 2.2.8

aholovan commented 4 years ago

@erikhansen, Thanks for the work on this issue. We have applied the code You provided and got the issue resolved in Magento 2.3.3 (including new versions of Amasty extensions installed). With the patch queries like INSERT INTO search_tmp_... don't freeze any more. But using Mysql 5.7 to run that optimized queries we have to disable ONLY_FULL_GROUP_BY sql mode. (http://johnemb.blogspot.com/2014/09/adding-or-removing-individual-sql-modes.html)

jamie-selesti commented 4 years ago

Does anyone know if this issue has been resolved in 2.3.4?

acmetech2009 commented 4 years ago

Hi jamie-selesti Seems not, I've got 2.3.4 and the same problem for over one week since upgrade from 2.2.7. This night I'm going to apply erikhansen patch to check it.

acmetech2009 commented 4 years ago

Hi, erikhansen's path works on 2.3.4 Thank You.

teme12 commented 4 years ago

Gros mal

teme12 commented 4 years ago

Bonjour

teme12 commented 4 years ago

Good

seffparker commented 4 years ago

We noticed this error after upgrading MySQL to 5.7. Simply disabling the SQL mode ONLY_FULL_GROUP_BY via my.cnf permanently fixed the issue. I know it's a work-around. This is a bug with Magento2 and should be fixed by the devs.

zw1111 commented 4 years ago

@erikhansen I applied the patch that you suggested on a Magento 2.3.2 instance, but I couldn't filter because of this issue

I was able to fix that issue by adding the source_id field here

I'm adding my patch here in case someone needs it. https://gist.github.com/toanthree/26273f83c1ccb786c311ae21dabf566c

@toanthree Thanks! I also encountered the search filter issue and your adjustment to the @erikhansen patch worked for me!

I did not have this problem previously... I think I was using Magento OS 2.3.3, but started having the problem immediately after upgrading to Magento OS 2.3.5-p1.

trapchik commented 4 years ago

Just applying the patch from @vadim-shulyak didn't work for me. However the extension that @sebastian-virtua posted about did fix my issue, as it edits one additional method.

I prefer to apply changes like this as a patch, as if the underlying code ever changes, the patch will fail to apply and you can remedy the issue. So I converted the we-are-virtua/magento2-search-fix extension to a patch, and you can find that patch here: https://gist.github.com/erikhansen/a813f8cf9fe8e9836eea467bdd205659

anything else need to be done after applying your patch?

thanks

feavilan04 commented 4 years ago

We had same issue, we found problem on magento 2.3.4. Issue was fixed by using AWS ElasticSearch as catalog_search system https://devdocs.magento.com/guides/v2.4/config-guide/elasticsearch/configure-magento.html

magento / magento2