search

johngodley / search-regex

Search Regex adds a powerful set of search and replace functions to WordPress that go beyond the standard searching capabilities, with full regex support.
https://searchregex.com
GNU General Public License v3.0
32 stars 8 forks source link

REGEX keeps crashing server #76

Closed samjco closed 3 years ago

samjco commented 3 years ago

every time i run the plugin to clean up 37k posts, with all plugins disabled, it crashes my 8-core/32GB server and produces this error: SyntaxError: Unexpected token < in JSON at position 0

Unexpected token < in JSON at position 0 (SyntaxError)

And this error: The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.

https://cdn.livechat-files.com/api/file/lc/att/4817491/1ae3c2c5f7677f93d0e884e76f23eb7e/2021-01-12_16-36-54.png

johngodley commented 3 years ago

Crash the entire server and it has to be restarted? Or just return an error?

You will need to look at your server and see what the error is. For example, is it running out of memory, is it timing out etc.

samjco commented 3 years ago

Plugin crashing the whole server.. Then server seems to reboot on it own. Plugin is making 91 calls.. I only have the plugin running as the only plugin I am trying run this : <p><span(\s\w+?[^=]*?="[^"]*?")*?\s+?class="(\S+?\s)*?connect-button(\s\S+?)*?".*?>(.*?)<\/span></p> to remove html off each of the posts in post table

johngodley commented 3 years ago

Your screenshot is too small to really see anything. The arrow appears to be pointing at admin-ajax.php, which the plugin does not use.

It should not be possible for anything to crash your server, and it would imply unrelated issues with your configuration.

It's entirely possible that your regular expression is exceeding the PHP memory limit, and that is something you will know by looking in your server logs.

samjco commented 3 years ago

I boost up the PHP memory limit to 4GB and it still crashes

https://cdn.livechat-files.com/api/file/lc/att/4817491/1ae3c2c5f7677f93d0e884e76f23eb7e/2021-01-12_16-36-54.png https://cdn.livechat-files.com/api/file/lc/att/4817491/ccdedba2ded079249b9af804cde0f8b7/2021-01-12_16-48-47.png

image

If i do it singularly (1 by 1)it works

johngodley commented 3 years ago

Ok, as I said, the plugin doesn't use admin-ajax.php so that is not related.

The second screenshot seems to be an issue with your REST API. That isn't connected.

As I mentioned above, you should look at your server error logs and see what is happening.

If the regular expression is running out of memory then you will need to change the expression, or do fewer replacements at once. There's not much the plugin can do about this.

samjco commented 3 years ago

Ok, as I said, the plugin doesn't use admin-ajax.php so that is not related.

The second screenshot seems to be an issue with your REST API. That isn't connected.

As I mentioned above, you should look at your server error logs and see what is happening.

If the regular expression is running out of memory then you will need to change the expression, or do fewer replacements at once. There's not much the plugin can do about this.

Its not the ajax... the plugin is still running 91 calls. I've disabled everything

johngodley commented 3 years ago

Its not the ajax... the plugin is still running 91 calls. I've disabled everything

The plugin will make as many calls as is required to perform the replace. 91 calls will not crash your server.

samjco commented 3 years ago

@johngodley So I've had tried everything I can possibly think of....But it keeps cause my server to time out or crash. I get a JSON syntax error. The crazy part is that I know that my RegEx works because when I click SEARCH, within the plugin it pulls out the correct results..... But when I click Replace ALL... that when it crashes.

image

Then you have this strange thing: Process show 60% done but no items affected out of 37K posts.. image

johngodley commented 3 years ago

But it keeps cause my server to time out or crash.

Which one is it? You've said several different things, and shown several different problems, but they're not consistent and I really don't know what the situation is and what is or isn't working.

I will restate the above which is that if a plugin can cause your server to reboot then there is something wrong with your server - nothing should be able to cause it to reboot. The information you've given is a bit contradictory.

The crazy part is that I know that my RegEx works

I'm not saying that your regular expression doesn't work, just that it may be causing excessive memory usage.

Process show 60% done but no items affected out of 37K posts

Sure. Maybe the first 60% of posts don't match.

Why are there big green buttons on your search form? That is not part of the plugin.

samjco commented 3 years ago

OK.. I finally got it working... Not sure which fix it.. But it is definitely something worth looking into. I change my regex to capture from starting point to endpoint. Simplifying the regex a bit might have worked... BUT I also remove Cloudflare (turning OFF the cloud icon in the DNS) and using a server SSL let encrypt... I've also changed "REMOVE" to "SINGLE" and in the single field I just add a space (in order to get the Repace All button to enable)

<p><span class="connect-button[\s\S]*?/a></p>

image

My Regexr.com example: https://regexr.com/5k79h

BTW, you should consider adding the link to regexr.com in your plugin

samjco commented 3 years ago

@johngodley you should consider connecting to replacing inside the DB as an option instead of entirely working through JSON REST.

samjco commented 3 years ago

@johngodley OK. here is my final verdict... There is a bug somewhere in the plugin when dealing with thousands of posts and certain Expressions.

Which causes a 502 Bad Gateway Error... This is the error that was meant by crashing my server OR timing out the server.

This error happens when something is not matched Over and Over within the search process...

As mentioned... I simplify my expression a bit to: <p><span class="connect-button[\s\S]*?/a></p> which worked! But then when I tried the same code changing just the class name to optin-button <p><span class="optin-button[\s\S]*?/a></p> I know that the expression is simple and correct because when I run a search: image

The problem happened again... So again my verdict is that when you have 37k or more posts and the processor doesn't find a match readily, this causes the server to produce a 504 error, I think to do time-to-live which I've set to 3000 seconds on my server...

My server has plenty of RAM and Cores I've given 4GB to PHP and Mysql alone. plenty of time for execution... so there should be no issues.

johngodley commented 3 years ago

regexr.com in your plugin

There is a link to regex101.com in the documentation.

you should consider connecting to replacing inside the DB as an option instead of entirely working through JSON REST

That is not possible.

The plugin uses PHP to perform regular expressions and has no control over how that works. As I've mentioned several times, it is possible to construct a regular expression that causes issues.

samjco commented 3 years ago

This issue is still unresolved. Why the close?

johngodley commented 3 years ago

If you have solved the problem by simplifying your regular expression then that is the solution - you are in control of the expression.