Automattic / jetpack

Security, performance, marketing, and design tools — Jetpack is made by WordPress experts to make WP sites safer and faster, and help you grow your traffic.
https://jetpack.com/
Other
1.58k stars 797 forks source link

Upgrade to 11.5 broke WAF #27209

Closed Ipstenu closed 1 year ago

Ipstenu commented 1 year ago

Impacted plugin

Protect

Steps to Reproduce

My site is set to auto update Jetpack. Today it did so at 9:58am, and immediately my site went down.

The error log was full of PHP Fatal error: require(): Failed opening required '/home/wp_w9hpj2/lezwatchtv.com/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-waf/src/../rules/allow-ip.php' (include_path='.:') in /home/wp_w9hpj2/lezwatchtv.com/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-waf/rules/rules.php on line 2', referer: https://lezwatchtv.com/

I went into that file and found these:

//if ( require('/home/wp_w9hpj2/lezwatchtv.com/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-waf/src/../rules/allow-ip.php') ) { return; } //if ( require('/home/wp_w9hpj2/lezwatchtv.com/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-waf/src/../rules/block-ip.php') ) { return $waf->block('block', -1, 'ip block list'); }

So I commented them out and the site came back.

A clear and concise description of what you expected to happen.

An upgrade shouldn't break things.

What actually happened

Upgrade caused a white screen of death.

Browser

Google Chrome/Chromium, Mozilla Firefox, Apple Safari

Other information

No response

Platform (Simple, Atomic, or both?)

Simple

Reproducibility

Consistent

Severity

Some (< 50%)

Available workarounds?

Yes, easy to implement

Workaround details

Comment out the top lines:

//if ( require('/home/wp_w9hpj2/lezwatchtv.com/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-waf/src/../rules/allow-ip.php') ) { return; } //if ( require('/home/wp_w9hpj2/lezwatchtv.com/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-waf/src/../rules/block-ip.php') ) { return $waf->block('block', -1, 'ip block list'); }

Prof9 commented 1 year ago

Same issue here. Our WordPress install became completely non-functional following an auto-update from Jetpack 11.4 to 11.5.

Commenting out those two lines fixed the issue here also.

EDIT: It looks like WAF writes its lists of allowed IPs and blocked IPs into allow-list.php and block-list.php respectively. Performing an auto-update removes those two files, causing the includes in rules.php to break.

I found that you can re-generate the allow-list.php and block-list.php files by disabling and re-enabling the "Protect your site with Jetpack's Web Application Firewall" option in Jetpack settings. Otherwise, the IP lists probably won't be applied if you only comment out the lines in rules.php.

samiff commented 1 year ago

Thanks for the detailed reports! Our team is investigating this at the moment.

Related references:

anomiex commented 1 year ago

I found that you can re-generate the allow-list.php and block-list.php files by disabling and re-enabling the "Protect your site with Jetpack's Web Application Firewall" option in Jetpack settings. Otherwise, the IP lists probably won't be applied if you only comment out the lines in rules.php.

Also, if your site is already down due to this issue, you could try deleting that rules.php file which should (temporarily) let you back in. You can then toggle the option in the settings to regenerate all three files.

Or, if you have WP CLI access, you can do wp jetpack module deactivate waf and then wp jetpack module activate waf to toggle it without going through the web interface.

Ipstenu commented 1 year ago

EDIT: It looks like WAF writes its lists of allowed IPs and blocked IPs into allow-list.php and block-list.php respectively. Performing an auto-update removes those two files, causing the includes in rules.php to break.

Wait, it's writing to files in the Jetpack plugin folder? Well there's your problem. Why would it not write those to the wp-content/jetpack-waf folder? Updates will always delete everything in wp-content/plugins/jetpack -- and if we're moving towards automated updates for all things, this is gonna hammer a bunch of problems into people's sites.

Ipstenu commented 1 year ago

FYI this happened again this morning/overnight with the update to 6.1 WP. I left WAF on one site (multisite), and the rules again broke. I'm just leaving this off until y'all get 11.5.1 out at this point.

anomiex commented 1 year ago

11.5.1 is now out.

Ipstenu commented 1 year ago

Any word on moving the rules file out of the main plugin folder and into wp-content/jetpack-waf instead? Or, better, use it per-site (or heck, make WAF a network only tool that saves per-site since some of us use multisite ;) )

kraftbj commented 1 year ago

@Ipstenu The WAF team will take a fresh swing at the rule saving. The fix for 11.5.1 was meant to be a bandaid to prevent the issue pending a larger rewrite.

paulschreiber commented 1 year ago

The current location breaks checksum verification (wp plugin verify-checksum jetpack). It's creating a lot of noise — I have this checked hourly.

miguelxpn commented 1 year ago

@paulschreiber We have recently changed the location those files were changed (#28049), it should be included in the next release. I apologize for the inconvenience.