jeherve
commented
7 years ago Sorry for all the trouble!
We're seeing the jobs
jetpack_sync_cronandjetpack_sync_full_cronrunning, a lot. Like every minute
They do indeed run every minute by default.
what can be so important, that you must run these jobs pretty much back to back?
This synchronization allows your data to be synchronized back with WordPress.com, and consequently gives us more reliable / up to date data to be used with modules relying on WordPress.com, like Publicize, Subscriptions, Related Posts, Stats, and the site management tools on WordPress.com.
If you absolutely must run these sync jobs, then they either require a redesign, or a higher running interval, or a filter, so we, at least, can change the interval ourselves.
We've made a lot of changes to Sync in the past few weeks to address issues like those you've experienced, so it might be worth giving our current Alpha version a try. It's available here: https://github.com/Automattic/jetpack/archive/master-stable.zip
If you'd rather not use a development version of Jetpack on your production sites, you can use the jetpack_sync_incremental_sync_interval and jetpack_sync_full_sync_interval filters to change the frequency of the synchronization. The 2 filters are available in the current version of Jetpack (4.3.2). Here is how you could change the overall frequency to 10 minutes, for example:
function jp_support_5513_sync_schedule( $schedules ) {
if ( ! isset( $schedules['10min'] ) ) {
$schedules['10min'] = array(
'interval' => 10 * MINUTE_IN_SECONDS,
'display' => __( 'Every 10 minutes' ),
);
}
return $schedules;
}
add_filter( 'cron_schedules', 'jp_support_5513_sync_schedule' );
function __return_jp_support_5513_10_min() {
return '10min';
}
add_filter( 'jetpack_sync_incremental_sync_interval', '__return_jp_support_5513_10_min' );
add_filter( 'jetpack_sync_full_sync_interval', '__return_jp_support_5513_10_min' );If you get the chance to test the development version of Jetpack, let us know how it goes!
larssn
lezama
tillkruss
dd32
enejb
ebinnion
rmccue
earcos
Last Thursday, our entire cluster was brought to its knees, including all the Wordpress sites we host.
We identified Jetpack being indirectly responsible.
Before I get into that, a little background on the relevant part of our setup.
So we're hosted on Amazon, and we use their Elastic File System (EFS) to host our Wordpress files. This allows multiple instances to share the same Wordpress files, and we don't have to worry about instances having different files. The EFS has burstable performance, so if you need a sudden throughput of several GB/sec of data, it can do that. When it does that, you use what is called Burst Credits. For us, its pretty important that these don't run out.
We don't use WP Cron, as we require a reliable job scheduler, that runs at the right times. For this we use Cavalcade, which makes sure that the same job doesn't run on multiple instances. A job it does really well.
October 24, we had a credit balance of 32.4TB, and then something happened - not sure if Jetpack was updated, or the stars had the wrong alignment, but 3 days later our burst credits hit zero. Image: EFS dying when credits hit 0
We've since determined that Jetpack apparently have sync jobs running - nonstop, that apparently uses a lot of disk IO, and also a lot of CPU (at zero load, our instance CPUs averaged at 21%). We're seeing the jobs
jetpack_sync_cronandjetpack_sync_full_cronrunning, a lot. Like every minute... And they take more than a minute to finish, so we ended up seeing 3 of each of these jobs, per site (in our multisite-setup), running, constantly! This is crazy stuff!Quickfix
If you're reading this, and are in a similar situation, modify
sync/class.jetpack-sync-actions.php:We also changed the job running every minute, to once per hour. If it actually does anything with the above change, don't know. We don't really care at this point. Our main concern is our credits staying up. With the above change, the credits stay at a nice even balance.
We're very dependent on Jetpack, and we've always considered it a good plugin. We still do; this is IT, these things happen.
We have to ask though, what can be so important, that you must run these jobs pretty much back to back? If you absolutely must run these sync jobs, then they either require a redesign, or a higher running interval, or a filter, so we, at least, can change the interval ourselves.
Thanks for reading.
TLDR; Jetpack sync jobs + Amazon EFS + external job runner = bad.