search

woocommerce / action-scheduler

A scalable, traceable job queue for background processing large queues of tasks in WordPress. Specifically designed for distribution in WordPress plugins (and themes) - no server access required.
https://actionscheduler.org
GNU General Public License v3.0
636 stars 116 forks source link

Actions Scheduler run amok - 1.3 million scheduled actions, more every few seconds #517

Closed wowpages closed 4 years ago

wowpages commented 4 years ago

The Scheduled Actions on our new Woocommerce site has over 1.3 million entries and more keep getting created every few seconds. I don't know what is causing this or how to stop it, and I think it's slowing down the site. The table wp_actionscheduler_actions is 363 MB and the wp_actionscheduler_logs is 253 MB. I'm not sure what it's doing, but this doesn't seem normal, does it?

Screen Shot 2020-04-06 at 1 35 36 PM Screen Shot 2020-04-06 at 1 34 53 PM
rrennick commented 4 years ago

@wowpages check to see if you have the issue described in https://github.com/woocommerce/action-scheduler/issues/437#issuecomment-605999111 .

Edit to add: If not, how many orders are there in the site?

wowpages commented 4 years ago

The wp_actionscheduler_xxx tables are there. Was that the other issue? There are 1,261 orders total, 1,019 completed and 198 processing. I haven't seen any fatal errors. I'm the site dev, not a plugin dev btw.

rrennick commented 4 years ago

Confirming that sql_mode=STRICT_TRANS_TABLES,NO_ZERO_DATE produces an error on the default dates

@wowpages I was referring to the specific comment I linked to. The other condition is MySQL 5.7 (in the previous comment).

rrennick commented 4 years ago

@wowpages Also, you can install and activate https://github.com/woocommerce/action-scheduler-disable-default-runner to prevent AS from running actions (including the migration) while you investigate.

wowpages commented 4 years ago

MySQL version is 5.6.39. I installed the plugin you suggested - thanks! I don't know if WooCommerce is calling Action Scheduler or if it's one of the WC plugins. I'll keep an eye on things and see if anything goes awry by disabling AS with the plugin. Thanks for your help!

rrennick commented 4 years ago

@wowpages Thanks for the follow up.

thenbrent commented 4 years ago

@rrennick is this likely to be caused by the same issues reported in #516?

jessepearson commented 4 years ago

@wowpages Any chance you have WP Rocket installed?

wowpages commented 4 years ago

Thanks, but I do not have WP Rocket installed.

jessepearson commented 4 years ago

@wowpages Thanks for the confirmation, another dev team is looking into something that's happening in another plugin and I was just seeing if there was a common thread.

sixsixninenine commented 4 years ago

I'm seeing the same behaviour although not in the same volume - hundreds of thousands of entries with 'wc-admin_import_orders'. Using up server resources as more are added constantly.

There are only 1730 historical orders in total. I don't have WP Rocket installed.

I have disabled the default runner for now via the plugin linked above. I can provide a plugins list if that helps to identify a common cause.

rrennick commented 4 years ago

@wowpages @sixsixninenine If you have access to your database, could you look at your wp_actionscheduler_actions table and get the lowest action_id?

sixsixninenine commented 4 years ago

7455 is the lowest action_id in that table. It's an entry for a scheduled email from a third-party plugin.

rrennick commented 4 years ago

@sixsixninenine Thanks, I should have also asked for the most recent order ID from the order listing.

sixsixninenine commented 4 years ago

No problem. Most recent order ID on the site is 5674.

The highest order number currently referenced as an argument for wc-admin_import_orders hook in the action scheduler is 3176.

rrennick commented 4 years ago

@sixsixninenine Could you provide your WooCommerce System Status report? You can remove the site URL related items.

sixsixninenine commented 4 years ago 

### WordPress Environment ###

WordPress address (URL): https://www.XXXXXXXXXXX.co.uk
Site address (URL): https://www.XXXXXXXXXXX.co.uk
WC Version: 4.0.1
REST API Version: ✔ 1.0.7
WC Blocks Version: ✔ 2.5.14
Action Scheduler Version: ✔ 3.1.4
WC Admin Version: ✔ 1.0.3
Log Directory Writable: ✔
WP Version: 5.4
WP Multisite: –
WP Memory Limit: 256 MB
WP Debug Mode: –
WP Cron: –
Language: en_US
External object cache: –

### Server Environment ###

Server Info: Apache/2.4.41 (cPanel) OpenSSL/1.1.1d mod_bwlimited/1.4 mod_wsgi/4.6.5 Python/2.6
PHP Version: 7.1.33 - We recommend using PHP version 7.2 or above for greater performance and security. How to update your PHP version
PHP Post Max Size: 8 MB
PHP Time Limit: 30
PHP Max Input Vars: 1000
cURL Version: 7.68.0
OpenSSL/1.1.1d

SUHOSIN Installed: –
MySQL Version: 5.7.29-log
Max Upload Size: 2 MB
Default Timezone is UTC: ✔
fsockopen/cURL: ✔
SoapClient: ✔
DOMDocument: ✔
GZip: ✔
Multibyte String: ✔
Remote Post: ✔
Remote Get: ✔

### Database ###

WC Database Version: 4.0.1
WC Database Prefix: wp_
Total Database Size: 719.54MB
Database Data Size: 476.84MB
Database Index Size: 242.70MB
wp_woocommerce_sessions: Data: 4.02MB + Index: 0.06MB + Engine InnoDB
wp_woocommerce_api_keys: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_woocommerce_attribute_taxonomies: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_woocommerce_downloadable_product_permissions: Data: 0.02MB + Index: 0.05MB + Engine InnoDB
wp_woocommerce_order_items: Data: 0.48MB + Index: 0.16MB + Engine InnoDB
wp_woocommerce_order_itemmeta: Data: 2.52MB + Index: 3.02MB + Engine InnoDB
wp_woocommerce_tax_rates: Data: 0.02MB + Index: 0.06MB + Engine InnoDB
wp_woocommerce_tax_rate_locations: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_woocommerce_shipping_zones: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_woocommerce_shipping_zone_locations: Data: 0.02MB + Index: 0.05MB + Engine InnoDB
wp_woocommerce_shipping_zone_methods: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_woocommerce_payment_tokens: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_woocommerce_payment_tokenmeta: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_woocommerce_log: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_actionscheduler_actions: Data: 247.51MB + Index: 94.75MB + Engine MyISAM
wp_actionscheduler_claims: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_actionscheduler_groups: Data: 0.00MB + Index: 0.01MB + Engine MyISAM
wp_actionscheduler_logs: Data: 178.00MB + Index: 120.45MB + Engine MyISAM
wp_ac_abandoned_cart_history_lite: Data: 14.47MB + Index: 0.25MB + Engine MyISAM
wp_ac_email_templates_lite: Data: 0.01MB + Index: 0.00MB + Engine MyISAM
wp_ac_guest_abandoned_cart_history_lite: Data: 0.01MB + Index: 0.00MB + Engine MyISAM
wp_ac_sent_history_lite: Data: 0.01MB + Index: 0.00MB + Engine MyISAM
wp_adtribes_my_conversions: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_commentmeta: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_comments: Data: 1.52MB + Index: 0.53MB + Engine InnoDB
wp_ewwwio_images: Data: 1.52MB + Index: 1.67MB + Engine InnoDB
wp_failed_jobs: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_hugeit_slider_slide: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_hugeit_slider_slider: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_links: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_mailchimp_carts: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_masterslider_options: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_masterslider_sliders: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_nm_personalized: Data: 0.01MB + Index: 0.00MB + Engine MyISAM
wp_options: Data: 3.14MB + Index: 1.22MB + Engine InnoDB
wp_postmeta: Data: 14.50MB + Index: 13.75MB + Engine InnoDB
wp_posts: Data: 3.52MB + Index: 0.91MB + Engine InnoDB
wp_queue: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_termmeta: Data: 0.06MB + Index: 0.03MB + Engine InnoDB
wp_terms: Data: 0.05MB + Index: 0.03MB + Engine InnoDB
wp_term_relationships: Data: 0.22MB + Index: 0.14MB + Engine InnoDB
wp_term_taxonomy: Data: 0.06MB + Index: 0.03MB + Engine InnoDB
wp_usermeta: Data: 3.52MB + Index: 4.03MB + Engine InnoDB
wp_users: Data: 0.19MB + Index: 0.17MB + Engine InnoDB
wp_w3tc_cdn_pathmap: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_w3tc_cdn_queue: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_wcpdf_invoice_number: Data: 0.01MB + Index: 0.01MB + Engine MyISAM
wp_wc_admin_notes: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_wc_admin_note_actions: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_wc_category_lookup: Data: 0.00MB + Index: 0.01MB + Engine MyISAM
wp_wc_customer_lookup: Data: 0.13MB + Index: 0.11MB + Engine MyISAM
wp_wc_download_log: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_wc_order_coupon_lookup: Data: 0.00MB + Index: 0.01MB + Engine MyISAM
wp_wc_order_product_lookup: Data: 0.63MB + Index: 0.65MB + Engine MyISAM
wp_wc_order_stats: Data: 0.10MB + Index: 0.08MB + Engine MyISAM
wp_wc_order_tax_lookup: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_wc_product_meta_lookup: Data: 0.08MB + Index: 0.09MB + Engine MyISAM
wp_wc_tax_rate_classes: Data: 0.00MB + Index: 0.01MB + Engine MyISAM
wp_wc_webhooks: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_yoast_seo_links: Data: 0.05MB + Index: 0.01MB + Engine MyISAM
wp_yoast_seo_meta: Data: 0.02MB + Index: 0.02MB + Engine MyISAM

### Post Type Counts ###

acf: 1
acf-field: 6
acf-field-group: 1
attachment: 1289
custom_css: 1
customize_changeset: 11
jcd_promo_box: 3
mc4wp-form: 2
nav_menu_item: 94
oembed_cache: 3
page: 23
post: 52
product: 844
product_variation: 352
revision: 335
shop_coupon: 38
shop_order: 1738
shop_order_refund: 14
wcccf: 3
wccpf: 2
wpcf7_contact_form: 1

### Security ###

Secure connection (HTTPS): ✔
Hide errors from visitors: ✔

### Active Plugins (32) ###

a3 Lazy Load: by a3rev Software – 2.3.2
Action Scheduler - Disable Default Queue Runner: by Prospress Inc. – 1.0.0
Advanced Custom Fields: by Elliot Condon – 5.8.9
Cart2Cart: Password Migration: by MagneticOne – 1.1
Classic Editor: by WordPress Contributors – 1.5
Honeypot for Contact Form 7: by Nocean – 1.14.1
Contact Form 7: by Takayuki Miyoshi – 5.1.7
Cookie Notice: by dFactory – 1.2.51
XXXXXXXXXXXX WooCommerce: by XXXXXXXXXXX – 1.0 – Not tested with the active version of WooCommerce
Master Slider: by averta – 3.5.8
Find Products Without Shipping Class: by Shane Gowland –
N-Media WooCommerce PPOM PRO: by Najeeb Ahmad – 16.0 – Not tested with the active version of WooCommerce
Simple Multibuy Discount: by XXXXXXXXXXXX – 1.0 – Not tested with the active version of WooCommerce
W3 Total Cache: by BoldGrid – 0.13.1
WooCommerce UPC, EAN, and ISBN: by Scott Bolinger – 0.4.0 – Not tested with the active version of WooCommerce
Product Feed PRO for WooCommerce: by AdTribes.io – 7.8.4
WOO Product Table: by Saiful Islam – 100.2 – Not tested with the active version of WooCommerce
Abandoned Cart Lite for WooCommerce: by Tyche Softwares – 5.7.2
WooCommerce Extended Coupon Features FREE: by Soft79 – 3.1.4
WooCommerce PayPal Checkout Gateway: by WooCommerce – 1.6.20 – Not tested with the active version of WooCommerce
WooCommerce Stripe Gateway: by WooCommerce – 4.3.2 – Not tested with the active version of WooCommerce
WooCommerce Google Analytics Integration: by WooCommerce – 1.4.20
WooCommerce PDF Invoices & Packing Slips: by Ewout Fernhout – 2.4.7
PPOM for WooCommerce by N-MEDIA: by Najeeb Ahmad – 20.0 – Not tested with the active version of WooCommerce
WooCommerce Wholesale Lead Capture: by Rymera Web Co – 1.11 – Not tested with the active version of WooCommerce
WooCommerce Wholesale Order Form: by Rymera Web Co – 1.11 – Not tested with the active version of WooCommerce
WooCommerce Wholesale Prices Premium: by Rymera Web Co – 1.17 – Not tested with the active version of WooCommerce
WooCommerce Wholesale Prices: by Rymera Web Co – 1.11.2
WooCommerce: by Automattic – 4.0.1
WordPress Importer: by wordpressdotorg – 0.7
Yoast SEO: by Team Yoast – 13.4.1
WP Crontrol: by John Blackbourn & crontributors – 1.7.1

### Inactive Plugins (1) ###

PPOM Addon Cart Edit 2: by Najeeb Ahmad
Randall Kam – 2.0

### Dropin Plugins (1) ###

advanced-cache.php: advanced-cache.php

### Settings ###

API Enabled: ✔
Force SSL: –
Currency: GBP (£)
Currency Position: left
Thousand Separator: ,
Decimal Separator: .
Number of Decimals: 2
Taxonomies: Product Types: external (external)
grouped (grouped)
simple (simple)
variable (variable)

Taxonomies: Product Visibility: exclude-from-catalog (exclude-from-catalog)
exclude-from-search (exclude-from-search)
featured (featured)
outofstock (outofstock)
rated-1 (rated-1)
rated-2 (rated-2)
rated-3 (rated-3)
rated-4 (rated-4)
rated-5 (rated-5)

Connected to WooCommerce.com: –

### WC Pages ###

Shop base: 8 - /shop/
Cart: 9 - /cart/
Checkout: 10 - /checkout/
My account: 11 - /my-account/
Terms and conditions: 403 - /terms-and-conditions/

### Theme ###

Name: XXXXXXXXXXXX
Version: 1.0
Author URL: http://woocommerce.com
Child Theme: ✔
Parent Theme Name: Storefront
Parent Theme Version: 2.5.5
Parent Theme Author URL: https://woocommerce.com/
WooCommerce Support: ✔

### Templates ###

Overrides: XXXXXXXXXXXX/woocommerce/global/quantity-input.php
XXXXXXXXXXXX/woocommerce/single-product/add-to-cart/variation.php

### Action Scheduler ###

Canceled: 1
Oldest: 2020-04-04 15:48:41 +0000
Newest: 2020-04-04 15:48:41 +0000

Complete: 2,909
Oldest: 2020-03-09 11:46:31 +0000
Newest: 2020-04-09 11:25:43 +0000

Pending: 56
Oldest: 2020-04-09 11:37:43 +0000
Newest: 2020-04-09 14:11:31 +0000
rrennick commented 4 years ago

@wowpages Do you have W3 Total Cache installed?

wowpages commented 4 years ago

No, don't have W3 Total Cache. The client has it on DreamHost so it uses the DreamHost caching. It does have the Proxy Cache Purge plugin.

rrennick commented 4 years ago

@sixsixninenine If you search for wc-admin_import_orders in the Tools -> Scheduled Actions screen, how many actions does it find with that hook?

rrennick commented 4 years ago

@sixsixninenine It doesn't look like you have the same issue as @wowpages

The Scheduled Actions on our new Woocommerce site has over 1.3 million entries and more keep getting created every few seconds.

vs 

Pending: 56
Oldest: 2020-04-09 11:37:43 +0000
Newest: 2020-04-09 14:11:31 +0000

That's about 1 new action every 2 to 3 minutes. I'd suggest deactivating Disable Default Queue Runner and allow the queue to run for a bit. If you see more than 10,000 actions then you could reactivate.

sixsixninenine commented 4 years ago

Before I deleted them, there were 990,000+. I will disable the Disable Default Queue Runner.

sixsixninenine commented 4 years ago
Screen Shot 2020-04-09 at 20 59 51

The number of created and completed actions has already gone over 10k. The server and website performance is suffering as before.

I'm pretty sure this is what @wowpages was reporting. The actions are getting completed (~50 pending at one time) but the rate and volume at which they are created causes issues with the server performance and capacity (DB size going from 5MB to several hundred).

wowpages commented 4 years ago

I would say that what @sixsixninenine is seeing is exactly what I was seeing.

khalilifar commented 4 years ago

I have the same issue. over 600k actions and 2 million logs before I delete the completed actions and delete the logs but they keep coming fast. I have about 20k orders and 30k customers. I had w3-total-cache before but not right now

tcwmarketing commented 4 years ago

Chiming in myself. I've also run into this issue. We had WP Rocket installed but it was deactivated prior to starting the first import. I've tried stopping, deleting the queue and data in the tables then restarting with all plugins deactivated and using theme Twenty Twenty. Even reduced the data to last 30 days (only 32 orders and 7 customers).

Oddly if I leave Skip previously imported customers and orders checked, it shows 0 orders and 107 customers for the same time frame.

rrennick commented 4 years ago

For everyone reporting on this issue - Is it WooCommerce historical data import actions that are being created in large numbers?

wowpages commented 4 years ago

In my case, the Hook was always wc-admin_import_orders - see my OP screenshot. I don't know what was generating them. I was not even aware of Actions Scheduler until I noticed that the wp_actionscheduler_actions table was so huge and growing.

tcwmarketing commented 4 years ago

@rrennick - yes, the historical data import action was the cause of the issue for my case.

sixsixninenine commented 4 years ago

@rrennick Yes, that's right. As @wowpages account above. It began after the WC update where WC-Admin was integrated in to core and began to import historical data.

rrennick commented 4 years ago

Thanks all for the feedback. This looks like it might be an issue with the analytics dashboard vs Action Scheduler. I also work on that project so I will investigate further to see if I can reproduce the issue.

sixsixninenine commented 4 years ago

I have re-enabled the action scheduler (disabled the 'action scheduler disable default runner' plugin suggested above for debugging) and _dis_abled Woocommerce Admin with the following, for now:

https://gist.github.com/joshuatf/006111b6f43c3ab3d2cad39e3443822b

(Above link taken from a support topic for Woocommere regarding V4 and the new dashboard)

This means I can still have the action scheduler running, without being flooded with actions while this issue is being looked in to.

rrennick commented 4 years ago

We will continue the investigation into this issue on https://github.com/woocommerce/woocommerce-admin/issues/4255.