Logs flooded with coupon-sync

[f3bruary]

Both in standard as debug logs I see it getting flooded with coupon syncs. It just keeps doing this and repeating. It's not syncing orders or products either. And I have no 'sync' button on the sync tab.... No errors in WP debug logs, or in developer console...

NOTICE coupon_sync.success :: update promo rule :: #wmdc06

[f3bruary]

Some new errors appeared:

2018-05-01T22:22:47+00:00 ERROR order_submit.error :: update promo rule :: wmdf127dc :: A promo rule for store '97404026dc9791070c064fb7e2b90178' with the id '21416' already exists. on 1315 in /home/user/public_html/wp-content/plugins/mailchimp-for-woocommerce/includes/api/class-mailchimp-api.php
2018-05-01T22:22:48+00:00 NOTICE coupon_sync.success :: update promo rule :: #wmdf127dc
2018-05-01T22:22:48+00:00 NOTICE coupon_sync.success :: update promo rule :: #wmd9fa89d
2018-05-01T22:22:48+00:00 ERROR order_submit.error :: update promo rule :: wmd9fa89d :: A promo code for store '97404026dc9791070c064fb7e2b90178' with the promo_rule_id '21419' and promo_code_id '21419' already exists. on 1315 in /home/user/public_html/wp-content/plugins/mailchimp-for-woocommerce/includes/api/class-mailchimp-api.php
2018-05-01T22:22:49+00:00 ERROR order_submit.error :: update promo rule :: wmdbf2729 :: A promo rule for store '97404026dc9791070c064fb7e2b90178' with the id '21420' already exists. on 1315 in /home/user/public_html/wp-content/plugins/mailchimp-for-woocommerce/includes/api/class-mailchimp-api.php
2018-05-01T22:22:49+00:00 NOTICE coupon_sync.success :: update promo rule :: #wmdbf2729
2018-05-01T22:22:50+00:00 NOTICE coupon_sync.success :: update promo rule :: #wmdc99906
2018-05-01T22:22:50+00:00 ERROR order_submit.error :: update promo rule :: wmdc99906 :: Internal Server Error :: A deep, internal error has occurred during the processing of your request. Please contact support. on 1287 in /home/user/public_html/wp-content/plugins/mailchimp-for-woocommerce/includes/api/class-mailchimp-api.php
2018-05-01T22:22:51+00:00 NOTICE coupon_sync.success :: update promo rule :: #wmd2b5b21
2018-05-01T22:22:51+00:00 ERROR order_submit.error :: update promo rule :: wmd2b5b21 :: A promo code for store '97404026dc9791070c064fb7e2b90178' with the promo_rule_id '21427' and promo_code_id '21427' already exists. on 1315 in /home/user/public_html/wp-content/plugins/mailchimp-for-woocommerce/includes/api/class-mailchimp-api.php
2018-05-01T22:22:51+00:00 NOTICE coupon_sync.success :: update promo rule :: #wmdf127dc
2018-05-01T22:22:51+00:00 NOTICE coupon_sync.success :: update promo rule :: #wmd9fa89d

[ryanhungate]

@f3bruary thanks for reporting. Looks like you've got something wrong deep in the trenches at MailChimp given the Internal Server Error. Since this is not the actual support channel - It would be best for you to reach out them within your account area so they can start a technical support ticket.

The sync goes through all of your coupons for a reason - so you can use them in MailChimp's campaign builder - as of now we don't have a setting to ignore them - but if that's something folks would like to see, we can certainly run that up the chain to get some internal feedback.

Once your coupons have sync'd - regardless of the errors you're seeing - the products will follow - and then so should the orders. If your log gets too large, you can always delete it.

Let us know what happens after a little while has passed 👍

[f3bruary]

The coupon sync just keeps repeating. I've got little over 1000 coupons, and it's syncing 2-3 every second. It's been running for a day or two now...

As for the other errors, I will reach out to mailchimp support.

[ryanhungate]

@f3bruary if it's been doing that for so long - I have a feeling you've got a bunch of these jobs in your database queue table. Can you check that real quick and see what you're looking at? There should never be more than 100 in there at any time so if you see a bunch - maybe we need to do something a little more drastic like disable the plugin -> delete those database entries, and start over.

The new release of the plugin will be doing the clean up for you - but as of right now it's still in our QA process.

[f3bruary]

Queue and failed_jobs are both empty.

[ryanhungate]

@f3bruary i'm just checking back with you on this - your coupons eventually completed and started syncing products and orders right?

[f3bruary]

I had to deactivate the plugin because I got issues again with slow speeds. It kept syncing coupons too.

I received some new order and also added some new products before I deactivated the plugin and noticed it synced those with mailchimp. Just the newly added ones, not any older orders/products.

Some good news though. I think I found the cause of the slow speeds and I think somehow it's browser related. I have to test it again, but it might have been caused by using an outdated version of chromium for Debian. The repo version was outdated and I have switched to the latest version of Google Chrome for Debian instead. I immediately noticed a speed bump and stability improvements.

I will activate the plugin again and get back to you regarding the past issues I've had.

[ryanhungate]

Hey @f3bruary that's interesting to hear - please do let us know when you've had a chance to dig deeper into that. That's not really something that we've considered to be part of the problems you were experiencing just yet, so yes, very curious to know what the outcome is. 👍

[ryanhungate]

@f3bruary just wanted to ping you again on this subject. Any news? If you can PM me your MailChimp account email - we will put that in our support queue to take a deeper dive into this particular issue you're facing. I think we may need to go that route - given how you have some 'deep internal errors' to look at.

[f3bruary]

I activated the plugin but after only a few hours my hosting company (siteground) sent me an email telling me I almost exceeded my daily cpu seconds. I haven't had time to investigate but I assume it was the mailchimp-for-woocommerce plugin that made too many requests.

I will look into it further when I can free up some time.

[ryanhungate]

@f3bruary - the term daily cpu seconds is a little foreign to me - but it sounds like you have some sort of limit on the bandwidth your store is consuming for whatever reason. That's outside the scope of this type of issue tracking - but I would certainly like to try to find out 'how' that's coming up.

Do you have any sort of inventory management system that's constantly making updates to your system? If so - those products also end up syncing over to MailChimp as well. Same thing goes for orders. Whenever an update is made, we're updating records in MailChimp too.

Have you installed the latest plugin? We're on version 2.1.7 right now as of today. Can your hosting provider give you a better idea of what was going on? If so - just PM me the details and I'll take a closer look to see if anything stands out.

[f3bruary]

The daily cpu seconds is basically max allowed cpu usage per day. Since it's shared hosting, Siteground wants to enforce a fair usage policy. So it you're using too many resources, they'll block you for the remainder of the day. THe only solution is to get a dedicated hosting account which is much more expensive.

However, I was surprised to see that I hit the resource limit again just now, while the plugin is deactivated. I still think last time was caused by the plugin though. This time I contacted Siteground and they told me a lot of bots were hitting urls on my site (over 80K requests). They added some rules in my htaccess to block them.

I'm gonna re-enable the plugin again to see if it will cause the same issue again. I'm using 2.1.7.

[ryanhungate]

@f3bruary oh - ok, that actually sheds some light on the source of the problem where it just stops the sync in it's tracks. If they're limiting the amount of time that a php process can run per day - that would absolutely be the reason things are not working for some folks - especially in the initial sync which was meant to continue processing until the records were finished.

We will think about this - it's not exactly an easy fix to implement across the board because this is a server / hosting limitation - not the plugin itself. I do however understand that the behavior of the plugin is 'causing' this type of end result - but we will have to put some extra thought into this before starting a dev cycle.

Please do let us know if the plugin actually finishes - since you did say there were some other bots contributing to the CPU issue.

[da-mask]

@ryanhungate I share @f3bruary 's situation, I'm hosted with SiteGround (on their top tier shared hosting plan, so I have nowhere to go to increase usage). I had to use WP-CLI to get this plugin to work, however had to deactivate the plugin due to consuming too much CPU time, and this is with setting the cron job to run every 30 minutes (as opposed to the recommendation of running every 1 minute). I have a small new woocommerce site with only 43 variable products, 6 orders and 0 coupons.

I'm currently being forced to look at other MailChimp plugins in order to intergrate, let me know if there is any testing I can do.

[ryanhungate]

@DanielFloris thanks for reporting back - this really shouldn't be happening especially at that size - we're going to take a closer look into the SiteGround setup and report back with the findings. I'm sure we can figure out what's going on. Stay tuned.

[da-mask]

I've reactivated it about an hour ago, I navigated to /resync=1 and resynced, all seemed to go through and CPU usage seems fine atm. Will monitor...

[da-mask]

Deactivated MC plugin again, due to high usage.

[ryanhungate]

@DanielFloris if you could provide the most common requests being sent with the high usage that would be helpful - but we are going to also be grabbing an account at siteground as well to troubleshoot soon.

[da-mask]

@ryanhungate All I can gather is that it seems to be admin-ajax, is that what you mean? I've reactivated the plugin again, and will contact siteground support to find out what they can see after its been running for a while.

[da-mask]

@ryanhungate Did you end up getting a siteground account? I was initially excited by the quick responses I was getting on this issue, but this topic seems to have gone dark...

[ryanhungate]

@DanielFloris Sorry for the delays - we plan on doing that - but with the GDPR stuff rolling out this last week it's been quite hectic. We did actually contact SiteGround - and they said that they hadn't heard of this as a problem - but of course everyone has a unique situation that needs careful attention for one reason or another. As soon as we figure out what's going on we'll let you know.

[da-mask]

@ryanhungate OK, thanks for the update. I've had to go ahead and launch a new online store with no mailchimp integration, so I'm watching this very closely.

[ryanhungate]

@DanielFloris we've just started the process over at SiteGround - when things progress we will be in touch. Can you also list the plugins you're currently using as well just in case that is something we needed to look into as well?

[da-mask]

@ryanhungate Good to know. My current list of activated plugins is: code snippets v2.10.1.1 Custom Product Tabs for WooCommerce v1.6.4 Enhanced E-commerce for Woocommerce store v2.0.3 Heartbeat Control v1.2.3 (this is what siteground support suggested I install to remedy cpu running high) HTML in Category Descriptions v 1.2.1.1 MailChimp for WooCommerce v2.1.7 (deactivated) MIMo Woocommerce Order tracking v1.0.0 Popup Maker v1.7.25 Redirection v3.2 SG Optimizer v4.0.7 Smush v2.7.9.1 TablePress v1.9 UpdraftPlus v1.14.11 WC Password Strength Settings v2.0.2 Woo Align Buttons v3.3.5 WooCommerce v3.4.2 WooCommerce Extended Coupon Features v2.6.3 Woocommerce PayPal Express Checkout Gateway v1.5.6 WooCommerce Services v1.14.1 WooCommerce Stripe Gateway v4.1.7

Seems like a lot, now that I write it down. ¯_(ツ)_/¯

[f3bruary]

Activity Log
Breadcrumb NavXT
Broken Link Checker
Buttonizer - Smart Floating Action Button
Contact Form 7
Cookie Notice
CSS Plus
Custom Product Stickers voor Woocommerce
Custom Sidebars
Google Tag Manager for Wordpress
Easy Theme and Plugin Upgrades
Error Log Monitor
Google Analytics Dashboard for WP (GADWP)
Kiyoh Customerreview
Loco Translate
Loginizer
MailChimp for WordPress
Mollie Payments for WooCommerce
My Custom Functions
Nextend Social Login
PixelYourSite PRO
WooCommerce Product Filter
Product Catalog Feed Pro by PixelYourSite
WooCommerce Pushover Integration
Regenerate Thumbnails
Search & Replace
SG Optimizer
SiteOrigin CSS
Two Factor Authentication
User Rol Editor
WC Hide Shipping Methods
WooCommerce Advanced Bulk Edit
WooCommerce Advanced Free Shipping
WooCommerce DYMO Print Product Add-on
WooCommerce DYMO Print
Woocommerce Mailchimp Discount
WooCommerce MyParcel
WooCommerce PDF Invoices & Packing Slips
WooCommerce Point of Sale
WooCommerce Product SKU Generator
WooCommerce
WooSidebars
WordPress Importeerder
Yoast SEO
WP Instagram Widget
WP Migrate DB
WP-Optimize
WP Fortune
Woocommerce Google Cloud Print
YITH WooCommerce Wishlist

[da-mask]

@ryanhungate Has this issue been dropped?

[da-mask]

@ryanhungate My marketing team is getting pretty fed up about this, is there any other testing I can do? The last message I received said you would be investigating a siteground setup to see if you could 1. replicate and 2. resolve the issue. Has anything happened on this front? I can't see any commits on 2.1.9 branch on github that seems to be working on this issue, is there anything I can do?

[khungate-zz]

@DanielFloris apologies for the delay with updates on this. We've been testing over at Siteground using their "GoGeek" subscription plan and everything is functioning as normal. We've thrown hundreds of orders at the test store and performed multiple uninstall > reinstalls and everything syncs up as expected. The biggest difference is the plugins being used with your install (also others on this thread) and ours.

Is it possible that there's a plugin in your store using just enough resources that causes the CPU limits to go overboard when the MailChimp plugin is in use? We've seen situations in the past where a 3rd party order/inventory update tools can also add to calls on a site.

We'll continue to dig deeper, but at the moment we're having trouble replicating any issue on our end.

[f3bruary]

I've mentioned this in https://github.com/mailchimp/mc-woocommerce/issues/158#issuecomment-375917883. I use a Point of Sale plugin that stores orders made in my brick and mortar shop. These do not contain an email address and cause an error. But I don't know if this also causes the other sync issues with the plugin. Also, it doesn't look like @DanielFloris is using the same plugin anyways.

[da-mask]

Thanks @ryanhungate I appreciate your work. What was the cron command you were using at siteground, and were you able to get it to function with a siteground generated staging site? Were you even using the cron job?

Without the mc plugin running, my cpu usage looks like this: But with it, it is maxed out.

[ryanhungate]

@DanielFloris thanks for the info. The CPU credit issue you're seeing is simply a matter of how the plugin works to process jobs. It's staying alive for 20 seconds to process as much as it can within that time ( that's the default ) but can actually be adjusted by using a filter

http_worker_default_time_limit

Maybe you'll need to adjust this to process something like 2 seconds instead to see if that helps your scenario. We will try this in our testing and see if we run into any issues - and will report back.

[da-mask]

@ryanhungate OK I'll do that, is that done in wp-config? I've ssh'd in and run the wp-cli command and noticed once the initial sync is done, the process continues to run with :: loop # 60 ; ::loop # 80; :: loop # 100... etc

Is that what this refers to ?

[ryanhungate]

@DanielFloris That's great that you're using the CLI version - but I it's not going to solve your CPU problem. You have limits on your server as to the amount of time you are using it - so a 20 second sleep time ( to handle queued jobs ) is the same thing as a 20 second page load - which in turn causes this CPU clock to increment in a way that is not good for your situation. I'm assuming that the CLI version is doing the same thing as the default HTTP queue correct? Did you shut off the HTTP queue by specifying the constant DISABLE_WP_HTTP_WORKER = true?

The default time limit thing i'm referring to is the amount of time ( 20 seconds ) and it can be changed with a wordpress filter function - but you could also just change 1 line of code to see if this helps you out first... in this file just change the value FROM 20 to something like 4 to make this process do less on each attempt - but we have not done enough testing on this particular method to assure you it will work perfectly... All it's doing is saying 'only work for X number of seconds before it cancels out' - but will be re-spawned again to process the jobs on another request - which kind of puts us in the same situation as before.

We've put a lot of work into the queue system to make sure that jobs get handled on high volume stores - but it seems as if this just needs to be configured further to allow for your use case as well. We will continue to talk through this on our side to see what the best approach is and get back to you.

[da-mask]

Hi @ryanhungate , Yes, I've set DISABLE_WP_HTTP_WORKER to true in wp-config, and yes its still very taxing on the CPU So, I've gone and changed the 'http_worker_default_time_limit' to 4 as you advised, and I'll monitor. Hopefully this brings the CPU load down to an acceptable level.

[ryanhungate]

@DanielFloris I know this may be a little confusing but the HTTP WORKER and the CLI WORKER are 2 options to run the queue in a different context. If you're using the HTTP WORKER you'll need to make sure that the DISABLE_WP_HTTP_WORKER is set to false, otherwise it won't process anything. Once you do that - it will use the 'http_worker_default_time_limit' value you've set.

Also - don't run both at the same time either. If you're going to set the HTTP WORKER to false, make sure you turn off your CLI process since that will only add more to the problem.

The CLI version is meant to be kept alive on purpose too - so the usage of the CPU is really not something you can fix in the plugin's current state, other than possibly changing plans with your hosting company to account for a little more usage / CPU allotments.

Again this is something that we do take seriously and will try to come up with a valid solution to work on CPU consumption at some point in a future release but at the moment there is no timeline on when we're going to start this particular development cycle. We will certainly remember to let you know when something is ready.

[da-mask]

@ryanhungate Thanks for clarifying that for me. I was a bit confused. I've now removed the cron job (which initiates the CLI Worker?) and set DISABLE_WP_HTTP_WORKER to false in wp-config, which now as a time limit of 4 set. Will get back with results.

As I've said, as far as shared plans at siteground go, I'm already on the highest tier, and the price jump to cloud hosting removes it as an option. I had hoped/assumed MailChimp integration would have been achievable on shared hosting.

[da-mask]

Good news, after operating for a few hours with the above settings, this: So as long as the shop stays in sync, I might be good. I'll look into using a filter to change the time limit so it doesn't get overridden when a new version of mc-woocommerce gets released.

[ryanhungate]

@DanielFloris that's good to hear - i'm curious to see if this 4 second version you've configured is good enough to handle your abandoned cart activity too - I guess it's really going to depend on the activity of the site. A heavy traffic site will need these longer processes to handle everything - but maybe for most stores a smaller default is fine. Can you please let us know if you run into any snags at all in the next few days? Thanks for helping out - very insightful.

As for the filter - yes that's the right thing to do - but we may also provide a few queue settings in a later release if this seems to help because it's really important for folks to be able to configure their environments accordingly.

[da-mask]

@ryanhungate I might play with the time limit, see if I can push it a bit higher and still be OK. We do have a low traffic site, but we are working on that so will hopefully need more processing time down the track.

And yes, the ability to adjust the time limit within the plugin interface would be great.

[da-mask]

@ryanhungate I've updated to the latest mc-woocommerce plugin, and I'm using this snippet to override

function sg_http_worker_default_time_limit(){
  return 10;
}
add_filter( 'http_worker_default_time_limit', 'sg_http_worker_default_time_limit');

Does that look right? Sorry to ask, this is the first time I've created my own snippet and just wanted to make sure. And yes, I'm using a value of 10 now, and it seems to be causing no issues.

[ryanhungate]

Hey @DanielFloris sorry for the delay - and thank you for the update. Yes that snipped looks fine. We are also going to continue looking into a better solution for these types of server environments. If anything gets updated, you'll clearly be one of the first people we notify.

Since this is currently working for you, we can close this out, but will be more than happy to re-open and discuss if you needed more help.

[f3bruary]

I got myself a higher performing VPS after I noticed Siteground increased their prices (again). €30/m for that crappy hosting is not worth it to me. Once I got everything migrated and running, I'll test the plugin again.