humanmade / Cavalcade

A better wp-cron. Horizontally scalable, works perfectly with multisite.
https://engineering.hmn.md/projects/cavalcade/
Other
515 stars 46 forks source link

Possible issue with background tasks from WC Product Bundles #51

Closed xpd closed 6 years ago

xpd commented 6 years ago

Hey,

Not quite sure where to start debugging this tbh, but we've been having issues of tasks not getting cleared properly. We've also had a few times where we had to reenable plugins to make scheduled tasks run.

Feel free to push me in the right direction for getting some useful information that can be used for debugging this.

Some info from the developer of Product Bundles:

I can see that tasks are completing, which rules out a problem with

WC_PB_DB_Sync_Task_Runner::time_exceeded or WC_PB_DB_Sync_Task_Runner::memory_exceeded.

So WC_PB_DB_Sync_Task_Runner::task is returning false, which is correct.

However, there's likely a problem here: https://cl.ly/3q293d3y1s1Z

Tasks look like they are not being cleared out of the options table correctly

That's something you need to look into because the WC updater relies on the same library -- if it doesn't work correctly, you'll likely have issues with updating WooCommerce down the road.

Can you perhaps debug this on a standard / local server environment without any complex caching or task scheduling software running?

You should see this instead of the same task being processed over and over:

2017-10-31T07:48:52+00:00 INFO Sync complete.

If this is a problem that affects your live site only, it might be hard to diagnose it -- and something my support team won't be able to help with.

I am not very familiar with Cavalcade either -- if it's a caching or Cavalcade related issue, I'd have trouble figuring out the exact cause without looking at their code. Something like that could take days to debug :)

Can you share your findings after looking into it on the staging site?

Keep an eye on that log -- it should not keep running over and over with every request.

Tasks are stored in your options table -- they should be called:

wp_wc_pb_db_sync_task_runner_batch_xxx

If you have a lot of these queued, it might take some time until they get processed, but eventually you should see a

rmccue commented 6 years ago

Apologies for the delayed reply on this one.

For debugging this sort of stuff, you're best off looking at the wp_cavalcade_jobs and wp_cavalcade_logs tables directly. These record the scheduled and completed wp-cron jobs, as well as logging the return status of each run. Things to watch out for here are jobs marked failed (check the logs table to find out how it failed), or jobs marked running when they're actually not.

I'm not sure how the background process code in the Product Bundles code works, but as long as it's using the wp-cron API, it should all Just Work. There's nothing super complex about how Cavalcade works; the basic flow is this:

  1. When you call wp_schedule_event() or wp_schedule_single_event(), Cavalcade hijacks this and writes to the jobs table instead of the cron option.
  2. The Cavalcade Runner daemon monitors the jobs table and upon seeing any new, waiting† jobs, it spawns a worker
  3. The worker is spawned by running wp cavalcade run <id> --url=<site-url>
  4. The wp cavalcade run command reads the job and runs the appropriate hook, just like wp-cron.php

Hope that helps initial debugging; happy to help further if you can get further details.

xpd commented 6 years ago

Hi,

Thanks for the response. Seems like the product bundle issue ran a bit deeper within the background worked. So that one was unrelated to Cavalcade.

I see the other one we've had some issues with failed with "Failed to shutdown worker.". Any simple way to not pause failed tasks, or possibly fix that error.