search

humanmade / Cavalcade

A better wp-cron. Horizontally scalable, works perfectly with multisite.
https://engineering.hmn.md/projects/cavalcade/
Other
517 stars 46 forks source link

Runner crashes #6

Closed Djelibeybi closed 4 years ago

Djelibeybi commented 8 years ago

I'm trying to switch to Cavalcade for our multisite, but every time I start runner, it crashes after running the first set of jobs with the following error:

Sep 15 14:28:18 server php: PHP Notice:  Undefined offset: 0 in /opt/Cavalcade-Runner/lib/Runner.php on line 211
Sep 15 14:28:18 server php: PHP Fatal error:  Call to a member function is_done() on a non-object in /opt/Cavalcade-Runner/lib/Runner.php on line 212

I've been running it in the foreground, but it doesn't really output anything more useful:

 cavalcade
[27] Running wp cavalcade run 27 --url='f.q.d.n/site1/' (check_plugin_updates-user-role-editor-pro a:0:{})
[27] Started worker
[27] out:
[27] err:
[27] ret: 0
[28] Running wp cavalcade run 28 --url='f.q.d.n/site2/' (check_plugin_updates-user-role-editor-pro a:0:{})
[28] Started worker
[29] Running wp cavalcade run 29 --url='f.q.d.n/site2/' (wp_scheduled_delete a:0:{})
[29] Started worker

Actual hostnames and URLs replaced to protect the innocent, etc. Meanwhile, I'm also looking into why the stupid User Role Editor Pro checks for plugin updates on every single site.

Djelibeybi commented 8 years ago

No thoughts on this?

joehoyle commented 8 years ago

ping @rmccue

rmccue commented 8 years ago

Apologies for the late reply here; totally missed it before!

It looks like somehow $pipes is getting changed between when it's set and when we loop. The only thing I can think of here is stream_select changing it somehow...

Can you dump $pipes and $this->workers just before the loop on L210 and see what's up?

Djelibeybi commented 8 years ago

Yep, something is definitely breaking somewhere. Here is the complete output from starting to the runner crashing. Same error as above in /var/log/messages:

I added the following to lib/Runner.php:

                var_dump($pipes);
                var_dump($this->workers);

And it produced the following output.

-bash-4.1$ bin/cavalcade /var/www/prod/
[7] Running wp cavalcade run 7 --url='joy.org.au/' (check_plugin_updates-user-role-editor-pro a:0:{})
[7] Started worker
[8] Running wp cavalcade run 8 --url='joy.org.au/' (wp_scheduled_delete a:0:{})
[8] Started worker
[9] Running wp cavalcade run 9 --url='joy.org.au/' (update_network_counts a:0:{})
[9] Started worker
[10] Running wp cavalcade run 10 --url='joy.org.au/joyeurovision/' (gform_polls_cron a:0:{})
[10] Started worker
[  ] Out of workers
[  ] Out of workers
[  ] Out of workers
[  ] Out of workers
[  ] Out of workers
array(4) {
  [0]=>
  resource(9) of type (stream)
  [1]=>
  resource(13) of type (stream)
  [2]=>
  resource(16) of type (stream)
  [3]=>
  resource(19) of type (stream)
}
array(4) {
  [0]=>
  object(HM\Cavalcade\Runner\Worker)#3 (6) {
    ["process"]=>
    resource(11) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(9) of type (stream)
      [2]=>
      resource(10) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#4 (10) {
      ["id"]=>
      string(1) "7"
      ["site"]=>
      string(1) "1"
      ["hook"]=>
      string(41) "check_plugin_updates-user-role-editor-pro"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:27:47"
      ["nextrun"]=>
      string(19) "2015-11-11 06:27:47"
      ["interval"]=>
      string(5) "43200"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
  [1]=>
  object(HM\Cavalcade\Runner\Worker)#5 (6) {
    ["process"]=>
    resource(15) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(13) of type (stream)
      [2]=>
      resource(14) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#6 (10) {
      ["id"]=>
      string(1) "8"
      ["site"]=>
      string(1) "1"
      ["hook"]=>
      string(19) "wp_scheduled_delete"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:27:48"
      ["nextrun"]=>
      string(19) "2015-11-11 06:27:48"
      ["interval"]=>
      string(5) "86400"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
  [2]=>
  object(HM\Cavalcade\Runner\Worker)#7 (6) {
    ["process"]=>
    resource(18) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(16) of type (stream)
      [2]=>
      resource(17) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#8 (10) {
      ["id"]=>
      string(1) "9"
      ["site"]=>
      string(1) "1"
      ["hook"]=>
      string(21) "update_network_counts"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:27:48"
      ["nextrun"]=>
      string(19) "2015-11-11 06:27:48"
      ["interval"]=>
      string(5) "43200"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
  [3]=>
  object(HM\Cavalcade\Runner\Worker)#9 (6) {
    ["process"]=>
    resource(21) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(19) of type (stream)
      [2]=>
      resource(20) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#10 (10) {
      ["id"]=>
      string(2) "10"
      ["site"]=>
      string(2) "73"
      ["hook"]=>
      string(16) "gform_polls_cron"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:27:49"
      ["nextrun"]=>
      string(19) "2015-11-11 06:27:49"
      ["interval"]=>
      string(4) "3600"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
}
[7] out:
[7] err:
[7] ret: 0
[8] out:
[8] err:
[8] ret: 0
[9] out:
[9] err:
[9] ret: 0
[10] out:
[10] err:
[10] ret: 0
[10] Running wp cavalcade run 10 --url='joy.org.au/joyeurovision/' (gform_polls_cron a:0:{})
[10] Started worker
[11] Running wp cavalcade run 11 --url='joy.org.au/hideandseek/' (nxs_hourly_event a:0:{})
[11] Started worker
[12] Running wp cavalcade run 12 --url='joy.org.au/hideandseek/' (nxs_querypost_event a:0:{})
[12] Started worker
[13] Running wp cavalcade run 13 --url='joy.org.au/' (wp_version_check a:0:{})
[13] Started worker
[  ] Out of workers
[  ] Out of workers
[  ] Out of workers
[  ] Out of workers
array(1) {
  [0]=>
  resource(23) of type (stream)
}
array(4) {
  [4]=>
  object(HM\Cavalcade\Runner\Worker)#10 (6) {
    ["process"]=>
    resource(25) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(23) of type (stream)
      [2]=>
      resource(24) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#9 (10) {
      ["id"]=>
      string(2) "10"
      ["site"]=>
      string(2) "73"
      ["hook"]=>
      string(16) "gform_polls_cron"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:27:49"
      ["nextrun"]=>
      string(19) "2015-11-11 07:27:49"
      ["interval"]=>
      string(4) "3600"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
  [5]=>
  object(HM\Cavalcade\Runner\Worker)#11 (6) {
    ["process"]=>
    resource(28) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(26) of type (stream)
      [2]=>
      resource(27) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#7 (10) {
      ["id"]=>
      string(2) "11"
      ["site"]=>
      string(3) "144"
      ["hook"]=>
      string(16) "nxs_hourly_event"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:28:39"
      ["nextrun"]=>
      string(19) "2015-11-11 06:28:39"
      ["interval"]=>
      string(4) "3600"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
  [6]=>
  object(HM\Cavalcade\Runner\Worker)#8 (6) {
    ["process"]=>
    resource(31) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(29) of type (stream)
      [2]=>
      resource(30) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#5 (10) {
      ["id"]=>
      string(2) "12"
      ["site"]=>
      string(3) "144"
      ["hook"]=>
      string(19) "nxs_querypost_event"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:28:39"
      ["nextrun"]=>
      string(19) "2015-11-11 06:28:39"
      ["interval"]=>
      string(2) "90"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
  [7]=>
  object(HM\Cavalcade\Runner\Worker)#6 (6) {
    ["process"]=>
    resource(34) of type (process)
    ["pipes"]=>
    array(2) {
      [1]=>
      resource(32) of type (stream)
      [2]=>
      resource(33) of type (stream)
    }
    ["job"]=>
    object(HM\Cavalcade\Runner\Job)#3 (10) {
      ["id"]=>
      string(2) "13"
      ["site"]=>
      string(1) "1"
      ["hook"]=>
      string(16) "wp_version_check"
      ["args"]=>
      string(6) "a:0:{}"
      ["start"]=>
      string(19) "2015-11-11 06:29:24"
      ["nextrun"]=>
      string(19) "2015-11-11 06:29:24"
      ["interval"]=>
      string(5) "43200"
      ["status"]=>
      string(7) "waiting"
      ["db":protected]=>
      object(PDO)#2 (0) {
      }
      ["table_prefix":protected]=>
      string(3) "wp_"
    }
    ["output":protected]=>
    string(0) ""
    ["error_output":protected]=>
    string(0) ""
    ["status":protected]=>
    NULL
  }
}
Djelibeybi commented 8 years ago

Ping @rmccue -- this is still broken, btw.

rmccue commented 8 years ago

OK, so, I think the issue is that stream_select reindexes the pipes, which I hadn't accounted for. PR incoming.

Djelibeybi commented 8 years ago

The matchpipes branch still crashes. Error from /var/log/messages:

Feb 10 14:37:08 joy-dvmh-web-01 php: PHP Notice:  Undefined offset: 0 in /opt/Cavalcade-Runner/lib/Runner.php on line 226
Feb 10 14:37:08 joy-dvmh-web-01 php: PHP Fatal error:  Call to a member function is_done() on a non-object in /opt/Cavalcade-Runner/lib/Runner.php on line 227