Open brikou opened 12 years ago
This would constitute a new batch strategy, one that I would like to support out of the box but haven't written yet. The createBatches()
method would return a special iterator whose current()
method waits until one of the first four batches exits before returning the fifth, for example.
Is there timeout for Chunk strategy? If one fork work in infinity loop, will it die after some timeout?
I made process() with new ChunkStrategy(10) but it still generate only 3 forks like default value. How i can make 10 or more forks ?
@seyfer how many data are you passing into the batch job? If you configure 10 chunks but only pass 3 data, there will only be 3 forks.
I have high load system and always have requests in database. I take from db 10 tasks and give it to Spork. All this happen in daemon. I made debug, and give 8 tasks. Then in wait() method of manager i print count($this->forks) and it prints 8.
But on server, or on local machine i see only 3-4 forks by "ps aux | grep minion".
Previosly i use AzaThread (https://github.com/amal/AzaThread) and when i configure 10 forks in Aza, i see 10 in system, when 8 - i see 8. And Aza faster process my tasks. Was forced to withdraw from this library due to a bug in libevent.
I think Spork don't create 10 forks. If you can - please test.
My example:
$requests_iterator = new ArrayIterator($requests_to_process);
$fork = $this->sporkManager->process($requests_iterator, function($request) {
try
{
$this->model->reconnect();
$asbresult = $this->asbProcessor->run($request);
if ($asbresult)
{
$id = $this->asbProcessor->getId();
$this->asbProcessor->update($id, $asbresult);
return TRUE;
}
return FALSE;
}
catch (\Exception $exc)
{
return $exc->getMessage();
}
}, new ChunkStrategy($this->maxThreads));
$this->sporkManager->wait();
$requests_to_process is array with 8 or 10 tasks.
$this->maxThreads = 10 in example
watch --interval=1 'ps aux | grep minion'
I have monitoring and see only 3 forks maximum, but configured 10.
Made better monitoring. There no erros, all works fine.
I think issue may be closed, becouse i write example for issue author. :) But there still need more examples for Strategies.
Thank for support!
As this library is brand new... capabilities need to be discovered, I wish some basic simple example where provided, through some gist for example.
NB: What I'm trying to achieve is parallel computation (~batch processing) with N job running at the same time... when on is finished append a new one until all job are done...
Cheer