We've got plenty of jobs and dependencies and we are facing a performance issue when the job runner is trying to fetch the next startable job - see JobRepository::findStartableJobs().
We have ~2k pending jobs that are scheduled in the future, thus not startable (executeAfter > now()) and we've got a lot of other jobs that depends on them. For the latter, we did not bother setting the executeAfter property, so they could be started as soon as all their dependencies have run successfully.
What's happening here is that the job runner wants to find the next startable jobs and start iterating over these pending - but not startable - jobs, and discards them one by one (pushing their id into $excludedIds). We've got plenty of them, so it iterates _a lot_ and it's taking a huge amount of time, something like 70 seconds. Just to find the next startable job.
We've got a pull here that fixes our case by fetching the next startable job in a single, raw SQL query.
What do you think? I could submit a pull rebased on the master branch of this repo, if it ever makes sense to you.
PS: As I was writing these lines, I was thinking of another workaround: adjust executeAfter property as soon as a new dependency is added, if the latter is scheduled in the future, so the former is not picked up by method findPendingJob().
Hello there,
We've got plenty of jobs and dependencies and we are facing a performance issue when the job runner is trying to fetch the next startable job - see
JobRepository::findStartableJobs().We have ~2k pending jobs that are scheduled in the future, thus not startable (
executeAfter > now()) and we've got a lot of other jobs that depends on them. For the latter, we did not bother setting theexecuteAfterproperty, so they could be started as soon as all their dependencies have run successfully.What's happening here is that the job runner wants to find the next startable jobs and start iterating over these pending - but not startable - jobs, and discards them one by one (pushing their id into
$excludedIds). We've got plenty of them, so it iterates _a lot_ and it's taking a huge amount of time, something like 70 seconds. Just to find the next startable job.We've got a pull here that fixes our case by fetching the next startable job in a single, raw SQL query.
What do you think? I could submit a pull rebased on the master branch of this repo, if it ever makes sense to you.
PS: As I was writing these lines, I was thinking of another workaround: adjust
executeAfterproperty as soon as a new dependency is added, if the latter is scheduled in the future, so the former is not picked up by methodfindPendingJob().