Run queue tasks continuously

[davidbarratt]

Is your feature request related to a problem? Please describe. We use Drupal's Queue feature to handle background tasks. However, many of these tasks are time sensitive and need to be executed in the background as soon as possible.

Drush doesn't provide a way to run a specific queue as a daemon, it can only be invoked on a schedule.

Describe the solution you'd like I would really like a way to run Drupal queue as a long running task in the foreground. Then I can use systemd, docker, or k8s to keep it running.

Maybe a queue:watch command that is like: https://github.com/drush-ops/drush/blob/151b70439cdacda5b95546ca853f271d0fe981cf/src/Commands/core/QueueCommands.php#L60-L113 but that is running in an infinite loop (with a usleep()?) until the time limit or item limit is reached? Basically, it should keep running even if there are zero items in the queue right now.

It might be nice to also limit the number of iterations by the amount of memory (i.e. memory_get_usage()) which would allow the script to be kept alive for as long as possible.

Describe alternatives you've considered

• Running in a cron tab, but the most frequent you can set that to is once a minute.
• Using a loop in a shell script that runs drush queue:run, but then we are doing the work to bootstrap Drupal and connect to the database on every iteration.

Related:

• Advanced Queue Runner

Additional context N/A

[davidbarratt]

~I think this might be blocked on https://www.drupal.org/project/drupal/issues/2218651~

The work-around is to bootstrap Drupal with each queue run, which isn't the worst, but I can put drush queue:run in a shell script with an infinite loop and a sleep and I think that's probably the best option for now.

[Chi-teck]

I would really like a way to run Drupal cron or a specific queue as a long running task in the foreground.

Cron as long running task sounds weird. Cron is meant for scheduled tasks. If you have a need to constantly process some large data, i.e. import products from external sources it's better to create a worker which could be an ordinary Drush command. Furthermore, even for scheduled custom tasks I would not mess with hook_cron but create a separate Drush command and put it to crontab like follows.

1 * * * * ./vendor/bin/drush example >> ./var/log/cron/exampe.log 2>&1

[davidbarratt]

Ah that's right, I forget that modules can run tasks directly on cron run so maybe just having a long-running queue runner is actually what I want.

[davidbarratt]

@Chi-teck I updated the description, I hope this is helpful. it's honestly probably just an option on the existing command tbh

[jonpugh]

I was tinkering with using (Advanced) Queue API for running server side commands. I thought there was a daemon!

FWIW... This is how Aegir does it, with a Drush 8 command "hosting-queued": https://git.drupalcode.org/project/hosting/-/blob/7.x-3.x/queued/hosting_queued.drush.inc?ref_type=heads#L52-214

Would this be useful? https://github.com/mac-cain13/daemonizable-command

Gonna look into this...

[webflo]

I implemented something similar as a command some time ago. However, I don't currently use it. https://github.com/ueberbit/drush_queue_daemon

[claudiu-cristea]

Not sure if this module I've built some time ago could help https://www.drupal.org/project/recurring_task