Queue worker and scheduler

[ggolda]

Hello! I'm curious if you have plans to support a queue worker and scheduler in Octane? Since we now have a single process for php application and as I can see people already working on a websocket server implementation, that would be cool.

Scheduler in Octane will also provide an option to schedule tasks with frequency less than 1 minute.

[themsaid]

Why would you want to run your queues under Octane?

As for the scheduler, feel free to submit a PR.

[sikhlana]

Scheduler in Octane will also provide an option to schedule tasks with frequency less than 1 minute.

There's already a very nice package for this: https://github.com/spatie/laravel-short-schedule. But yes, a scheduler using swoole would be a nice addition instead of using ReactPHP.

[ggolda]

@themsaid for queue worker the idea was that it will allow to use code that has swoole/octane related features and not worry about its underlying implementation.

If queue worker is a separate php process as it is today, I need to double check everything that is used in a queue task and verify that it has no dependencies on octane related features.

[sikhlana]

@ggolda it is better to keep each of the paradigms of a framework separate. The command for php artisan octane:start should not care about pending jobs in a queue. The only thing this command should care about is handling HTTP requests.

Yes, there are task workers, but that is mainly for the Octane::concurrently() method to add asynchronicity into the mixture.

Now let's say we implement handling queued jobs by the workers. This might hinder the actual usage of the package: "serving HTTP requests at supersonic speeds" when there are long-running jobs and no available worker.

[ggolda]

@sikhlana I disagree, in every other language a process that handles http requests also responsible for all other types of work - queues, scheduling and etc. Plus code can be reused between them freely.

Related to last point - as I understand those services (swoole/roadrunner) allows to create a separate worker whose responsibility would be only to handle queue tasks. So it will not affect any http handling performance.

[sikhlana]

in every other language a process that handles http requests also responsible for all other types of work - queues, scheduling and etc.

I guess by "every other language" you mean event-driven languages like Go and NodeJS? Because from what I've seen, Django uses celery as a separate process for scheduling and running jobs. Spring can handle all of them in a single process with the help of threads so that one paradigm is not affected by others. The main issue with a single PHP process to handle everything is that PHP is not asynchronous at the core level. Fibers is being added, let's see what we can get from this.

Yes, it would be nice the have an all-in-one command like php artisan start which will handle:

• php artisan octane:start
• php artisan schedule:work
• php artisan queue:work

But don't add running queued jobs feature in the current octane:start command is what I am saying.

[sikhlana]

Yes, it would be nice the have an all-in-one command like php artisan start which will handle:

This actually can be easily implemented using Symfony's Process component.

[themsaid]

I don't really see a point in running queues under Octane. Just run horizon or queue:work as a separate process.

Closing.

[iamoskvin]

I don't really see a point in running queues under Octane. Just run horizon or queue:work as a separate process.

Closing.

I need memory-lightweight queue workers (I plan to run approx 100 workers on small-middle size VPS). And each Laravel worker is a separate process, so each worker consumes a substantial amount of RAM. I suppose that with coroutine mechanics we could run each worker as a thread and not process.

[andrew-belac]

I have the same wish. We have 1000's of concurrent tasks. So having queue:work/schedule:work is fine if it could be run using octane with a definied number of workers the same way octane:start works.

[zek]

In my use case, I'm sending push notifications (APN) to users. Sending each notification requires TCP handshake with Apple servers first. If I can just use same Client, I can send notifications faster and it'll use less memory. Octane support would be perfect for processing same kind of tasks.

[xico42]

Another use case for this is when you have a queue worker deployed on kubernetes, for example.

It is useful to have the queue worker process also handle http requests for the probes (health check and liveness) or even a metrics endpoint (for prometheus scraping)

[bernardwiesner]

This would be a nice addition. Using symphony process is not the same since you need to bootstrap the whole framework in every process if you want to use all of laravel features which defeats the purpose of speed and performance.

Using octane and a queue worker you could concurrently process jobs using all features of the framework without needing a cache layer while maintaining high performance.

[sikhlana]

@bernardwiesner from my understanding, swoole itself forks child processes under the hood, so basically the same as creating separate processes; roadrunner creates separate PHP processes that is handled by the main go process.

And again, managing queues and scheduler is out of the scope of what Laravel Octane is meant for. Maybe create an issue in the main repo to add support for swoole for queues and scheduler?

[bernardwiesner]

@sikhlana The Symphony Process does not fork but creates a new process. That means no sharing of memory and also bootstrapping whole Laravel framework if you want to use all the features for every process.

Yes you can use forking manually and achieve something similar to swoole but that is not what symphony process does and forking manually comes with it's caveats you have to deal with.

Octane could have a method such as php artisan octane:run that runs all your configured commands in a long running process loop.