Can this be the queue package Vapor is missing?

[Casperhr]

I just saw this the other day And it's one of the things I talked with tanner & logan about.

In php we had a queue system https://laravel.com/docs/5.3/queues

But that was cause how php runs (request based)

With Vapor we have a runtime so this load can be done on the webservers

One feature (from doc) which is missing, is just firering a job once as an "async task" this can be useful for low response time apis.

Is this something this package can do?

Also thinking even further Moving the load away from webservers to another server (like the laravel queues system) can be very useful. and having it running as a queue with workers is a good way to deal high load.

Just wanted to open a discussion. If you see this package can be that one day or it should be another package?

Awesome work btw!

// Casper

[Casperhr]

Also what happens to all you timed jobs, if you redeploy? nothing is stored in a db?

[BrettRToomey]

Currently, I'm working on a date/time system to make queuing of scheduled jobs a lot easier than having to fuss with Swift Dates. Scheduled one-offs will be part of this update.

I specifically designed this package, its interface and dependencies to be able to run independent of the server's source code. That way, if someone is paranoid about the job system taking down their service processes or wants to use some type of load balancer they have the ability to do so.

When I finish the date and scheduling interfaces, I planned on tagging the code for 0.1 as it's a prerequisite for JSON-configurable jobs (0.2). JSON-configurable jobs will finish all of the prerequisites for persistent jobs and batches of jobs with batch controllers. Then, I will launch version 1.0 and will begin using the standard milestone/tag releases for software.

I'm sure that by 1.0 this package will be able to do everything you're looking for and with a minimalistic, Swifty interface.

[Casperhr]

Sounds awesome!

And regarding the question about, restart of the drop? Are you storing the info for when a job should be executed in a db or something? (Beanstalkd is using redis or a db fx)

[BrettRToomey]

The plan I had was more along the lines of some protocol that will be called to save/read queued jobs and it will pass an array of structs containing all the metadata. Then, the author can use any database they want to store and retrieve the information. The main issue is that json-job are the only types of jobs that can persist through a restart as I will lose reference to the closures when the memory is freed by the OS.

I've also been strongly considering adding support for jobs using the Lua scripting language. I have lots of experience with game engines and Lua integration and could support it relatively easily. This way, you can have more advanced scripted jobs that would be JIT compiled and you could even open up Swift functions to the Lua LVM to be able to call internal/Vapor functions within the scripts. Lua scripts would be able to persist through a restart, as well.

[Casperhr]

Ye It's not a easy task

Laravel had support for closure queues, but they removed it again in 5.3 it was just random buggy and they used a lib called Super Closure which tried to serialise closures. Never had that working stable in production

Today queues laravel have an interface for jobs, where you pass in params. Which can be serialised.

Looking forward to see where you can take this. It's a major part of a successful web framework to have a bulletproof queue system.

The scheduled tasks can be useful for something, but we always schedule commands through linux cron jobs. But this can work different

Cron is run every x time Queues starts instant, you can delay them 30-90 min with some providers But this can also be for smaller things, which can be triggered from the code.

I see the potential!