search

halaei / bredis

Redis queue driver for Laravel < 5.6 with blocking pop support (this is ported to laravel 5.6 core)
2 stars 0 forks source link

Delayed jobs #3

Open francislavoie opened 6 years ago

francislavoie commented 6 years ago

In the README:

Warning: bredis queue workers don't bother handling jobs that are delayed or reserved. So when using bredis workers, you have to have at least one redis worker as well.

Is there a specific reason for this? I'd like the add support for delayed jobs.

Some background: I needed to implement a shim for encrypting job payloads (domain requirement for all data at rest to be encrypted), but I was using bredis, so I decided to basically just copy bredis into my project and modify it to add support for that, because the way to implement encryption is to extend the existing queue driver. I can't really have two queue drivers with separate features... so I just decided to copy yours and make my changes on top of it. This also means that I'd need to have a separate driver which only has my encryption changes without blpop to handle delayed jobs. If there's an easier way to just have this one driver do everything, that'd obviously be better.

Also, would love if you took a look at #2 again! :stuck_out_tongue:

I'd appreciate being pointed in the right direction!

Thanks

halaei commented 6 years ago

Currently this package has nothing more than what laravel 5.6 already supports. So I encourage you to switch to 5.6, if you haven't done yet and replace 'bredis' driver with 'redis' and remove this package from your project dependencies.

The reason I didn't handle delayed/reserved jobs here was some minor performance considerations. Plus, I think you already know retrieving jobs from delayed/reserved queues cannot be blocking; a job might be ready in a delayed queue, but blocked workers cannot be notified until after some other worker moves the expired jobs to the main queue. This is due to the limitation of existing Redis commands.

Regarding issue #2 , I don't think brpoplpush is the best choice anymore, it is going to be Redis Modules API. I get to know Redis Module API for some other projects and I find that resolving all the issues of current Redis queue drivers can be done via modules API. So with that module you are going to expect these changes:

  1. Blocking pop on delayed/reserved queues: let say we have a worker waiting for a job, no job in the main queue, but a job is already inside the delayed queue that is due in 1.5 seconds. Immediately after 1.5 seconds, the job will be available for the worker :)
  2. Reliability of blocking pops, since the migrations are going to be done inside Redis.
  3. No need to introduce any change to the data-structures in Redis, hence 100% backward compatibility.
  4. some minor performance gain or loss.

I've already started the coding. My expectation is that the module will be ready if I can spend less than 5 work days on it. But I run short of time, so no guarantee on the release date.

francislavoie commented 6 years ago

Okay, sounds good, thanks for the elaborate answer!

I was hoping to stick with 5.5 as it's supposed to be the LTS release, but I'm seeing that I might as well just stick with latest at this point.

I'm excited to see more about what you get working with the modules API! Sounds promising.

halaei commented 6 years ago

Hello @francislavoie The redis module for laravel queue is almost ready. It uses redis unstable module api (will probably be available in redis 5.0 or 5.1). The code passes some basic tests, but it requires more careful tests and refactors. Also please note that I abandoned this repository in packagist :)