Persistence

[benwilson512]

Have you looked into extracting the queue into a behaviour that could possibly support multiple backends? With the scheduler / queue distinction mentioned in #1 it would be a lot easier to shrink the size and scope of the queue genserver, and define a behaviour that even a user supplied module could adhere to.

The advantage is that if someone wanted to write a GenServer that acted as a queue but also persisted the job states to redis, ets, etc they could do so without any extra effort on your part.

Thanks!

[koudelka]

I haven't, but I think that's a fantastic idea. I personally think a rabbitmq queue would be neat. :)

An early prototype of the library used Mnesia as the job queue, but I didn't feel confident enough in the solution, so I scrapped it. It'd be interesting to experiment with that again.

The method of persistence would, of course, be up to the implementor. However it worked though, it'd have to support bidirectional communications, so that the worker can return the result of a call to the requesting process. Though, I suspect that in cases where job persistence is needed, you wouldn't be using call anyways.

While I think it'd be really cool to offer a pluggable architecture, I'm also worried about trying to be everything to everyone, it sounds like a recipe for complexity. Maybe it's just a question of positioning, perhaps advertising Honeydew as "like the Task module, but with queues" would set expectations properly. (as an aside, I think moving the API to mimic Task is something that merits consideration)

Whacha think? :)

[koudelka]

@benwilson512 Fixed! You can now plug in your own job queue module, or use the provided Mnesia queue configured to use DETS. You can also provide your own GenStage dispatcher to control how jobs are sent to workers.

Want to give it a spin and let me know what you think?

[seivan]

@koudelka How would one use Redis as a backing store?

[koudelka]

@seivan It'd involve writing a "Queue" implementation, like https://github.com/koudelka/honeydew/blob/master/lib/honeydew/queue/erlang_queue.ex

You'd probably lose the pre-existing ability to send responses back to the calling process when the job completes, though.

[seivan]

@koudelka You mean a form of ACK/NACK ? I think it can be "simulated" by using BRPOPLPUSH to a secon queue and eventually either remove it or push it on a fail/retry queue?

[koudelka]

@seivan Yeah, that'd probably work. I don't plan on maintaining a Redis queue module myself, though.