Suggestion: Support the nats.io as Messageing system

[zfsamzfsam]

NATS is a simple, high performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures. https://nats.io/

[RichardKnop]

@zfsamzfsam If you want to contribute, adding NATS broker would mean implementing this interface: https://github.com/RichardKnop/machinery/blob/master/v1/brokers/interfaces.go#L8

You can take a look at AMQP and Redis broker implementations to get some idea how it's done currently.

https://github.com/RichardKnop/machinery/blob/master/v1/brokers/amqp.go https://github.com/RichardKnop/machinery/blob/master/v1/brokers/redis.go

How would you implement delayed tasks with NATS? Currently we finally got that working with both AMQP and Redis backends so any new brokers we add would have to support this feature to stay consistent.

Anyways, if you have time to implement NATS broker I'd gladly review it.

[paulm17]

@zfsamzfsam It's been a while, but since no-one has approached this. I want this as well, so will be working on it. If I'm successful, PR to follow...

[paulm17]

Thought I would update. I have Nats working for a Broker and I have a MemSQL cluster working as a backend, similarly to how memcache works.

INFO: 2018/04/30 15:23:49 file.go:19 Successfully loaded config from file config.yml
INFO: 2018/04/30 15:23:49 worker.go:43 Launching a worker with the following settings:
INFO: 2018/04/30 15:23:49 worker.go:44 - Broker: nats://localhost:4222
INFO: 2018/04/30 15:23:49 worker.go:45 - DefaultQueue: machinery_tasks
INFO: 2018/04/30 15:23:49 worker.go:46 - ResultBackend: memsqlcache://root:f00b4r@tcp(192.168.0.28:3306)/machinery
INFO: 2018/04/30 15:23:49 nats.go:45 [*] Waiting for messages. To exit press CTRL+C

Tomorrow, I will use NATS Streaming instead of the basic one. As this includes things acknowledgements and might as well have it as close to the AMPQ implementation.

[Blogoslov]

Thought I would update. I have Nats working for a Broker and I have a MemSQL cluster working as a backend, similarly to how memcache works.

INFO: 2018/04/30 15:23:49 file.go:19 Successfully loaded config from file config.yml
INFO: 2018/04/30 15:23:49 worker.go:43 Launching a worker with the following settings:
INFO: 2018/04/30 15:23:49 worker.go:44 - Broker: nats://localhost:4222
INFO: 2018/04/30 15:23:49 worker.go:45 - DefaultQueue: machinery_tasks
INFO: 2018/04/30 15:23:49 worker.go:46 - ResultBackend: memsqlcache://root:f00b4r@tcp(192.168.0.28:3306)/machinery
INFO: 2018/04/30 15:23:49 nats.go:45 [*] Waiting for messages. To exit press CTRL+C

Tomorrow, I will use NATS Streaming instead of the basic one. As this includes things acknowledgements and might as well have it as close to the AMPQ implementation.

Hello @paulm17 , how did you implemented NATS as a broker? Do you have its own machinery fork? I'm also interested in NATS as a broker. Could you please share your implementation?

[paulm17]

@Blogoslov Take a look here: https://github.com/paulm17/machinery/blob/master/v1/brokers/nats.go

I have created a repo with the entire code. You are going to have to go through all of the codebase to see how I got it all working.

There may be things missing compared to the AMPQ implementation as Nats may have things missing.

Also bear in mind that the API for nats may have been changed in the last year. Again, you'll have to go through the code and maybe debug it. But at least it's a starting point.

Anyway, this is the best that I can do. I no longer use machinery, I have developed my own solution.

[Blogoslov]

Anyway, this is the best that I can do. I no longer use machinery, I have developed my own solution.

Hello @paulm17 . Thanks for the sharing, I will review and check your solution. Thanks a lot.

By the way - does your new solution is available on github or some other place?

[paulm17]

@Blogoslov unfortunately it's not. I really recommend looking at all the other job queue solutions out there and breaking down how they work and then building something that suits your own workflow. It's essentially what I did.