Message Broker implementation for in-process use case

[marksto]

Hi fine folks of Nilenso!

Have a question on whether or not the Goose design is well-suited for using an in-process Message Broker, i.e. when an up itself mediates the jobs (enqueues, retries, replays orphan, etc.), using a simple in-memory data structure (say, a Clojure built-in seque or a java.util.concurrent.BlockingQueue directly) or something more elaborate and durable (such as a durable-queue).

I strongly appreciate your work on Goose, well done! Just wondering if an in-process scenario is a good fit for it or not.

Cheers, Mark

[olttwa]

Hey @marksto,

Thanks for the kind words. We hope Goose is of help to you and your team!

Conventionally, server, broker and worker infra is hosted independently to help with decoupling and scalability. An in-process Message Broker would compromise these advantages of an asynchronous system, as that would involve running all 3 components on the same machine.

Can you give some applications and use-cases where an in-process broker would be helpful?

Thanks!

[olttwa]

Hey @marksto,

I'm marking this issue as closed for now. Feel free to re-open if you'd like to share something new.

We're open to contributions for an in-process broker. However, I don't see a production usecase for that yet, so would caution against one. A simple in-process fan-out/fan-in could be implemented using core.async as well.

In other news, we have begun work for adding support of a Goose Console for administration and dashboard templates for monitoring, and the mock-ups are looking very promising. Sharing a sneak peek here :)