Other alternatives to RabbitMQ

[giovannipizzi]

Hi Chris, just adding here the link to another message broker that a colleague just mentioned to me today: https://mqtt.org/ This seems to be much more lightweight than RabbitMQ, as it's designed for IoT devices, but has similar concepts (actually I think there is a RabbitMQ plugin for the mqtt protocol).

It seems there are many py client libraries, a few are unmaintained but at least one or two seem to be actively maintained (like the eclipse one). Beside checking if this would work for our usecase, one would need also to check if there are robust and easy to install servers/brokers (otherwise we just shift the problem rather than fixing it :-) ).

I just wanted to mention it here so we don't forget - a solution as proposed in this repo without additional servers has clearly that pro, but I fear that implementing a robust message passing protocol with guarantees is a lot of work, and relying on some existing protocol and implementation (but simpler to install and use than RabbitMQ) could be a good compromise.

(As a note there's also ZeroMQ but I think it's a much more low-level communication library, while mqtt has a full protocol definition).

[chrisjsewell]

Cheers Gio,

One thing I would kind of "counter" with though, is that we are not really looking for a message broker per se; this is our whole problem with our (mis)use of rabbitmq, a message broker is for passing messages, not for running processes.

I was actually talking recently to Marnik about things like rate limiting the amount calculations that can be submitted to a scheduler at a time, e.g. to deal with job limits on HPCs. I noted that if you can get this "vertical integration" of a process coordinator right, this kind of thing becomes trivial to implement.

I do take your point about implementation work though; (good) re-use of existing libraries is always something I would look for; although again our current reliance on kiwipy + aiopika + aiormq is not exactly simple or easy to debug. I noted recently (https://github.com/aiidateam/kiwipy/pull/100), for example, that this is the sequence of calls required just to send a simple kill message:

verdi process kill

plumpy.process_comms.RemoteProcessThreadController.kill_process msg={"intent": "kill", "message": "..."}
-> kiwipy.rmq.threadcomms.RmqThreadCommunicator.rpc_send (wrap syncio await with timeout)
-> async kiwipy.rmq.communicator.RMQCommunicator.rpc_send
-> async kiwipy.rmq.communicator.RmqPublisher.rpc_send (converts message to aio_pika.Message)
-> async kiwipy.rmq.messages.BasePublisherWithReplyQueue.publish_expect_response
-> async kiwipy.rmq.messages.BasePublisherWithReplyQueue.publish
-> async aio_pika.exchange.Exchange.publish
-> async aiormq.channel.Channel.basic_publish

As a note there's also ZeroMQ

FYI, this is what circus already uses

[giovannipizzi]

I agree with your points, and we'll need to say if a simple solution (e.g. the one you propose) is simple enough (probably yes) and robust enough (hard to say without proper testing). I guess the discussion on a replacement for RabbitMQ is a good topic for one of the discussions of a coding week :-)

[chrisjsewell]

Indeed 👍

[csadorf]

I've made a comment on the associated AEP relevant to this question: https://github.com/aiidateam/AEP/pull/30#discussion_r813895745 .