Is there any way to implement transport initialization asynchronously?

[glazkovalex]

In Apache Kafka, rebalancing when connecting consumers is difficult and time-consuming. It can consist of several asynchronous operations and last for minutes or longer. Now I have to initialize inside the RegisterSubscriber and delay the start of receiving messages until it is completed. Is there any way to implement transport initialization asynchronously?

[mookid8000]

Is there any way to implement transport initialization asynchronously?

Yes - by starting it during initialization and the letting it run in the background.

You'll of course have to implement the proper synchronization mechanisms to ensure that the first call to e.g.

await bus.Subscribe<Whatever>();

blocks asynchronously (e.g. on an AsyncManualResetEvent from Nito.Asyncex) so it will actually not send anything before the transport has been initialized.

It will of course also lose the ability to fail fast during initialization, if something is not right - so. e.g. if the connection string to Kafka points to an invalid hostname, you'll have to monitor your logs to find out.

[mookid8000]

By the way: Kafka is very different from queue-based message brokers in many ways, and I've never felt that it would be a good Rebus transport.

When I needed "a Rebus, but for Kafka" myself, I built Topos, which basically wraps Kafka and makes it pleasant to work with and has a little bit of the Rebus feeling when you configure it.

As a bonus, it can also run in "embedded mode", using Microsoft FASTER (the WAL part) as an embedded, file-based event store.

I'm not saying it can't be done - but you'll probably meet friction in various places (as you are now).