Support multi-tenant multi-database for outbox and inbox patterns for the distributed event bus

[hikalkan]

https://github.com/abpframework/abp/pull/10008 adds inbox & outbox patterns to the distributed event bus. However, it doesn't support multi-tenancy with a tenant has a dedicated database. We don't support it since it is not feasible. To make inbox/outbox consistent, we should save the message within the same database that is related to the event publishing/handling logic. The essential problem is in the background thread that publishes/processes outbox/inbox events. If we have thousands of databases, it would be a real problem to read events from these databases and publish/process. Also, we should dynamically re-arrange background worker threads for new created or deleted tenants.

That's really complicated. For a reference, the CAP library doesn't support this scenario (https://github.com/dotnetcore/CAP/issues/699). It also doesn't support multiple db scenario (https://github.com/dotnetcore/CAP/issues/998), but ABP supports it currently, so it works good for a modular system where each module has its own db.

I am opening a backlog item. I could not find a system that scales for multi-db multi-tenant scenario, but we may implement a limited solution where it works with low number of different tenant database but can't work as performant with high number of databases ( I don't know how many is low and high yet :) ).

[leonkosak]

For instance 50k tenants @hikalkan . :) https://github.com/abpframework/abp/issues/9467#issue-933549602

[gdlcf88]

Can we implement the inbox pattern by delayed ACK? That will support multi-tenant and multi-database.

Assume we are using RabbitMQ. When the event bus receives a message, it doesn't ACK yet, but only processes the message (event). After that, it invokes the RabbitMQ API to ACK the msg above.

I think the outbox/inbox should be in an independent module (like Volo.Abp.EventOutbox and Volo.Abp.EventInbox, or Volo.Abp.EventBus.Boxes). That will allow developers to use different implementations of the outbox/inbox. We can move the current inbox design to a new module so that it can be chosen as the default implementation if an event bus provider doesn't support the delayed ACK way.

[gdlcf88]

Hi Mr @hikalkan.

I saw your new issue https://github.com/abpframework/abp/issues/13338.

Check if we need (and can) use ABP's outbox and inbox patterns alongside Dapr (maybe dapr has a similar system already and we don't need it).

I'm afraid the Dapr team will not implement https://github.com/dapr/dapr/issues/4233 and https://github.com/dapr/dapr/issues/4247 since they are too heavy to the Dapr runtime and SDK. I asked about it from a friend who works in the Dapr team and got the above answer.

As I know, ABP's built-in event box implementation doesn't support the multi-tenant & multi-DB scenario. Some day I found and tested a new way to implement the event box pattern: DTM's 2PC messaging. I have created a community module https://github.com/EasyAbp/Abp.EventBus.Boxes.Dtm and published a post to introduce it.

However, the event box is a core infrastructure that should natively support the multi-tenant & multi-DB scenario. I believe the ABP team will eventually resolve it instead of relying on a community module.

The problem with the integration of DTM is that it depends on the DTM Server. So I created a new .NET package for a new "distributed job" definition. It still depends on a TM provider, but I'm ready to design a simple "local-TM" as the TM provider for a startup, and the DTM Server is not necessary by default.

https://github.com/TeamStepping/Stepping.NET

This project is named "Stepping". I try to follow Volosoft's code style, so it is expected to be easy to extend and override as an ABP module. Maybe we create a "Volo.Abp.Stepping" module first and then use it to implement the "Volo.Abp.EventBus.Boxes.Stepping" module (as default or an option for the app startup template). I will define some interfaces for local-TM like ITransactionStore so we can use an ABP repository to implement it. That means developers can install it as a regular application module.

The TM provider is easy to switch. If someone wants the app host to be lighter or to get more types of distributed transaction patterns, he can switch to using DTM Server. And the good news is that the DTM Server is trying to join the Dapr as a new building block. So Stepping will be easier to use DTM Server as the TM provider in the future.

Thanks to the "Volo.Abp.Stepping" module installed, developers can also use the API of Stepping directly. It helps to execute a BASE job with multi-steps. For example, I can ensure an email will eventually be sent after the local DB transaction is committed by using Stepping. That is also the principle we use to implement the event boxes.

DTM's author @yedf2 also favors Stepping.NET😄, and we discussed it for a long time in the online meeting. What I want to know is, is it possible for the ABP team to use Stepping? (That's important since it's almost designed for ABP) If so, I will archive the Abp.EventBus.Boxes.Dtm module and wait for Volosoft's implementation.

Finally, thanks for your reading.

UPDATE: The Stepping.NET 1.0.0 has been released.

[payoff]

Making a feature available and doing a partial development, which predicts the wrong malfunction in most cases is more harmful than not having the feature. The fact that using Event Inbox / Outbox with Tenats does not throw exceptions, but fails tacitly (I write in the tenant I read in the host) waste days. To be proactive I would say that ABP should follow one of these paths:

• Put ASAP in the documentation which is not supported in multi-tenant environment
• With minimal effort put a throw NotImplemented if used in multi-tenant

Consideration. Other developments in ABP have used different logics, backgroudjob for example works only in host. Having said this ,advice to those who want to use it, to avoid external tools that in fact flatten the use and replicate what is possible by implementing tenantless IDbContextEventOutbox.

Overriding the e class:

        using (_currentTenant.Change (null))
        {
            return base.EnqueueAsync (outgoingEvent);
        }

Everything turns beautifully.

[JCEdward]

Hi @hikalkan How about this issue, how to use inbox/ outbox with multiple tenant db?