Particular / NServiceBus.Transport.AzureServiceBus

Azure Service Bus transport
Other
22 stars 19 forks source link

Support configuring event subscription expressions to limit total subscriptions #780

Open DavidBoike opened 1 year ago

DavidBoike commented 1 year ago

Describe the feature.

Is your feature related to a problem? Please describe.

Let's say you have an event MyNamespace.MyEvent and need to version it so you create MyNamespace.V2.MyEvent.

On the subscription bundle, 2 rules will be created:

The way filters are created is not currently configurable. Although you can configure the Azure Service Bus entities however you like and the messages will flow correctly, you can't make NServiceBus installers do anything but create the 2 rules above.

In a system with many event types where versioning like this is required (for whatever reason, that's unimportant) this can quickly get out of hand since subscriptions on a bundle are a limited resource.

Describe the requested feature

Instead, an advanced option could allow customization of the filter. Something like a Func<Type, string> for creating the filter:

var asb = new AzureServiceBusTransport(…);

asb.CreateFilterExpression = type => $"%{type.Name}%";

This results of expression would be deduped, so the previous example would only create one rule:

Describe alternatives you've considered

There doesn't seem to be any other alternative, as the feature is to make something configurable that is currently hard-coded.

Additional Context

When implementing, the following should be considered:

Original idea from @SeanFeldman

danielmarbach commented 1 year ago

I think, conceptually, we should take a step back and look at the broader picture a bit (*). Currently, the transport rule creation policy is centered around a 1:1 relationship between the event type and the rule. This means for every event type subscribed, a rule will be created that turns the event type into the SQL filter described here in the issue. While exposing a function delegate as suggested here might allow tweaking the rule filter, such a solution might suffer from the shortcoming of "not knowing the full picture".

With the v8 transport seam redesign, auto-subscribe passed all subscribed event types to the subscription manager. This means the subscription topology is known during the auto-subscribe time. Once that full picture is known, it becomes possible to ask the question of "given all these subscription intents what is the minimal set of rules required to represent the subscription rules".

The rules expressed need to be stable not just for the auto-subscription phase but also for the manual subscribe/unsubscribe phase to make sure the rule(s) outcome is deterministic. A Func<Type, string> would not be able to take that into account.

In Amazon SQS we had a similar challenge to solve. The number of policies on a queue is a finite and small number. The number of event types subscribed on a single endpoint could easily be larger in the context of 1:1 (event type to policy) and then the policy propagation would fail. We solved this in SQS by having policy rules that either match a namespace with wildcard or an account level. This allows to "condense" multiple event types subscribed to only a handful of policies.

I think what we are discussing here falls into a similar territory because, after all we intend to condense potentially multiple event types subscribed to a smaller number of rules. As far as I understand, rule evaluation comes with a runtime cost associated with it. Given that the topology design requires SQL filters (which are more expensive than correlation filters) I would assume having less rules would benefit the rule evaluation and thus improve the throughput. Furthermore, it would enable scenarios in this issue too.

(*) I think the title of this issue does that pretty well already, while the description is centered around a possible approach/solution that takes a single event type into account and my comments are referring to that part by trying to think "outside the box of the solution proposal".

SeanFeldman commented 1 year ago

The primary concern is not the cost but the number of entities a namespace can have. If it would be solvable by money, it wouldn't be an issue.