search

Particular / NServiceBus.Transport.AzureServiceBus

Azure Service Bus transport
Other
22 stars 19 forks source link

Ensure message pump handles rate limiting requests #881

Open ramonsmits opened 1 year ago

ramonsmits commented 1 year ago

Describe the feature.

Currently when there are rate limiting failures during processing messages are being fetched immediately for processing again and the transport is not throttling to adhere to the rate limiting request

Considerations:

  1. Thottle / delay processing if the error is due to rate limiting
  2. Support rate limiting so that instance are not overloading Azure Service Bus

Additional Context

No response

SeanFeldman commented 1 year ago

Throttling by time, by # of messages, or a custom metric that is related to the number of messages but not 1 to 1.

ramonsmits commented 1 year ago

but not 1 to 1.

@SeanFeldman What do you mean by this?

SeanFeldman commented 1 year ago

For example, Intercom API allows up to 16 operations per second. But a single message might represent one or more operations against a 3rd party. Therefore, it's not always 16 messages per second but rather less. The endpoint needs to know how many operations it has performed to slow down or stop receiving until it can resume.

Another example is CosmosDB or any other resource that has throttling in place and reports back the quota of operations used or left.

ramonsmits commented 1 year ago

@SeanFeldman ok, rate limiting from the handlers processing perspective that is connecting to services. This issue is about rate limiting on the broker.

SeanFeldman commented 1 year ago

On the broker? The broker will give what it can. This is on endpoints. Endpoints pump messages.

ramonsmits commented 1 year ago

The broker will rate limit / throttle the namespace when consumers fetch too many messages. I'm not sure about the reason as docs mention connection limits too but the transport should gracefully handle that and not immediately fetch/reconnect when that happens.

SeanFeldman commented 1 year ago

That's doesn't sound right, @ramonsmits. How would the broker know what is "too many messages"? The broker will only throttle based on your namespace tier and MUs provisioned in Premium. Nothing else. The throughput of an endpoint is an endpoint business, not the broker. Hence this issue.

Think about it, if a namespace can send you N messages that result in O operations, where N < O, the namespace is not throttling anything. It's the 3rd part that does because it cannot accept more that Max messages, where N <= Max < O.