Bug - Quota Exceeded exception when publishing messages via ServiceBus trigger using AMQP

Repro steps

Create new webjob project
Install WindowsAzure.ServiceBus nuget package (I am using 3.4.0 but have tested earlier versions)
Create a new method triggered from service bus topic and publishing to a service bus topic - ` public void EventSubscriptionToNotificationTopic( [ServiceBusTrigger("eventtopic", "sasSubscription")] BrokeredMessage input, [ServiceBus("notificationtopic")] out BrokeredMessage output) { NumberWriter.PrintCount(); output = new BrokeredMessage(); }
```
private static int count = 0;

public static void PrintCount()
{
    Console.WriteLine($"Number Processed = {count}");
    count++;
}`
```
I added a number writer here to identify via the console how many messages have been processed.
Set the transport type in the connection string to AMQP - ;TransportType=Amqp
Place 6000 message on the test input topic. I used service bus explorer to do this.
Run the web job

Expected behavior

All messages to be processed and 6000 new messages to appear on the test output topic

Actual behavior

Exception is thrown after reaching around 5000 messages -

Function had errors. See Azure WebJobs SDK dashboard for details. Instance ID is '95f16cad-da8a-4638-aff4-d9982bb174ca' Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: WorkflowWebJob.EventSubscriptionToNotificationTopic ---> System.InvalidOperationException: Error while handling parameter output after function returned: ---> Microsoft.ServiceBus.Messaging.QuotaExceededException: Cannot allocate more handles to the current session or connection. The maximum number of handles allowed is 4999. Please free up resources and try again. at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult1.End(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.Amqp.FaultTolerantObject1.OnEndCreateInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.SingletonManager1.EndGetInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessageSender.OnEndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.ClientEntity.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.OpenOnceManager.OnEndCreateInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.SingletonManager1.EndGetInstance(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult1.OpenComplete(IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult.SyncContinue(IAsyncResult result) at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult1..ctor(OpenOnceManager openOnceManager, TimeSpan openTimeout, AsyncCallback callback, Object state, Func3 beginOperation, EndOperation1 endOperation) at Microsoft.ServiceBus.Messaging.OpenOnceManager.Begin(AsyncCallback callback, Object state, Func3 beginOperation, Action1 endOperation) at Microsoft.ServiceBus.Messaging.MessageSender.BeginSend(TrackingContext trackingContext, IEnumerable1 messages, TimeSpan timeout, AsyncCallback callback,Object state) at Microsoft.ServiceBus.Messaging.MessageSender.BeginSend(BrokeredMessage message, AsyncCallback callback, Object state) at Microsoft.ServiceBus.Messaging.MessageSender.<>c__DisplayClass40_0.<SendAsync>b__0(AsyncCallback c, Object s) at System.Threading.Tasks.TaskFactory1.FromAsyncImpl(Func3 beginMethod, Func2 endFunction, Action1 endAction, Object state, TaskCreationOptions creationOptions) at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CreateTask(Func3 begin,Action`1 end, Object state)--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.ServiceBus.Bindings.MessageSenderExtensions.d0.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.ServiceBus.Bindings.BrokeredMessageArgumentBinding.MessageValueBinder.d0.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d31.MoveNext() --- End of inner exception stack trace --- at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d31.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d2c.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d13.MoveNext() --- End of inner exception stack trace --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d13.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d1.MoveNext()

Known workarounds

Switching back to NetMessaging protocol by removing the transport type setting in the connection string.

Related information

It appears that some AMQP element is either not being used as a singleton correctly or is not being disposed. Looking at the heap whilst it is running shows many instances of SendingAMQPLink and ActiveClientLink. This is where the issue is I think.

Azure / azure-webjobs-sdk