ServiceBusListener is returning immediately when StopAsync is called.

[Thnslj]

I have implemented a draining flag to allow current Function runs to complete before stopping the process. When the flag is set i call the cancellationTokenSource.Cancel() and then calls jobHost.Stop().

jobHost.StartAsync(cancellationTokenSource.Token).GetAwaiter().GetResult();                
while (!settingsService.GetSettings().DisableWebJobs)
{
   Thread.Sleep(TimeSpan.FromSeconds(4));
}
cancellationTokenSource.Cancel();
jobHost.Stop();

Expected behavior

jobHost.Stop(); Should wait for functions started by ServiceBusListener to complete. Just like TaskSeriesTimer is implemented.

From TaskSeriesTimer.cs

   private async Task StopAsyncCore(CancellationToken cancellationToken)
        {
            await Task.Delay(0);
            TaskCompletionSource<object> cancellationTaskSource = new TaskCompletionSource<object>();

            using (cancellationToken.Register(() => cancellationTaskSource.SetCanceled()))
            {
                // Wait for all pending command tasks to complete (or cancellation of the token) before returning.
                await Task.WhenAny(_run, cancellationTaskSource.Task);
            }

            _stopped = true;
        }

Actual behavior

jobHost.Stop(); Returns immediately

Related information

Package version: <package id="Microsoft.Azure.WebJobs" version="2.0.0-beta1-10456" targetFramework="net461" />

[brettsam]

It looks like our Listeners are inconsistent wrt whether they allow in-process invocations to complete when StopAsync() is called. Some examples:

• Queues (and thus Blobs) wait for all message processing to complete: https://github.com/Azure/azure-webjobs-sdk/blob/dd176eed31aecd6d46e1df6bec1490cd6975f340/src/Microsoft.Azure.WebJobs.Host/Queues/Listeners/QueueListener.cs#L106
• Service Bus (the listener in question here) closes the receiver and returns, with no waiting: https://github.com/Azure/azure-webjobs-sdk/blob/dd176eed31aecd6d46e1df6bec1490cd6975f340/src/Microsoft.Azure.WebJobs.ServiceBus/Listeners/ServiceBusListener.cs#L78
• TimerTrigger cancels the token but doesn't wait for any in-progress execution to complete: https://github.com/Azure/azure-webjobs-sdk-extensions/blob/44c9bbbb873e2f371c48e66d1dd729e0eebbd924/src/WebJobs.Extensions/Extensions/Timers/Listener/TimerListener.cs#L130

@MikeStall / @mathewc / @davidebbo -- is there a 'best practice' for how Listeners should behave in this situation? It seems like waiting for current executions to complete is desirable. Should we try to get all Listeners to behave that way?

The drawback is that it's possible that a call to StopAsync() never returns -- or takes a very long time.

[MikeStall]

It'd be good for listeners to behave consistently. Even better if we can structure the extensibility model such that consistency is enforced. There's a few things to consider:

1. Once StopAsync() is called. When is it's task signaled?
2. allowing already-running functions to run to completion. Nice for short functions.
3. soft exit: firing a cancellation token to give a long-running functions a chance to softly abort
4. hard exit: forcibly killing long-running functions. In some cases, once the listener (or other "keep alive" tasks stops), there's no point in the function continuing to run since it may not be able to save its results.