Host crashes on StorageBaseDistributedLockManager.SingletonLockHandle error - StorageException: Conflict - There is currently no lease on the blob/container

Please provide a succinct description of the issue.

Repro steps

Provide the steps required to reproduce the problem

Use following packages in csproj,

<PackageReference Include="Azure.Messaging.EventHubs" Version="5.6.0" />
<PackageReference Include="Azure.Storage.Queues" Version="12.8.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="3.0.6" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="5.0.0" />

create a WebJob host as follows:

    [Singleton]
    public void RunOnTimerTrigger2([TimerTrigger("*/1 * * * * *", UseMonitor = false)] TimerInfo myTimer, ILogger log)
    {
        Trace.WriteLine($"[{DateTime.UtcNow.ToString("o")}] x");
    }

Run this with

        using (host)
        {
            var jobHost = host.Services.GetService(typeof(IJobHost)) as JobHost;
            await host.StartAsync();
            Console.ReadKey();
            await host.StopAsync();
        }

Expected behavior

Application/Host shouldn't crash if a single Function has issues with Singleton; instead maybe cancel the token for that specific Function.

Actual behavior

Program throws following exception and crashes/restarts... This occurs with local development - haven't tried with Azure Storage. Crashed in 20 minutes. Similar to...

https://github.com/Azure/azure-webjobs-sdk/issues/2103

[2022-03-09T20:12:34.9849314Z]
...
[2022-03-09T20:31:43.5061751Z]
Unhandled exception. [2022-03-09T20:31:37.6380363Z] VatInformationUpdate_AppsConsolidatedDBAzure1_19_0
Unhandled exception. Microsoft.WindowsAzure.Storage.StorageException: Conflict
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.Azure.WebJobs.Host.StorageBaseDistributedLockManager.SingletonLockHandle.RenewAsync(ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host.Storage\Singleton\BlobLeaseDistributedLockManager.cs:line 380
   at Microsoft.Azure.WebJobs.Host.SingletonManager.RenewLeaseCommand.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonManager.cs:line 341
   at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.RunAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\TaskSeriesTimer.cs:line 147
   at Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.<>c__DisplayClass3_0.<OnUnhandledExceptionAsync>b__0() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\WebJobsExceptionHandler.cs:line 55
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadHelper.ThreadStart()
Request Information
RequestID:
RequestDate:Wed, 09 Mar 2022 12:31:41 GMT
StatusMessage:Conflict
ErrorCode:
ErrorMessage:There is currently no lease on the blob/container.

Microsoft.WindowsAzure.Storage.StorageException: Conflict
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.Azure.WebJobs.Host.StorageBaseDistributedLockManager.SingletonLockHandle.RenewAsync(ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host.Storage\Singleton\BlobLeaseDistributedLockManager.cs:line 380
   at Microsoft.Azure.WebJobs.Host.SingletonManager.RenewLeaseCommand.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonManager.cs:line 341
   at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.RunAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\TaskSeriesTimer.cs:line 147
   at Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.<>c__DisplayClass3_0.<OnUnhandledExceptionAsync>b__0() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\WebJobsExceptionHandler.cs:line 55
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadHelper.ThreadStart()
Request Information
RequestID:
RequestDate:Wed, 09 Mar 2022 12:31:40 GMT
StatusMessage:Conflict
ErrorCode:
ErrorMessage:There is currently no lease on the blob/container.

The easiest way to repro would be to make the client console app stuck, by freezing Console output as follows.

This will simulate a scenario where Lease would be lost to the client, and this process crashes. Please note, in Production scenarios this exact scenario may occur if CPU runs on high for very long OR ThreadPool got starved, and lease renewal process didn't complete successfully.

Known workarounds

Provide a description of any known workarounds.

Related information

Provide any related information

Package version
Links to source

Azure / azure-webjobs-sdk