Issues with web job that uses Managed Identity to connect to Azure SQL Database

Using Managed Identity to connect to Azure SQL Database, both when developing in Visual Studio or when hosted as web job in Azure, throws exception and it shouldn't.

Repro steps

Create .NET 7.0 console app
Reference latest versions of the following NuGet packages:
- Microsoft.Azure.WebJobs.Extensions (5.0.0)
- Microsoft.Azure.WebJobs.Host.Storage (5.0.0)
- Microsoft.Data.SqlClient NuGet package (5.1.1)
- Azure.Identity (1.9.0)
Configure the console app as web job by calling .ConfigureWebJobs(b => { b.AddAzureStorageCoreServices();}) on the host builder and configure logging (console. file etc.)
Add NoAutomaticTrigger function within which you try to open connection to an Azure SQL Database with appropriate connection string (e.g. Server=tcp:<server-name>.database.windows.net,1433;Authentication=Active Directory Default;Database=TestDatabase;) and additionaly you may select some records from a table in the database.
Run the web job

Expected behavior

The connection to the database is open with success and records retrieved. No FAIL error logged in the console except some exceptions logged as INFO coming from Azure.Identity when figuring out which option works to fetch access token to connect to the database (e.g. EnvironmentCredential.GetToken, WorkloadIdentityCredential.GetToken, ManagedIdentityCredential.GetToken, VisualStudioCredential.GetToken etc.)

Actual behavior

While figuring out which option works to fetch access token to connect to the database, when ManagedIdentityCredential.GetToken is called it throws exception

fail: Azure.Identity[10]
      False MSAL 4.49.1.0 MSAL.NetCore .NET 7.0.5 Microsoft Windows 10.0.19045 [2023-05-22 12:57:13Z - 3db4e06a-2576-4e47-b61b-8c93fe9f0d7f] Exception type: Azure.Identity.CredentialUnavailableException
      ---> Inner Exception Details
      Exception type: System.AggregateException
      ---> Inner Exception Details
      Exception type: Azure.RequestFailedException
      ---> Inner Exception Details
      Exception type: System.Net.Http.HttpRequestException
      ---> Inner Exception Details
      Exception type: System.Net.Sockets.SocketException

         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|281_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
      === End of inner exception stack trace ===

         at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
         at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
      === End of inner exception stack trace ===

         at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
         at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
         at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
      === End of inner exception stack trace ===

         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
      === End of inner exception stack trace ===

         at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
         at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToProviderAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)

Known workarounds

If you use version 1.7.0 or below of Azure.Identity this does not happen. All version of Azure.Identity higher than 1.7.0 throw the exception.

Related information

If the console app is not configured as WebJob thus the ConfigureWebJobs is not called at all, any version of Azure.Identity works without any problems, no error logs show up. We cannot downgrade to Azure.Identity version 1.7.0 since another NuGet package we use has dependency of Azure.Identity version >= 1.8.2

There are additional messages logged (False MSAL 4.49.1.0 MSAL.NetCore .NET 7.0.5....) in the console such as these below which are not logged when the console app is not configured as WebJob:

All this happens ONLY when we configure the app as web job.

Package version
Microsoft.Azure.WebJobs.Extensions (5.0.0)
Microsoft.Azure.WebJobs.Host.Storage (5.0.0)
Microsoft.Data.SqlClient NuGet package (5.1.1)
Azure.Identity (1.8.0 - 1.9.0)
Links to source

Azure / azure-webjobs-sdk